Oct 30 23:40:43.813613 ip-10-0-141-180 systemd[1]: Starting Kubernetes Kubelet... Oct 30 23:40:44.259470 ip-10-0-141-180 kubenswrapper[2550]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 23:40:44.259470 ip-10-0-141-180 kubenswrapper[2550]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 23:40:44.259470 ip-10-0-141-180 kubenswrapper[2550]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 23:40:44.259470 ip-10-0-141-180 kubenswrapper[2550]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 23:40:44.264099 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.264017 2550 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 23:40:44.264099 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.264099 2550 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 23:40:44.264168 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.264103 2550 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 23:40:44.264168 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.264106 2550 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 23:40:44.264168 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.264110 2550 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 23:40:44.264168 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.264113 2550 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 23:40:44.264168 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.264116 2550 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 23:40:44.264168 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.264119 2550 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 23:40:44.264168 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.264122 2550 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 23:40:44.264168 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.264125 2550 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 23:40:44.264168 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.264128 2550 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 23:40:44.264168 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.264131 2550 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 23:40:44.264168 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.264133 2550 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 23:40:44.264168 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.264137 2550 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 23:40:44.264168 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.264140 2550 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 23:40:44.264168 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.264143 2550 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 23:40:44.264168 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.264145 2550 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 23:40:44.264168 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.264148 2550 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 23:40:44.264168 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.264151 2550 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 23:40:44.264168 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.264153 2550 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 23:40:44.264168 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.264160 2550 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 23:40:44.264168 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.264163 2550 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 23:40:44.264652 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.264166 2550 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 23:40:44.264652 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.264168 2550 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 23:40:44.264652 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.264171 2550 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 23:40:44.264652 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.264174 2550 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 23:40:44.264652 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.264177 2550 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 23:40:44.264652 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.264180 2550 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 23:40:44.264652 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.264183 2550 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 23:40:44.264652 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.264186 2550 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 23:40:44.264652 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.264188 2550 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 23:40:44.264652 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.264191 2550 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 23:40:44.264652 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.264194 2550 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 23:40:44.264652 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.264196 2550 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 23:40:44.264652 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.264199 2550 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 23:40:44.264652 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.264202 2550 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 23:40:44.264652 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.264204 2550 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 23:40:44.264652 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.264207 2550 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 23:40:44.264652 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.264210 2550 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 23:40:44.264652 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.264213 2550 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 23:40:44.264652 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.264215 2550 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 23:40:44.265097 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.264218 2550 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 23:40:44.265097 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.264221 2550 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 23:40:44.265097 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.264223 2550 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 23:40:44.265097 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.264226 2550 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 23:40:44.265097 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.264229 2550 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 23:40:44.265097 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.264231 2550 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 23:40:44.265097 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.264234 2550 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 23:40:44.265097 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.264237 2550 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 23:40:44.265097 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.264239 2550 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 23:40:44.265097 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.264242 2550 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 23:40:44.265097 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.264245 2550 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 23:40:44.265097 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.264247 2550 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 23:40:44.265097 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.264251 2550 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 23:40:44.265097 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.264254 2550 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 23:40:44.265097 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.264257 2550 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 23:40:44.265097 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.264259 2550 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 23:40:44.265097 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.264262 2550 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 23:40:44.265097 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.264265 2550 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 23:40:44.265097 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.264268 2550 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 23:40:44.265097 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.264270 2550 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 23:40:44.265580 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.264273 2550 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 23:40:44.265580 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.264276 2550 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 23:40:44.265580 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.264279 2550 feature_gate.go:386] unrecognized feature gate: Example Oct 30 23:40:44.265580 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.264281 2550 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 23:40:44.265580 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.264286 2550 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 23:40:44.265580 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.264291 2550 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 23:40:44.265580 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.264294 2550 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 23:40:44.265580 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.264297 2550 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 23:40:44.265580 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.264300 2550 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 23:40:44.265580 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.264303 2550 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 23:40:44.265580 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.264307 2550 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 23:40:44.265580 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.264312 2550 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 23:40:44.265580 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.264315 2550 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 23:40:44.265580 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.264319 2550 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 23:40:44.265580 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.264322 2550 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 23:40:44.265580 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.264324 2550 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 23:40:44.265580 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.264327 2550 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 23:40:44.265580 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.264342 2550 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 23:40:44.265580 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.264345 2550 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 23:40:44.266029 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.264348 2550 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 23:40:44.266029 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.264350 2550 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 23:40:44.266029 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.264353 2550 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 23:40:44.266029 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.264356 2550 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 23:40:44.266029 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.264359 2550 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 23:40:44.266029 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.264362 2550 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 23:40:44.266029 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.264366 2550 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 23:40:44.266029 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.264369 2550 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 23:40:44.266029 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.264372 2550 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 23:40:44.266029 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.264375 2550 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 23:40:44.266029 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.264378 2550 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 23:40:44.266029 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.264381 2550 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 23:40:44.266029 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.264384 2550 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 23:40:44.266029 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.264387 2550 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 23:40:44.266029 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.264389 2550 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 23:40:44.266029 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.264392 2550 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 23:40:44.266029 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.264395 2550 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 23:40:44.266029 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.264398 2550 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 23:40:44.266029 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.264400 2550 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 23:40:44.266517 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.264403 2550 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 23:40:44.266517 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.264406 2550 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 23:40:44.266517 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.264409 2550 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 23:40:44.266517 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.264413 2550 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 23:40:44.266517 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.264415 2550 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 23:40:44.266517 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.264418 2550 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 23:40:44.266517 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.264421 2550 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 23:40:44.266517 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.265211 2550 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 23:40:44.266517 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.265218 2550 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 23:40:44.266517 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.265221 2550 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 23:40:44.266517 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.265224 2550 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 23:40:44.266517 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.265226 2550 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 23:40:44.266517 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.265229 2550 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 23:40:44.266517 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.265232 2550 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 23:40:44.266517 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.265237 2550 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 23:40:44.266517 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.265241 2550 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 23:40:44.266517 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.265246 2550 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 23:40:44.266517 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.265249 2550 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 23:40:44.266517 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.265252 2550 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 23:40:44.267016 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.265255 2550 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 23:40:44.267016 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.265258 2550 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 23:40:44.267016 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.265260 2550 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 23:40:44.267016 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.265263 2550 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 23:40:44.267016 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.265266 2550 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 23:40:44.267016 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.265269 2550 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 23:40:44.267016 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.265272 2550 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 23:40:44.267016 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.265274 2550 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 23:40:44.267016 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.265277 2550 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 23:40:44.267016 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.265280 2550 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 23:40:44.267016 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.265282 2550 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 23:40:44.267016 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.265285 2550 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 23:40:44.267016 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.265287 2550 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 23:40:44.267016 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.265290 2550 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 23:40:44.267016 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.265292 2550 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 23:40:44.267016 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.265295 2550 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 23:40:44.267016 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.265298 2550 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 23:40:44.267016 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.265300 2550 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 23:40:44.267016 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.265303 2550 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 23:40:44.267016 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.265306 2550 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 23:40:44.267505 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.265309 2550 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 23:40:44.267505 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.265341 2550 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 23:40:44.267505 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.265346 2550 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 23:40:44.267505 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.265349 2550 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 23:40:44.267505 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.265352 2550 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 23:40:44.267505 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.265355 2550 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 23:40:44.267505 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.265358 2550 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 23:40:44.267505 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.265360 2550 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 23:40:44.267505 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.265363 2550 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 23:40:44.267505 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.265366 2550 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 23:40:44.267505 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.265368 2550 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 23:40:44.267505 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.265371 2550 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 23:40:44.267505 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.265374 2550 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 23:40:44.267505 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.265377 2550 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 23:40:44.267505 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.265380 2550 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 23:40:44.267505 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.265383 2550 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 23:40:44.267505 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.265386 2550 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 23:40:44.267505 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.265388 2550 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 23:40:44.267505 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.265391 2550 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 23:40:44.267505 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.265394 2550 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 23:40:44.267980 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.265396 2550 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 23:40:44.267980 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.265399 2550 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 23:40:44.267980 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.265401 2550 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 23:40:44.267980 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.265404 2550 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 23:40:44.267980 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.265407 2550 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 23:40:44.267980 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.265412 2550 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 23:40:44.267980 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.265416 2550 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 23:40:44.267980 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.265421 2550 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 23:40:44.267980 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.265424 2550 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 23:40:44.267980 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.265428 2550 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 23:40:44.267980 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.265430 2550 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 23:40:44.267980 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.265433 2550 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 23:40:44.267980 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.265436 2550 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 23:40:44.267980 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.265439 2550 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 23:40:44.267980 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.265442 2550 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 23:40:44.267980 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.265444 2550 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 23:40:44.267980 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.265447 2550 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 23:40:44.267980 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.265450 2550 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 23:40:44.267980 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.265453 2550 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 23:40:44.268459 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.265455 2550 feature_gate.go:386] unrecognized feature gate: Example Oct 30 23:40:44.268459 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.265458 2550 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 23:40:44.268459 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.265461 2550 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 23:40:44.268459 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.265463 2550 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 23:40:44.268459 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.265466 2550 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 23:40:44.268459 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.265469 2550 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 23:40:44.268459 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.265472 2550 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 23:40:44.268459 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.265474 2550 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 23:40:44.268459 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.265477 2550 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 23:40:44.268459 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.265480 2550 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 23:40:44.268459 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.265482 2550 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 23:40:44.268459 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.265485 2550 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 23:40:44.268459 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.265488 2550 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 23:40:44.268459 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.265490 2550 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 23:40:44.268459 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.265493 2550 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 23:40:44.268459 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.265495 2550 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 23:40:44.268459 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.265498 2550 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 23:40:44.268459 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.265501 2550 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 23:40:44.268459 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.265504 2550 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 23:40:44.268459 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.265507 2550 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 23:40:44.268943 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.265511 2550 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 23:40:44.268943 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.265513 2550 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 23:40:44.268943 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.265516 2550 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 23:40:44.268943 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.265518 2550 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 23:40:44.268943 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.265521 2550 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 23:40:44.268943 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.265524 2550 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 23:40:44.268943 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.265527 2550 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 23:40:44.268943 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.265530 2550 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 23:40:44.268943 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.265532 2550 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 23:40:44.268943 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.265535 2550 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 23:40:44.268943 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.265538 2550 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 23:40:44.268943 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.265540 2550 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 23:40:44.268943 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.265543 2550 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 23:40:44.268943 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.265545 2550 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 23:40:44.268943 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.265549 2550 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 23:40:44.268943 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.266024 2550 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 23:40:44.268943 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.266031 2550 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 23:40:44.268943 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.266034 2550 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 23:40:44.268943 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.266037 2550 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 23:40:44.268943 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.266040 2550 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 23:40:44.269477 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.266043 2550 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 23:40:44.269477 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.266046 2550 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 23:40:44.269477 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.266048 2550 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 23:40:44.269477 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.266051 2550 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 23:40:44.269477 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.266054 2550 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 23:40:44.269477 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.266057 2550 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 23:40:44.269477 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.266060 2550 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 23:40:44.269477 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.266063 2550 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 23:40:44.269477 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.266065 2550 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 23:40:44.269477 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.266068 2550 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 23:40:44.269477 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.266071 2550 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 23:40:44.269477 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.266074 2550 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 23:40:44.269477 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.266079 2550 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 23:40:44.269477 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.266082 2550 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 23:40:44.269477 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.266084 2550 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 23:40:44.269477 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.266087 2550 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 23:40:44.269477 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.266090 2550 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 23:40:44.269477 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.266092 2550 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 23:40:44.269477 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.266095 2550 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 23:40:44.269477 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.266098 2550 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 23:40:44.269957 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.266101 2550 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 23:40:44.269957 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.266104 2550 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 23:40:44.269957 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.266107 2550 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 23:40:44.269957 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.266109 2550 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 23:40:44.269957 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.266112 2550 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 23:40:44.269957 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.266115 2550 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 23:40:44.269957 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.266117 2550 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 23:40:44.269957 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.266120 2550 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 23:40:44.269957 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.266123 2550 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 23:40:44.269957 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.266125 2550 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 23:40:44.269957 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.266128 2550 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 23:40:44.269957 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.266131 2550 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 23:40:44.269957 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.266134 2550 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 23:40:44.269957 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.266137 2550 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 23:40:44.269957 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.266140 2550 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 23:40:44.269957 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.266142 2550 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 23:40:44.269957 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.266145 2550 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 23:40:44.269957 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.266148 2550 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 23:40:44.269957 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.266150 2550 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 23:40:44.269957 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.266153 2550 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 23:40:44.270445 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.266156 2550 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 23:40:44.270445 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.266158 2550 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 23:40:44.270445 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.266161 2550 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 23:40:44.270445 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.266164 2550 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 23:40:44.270445 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.266168 2550 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 23:40:44.270445 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.266171 2550 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 23:40:44.270445 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.266173 2550 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 23:40:44.270445 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.266176 2550 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 23:40:44.270445 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.266179 2550 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 23:40:44.270445 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.266181 2550 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 23:40:44.270445 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.266184 2550 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 23:40:44.270445 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.266187 2550 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 23:40:44.270445 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.266190 2550 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 23:40:44.270445 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.266193 2550 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 23:40:44.270445 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.266195 2550 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 23:40:44.270445 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.266198 2550 feature_gate.go:386] unrecognized feature gate: Example Oct 30 23:40:44.270445 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.266201 2550 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 23:40:44.270445 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.266203 2550 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 23:40:44.270445 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.266206 2550 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 23:40:44.270445 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.266209 2550 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 23:40:44.270910 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.266211 2550 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 23:40:44.270910 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.266214 2550 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 23:40:44.270910 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.266216 2550 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 23:40:44.270910 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.266219 2550 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 23:40:44.270910 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.266222 2550 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 23:40:44.270910 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.266224 2550 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 23:40:44.270910 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.266227 2550 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 23:40:44.270910 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.266229 2550 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 23:40:44.270910 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.266232 2550 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 23:40:44.270910 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.266235 2550 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 23:40:44.270910 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.266237 2550 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 23:40:44.270910 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.266240 2550 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 23:40:44.270910 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.266243 2550 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 23:40:44.270910 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.266245 2550 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 23:40:44.270910 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.266248 2550 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 23:40:44.270910 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.266251 2550 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 23:40:44.270910 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.266254 2550 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 23:40:44.270910 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.266257 2550 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 23:40:44.270910 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.266260 2550 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 23:40:44.270910 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.266263 2550 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 23:40:44.271397 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.266265 2550 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 23:40:44.271397 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.266268 2550 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 23:40:44.271397 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.266272 2550 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 23:40:44.271397 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.266276 2550 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 23:40:44.271397 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.266278 2550 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 23:40:44.271397 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.266281 2550 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 23:40:44.271397 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.266284 2550 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 23:40:44.271397 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.266287 2550 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 23:40:44.271397 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.266290 2550 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 23:40:44.271397 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.266292 2550 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 23:40:44.271397 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.266295 2550 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 23:40:44.271397 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.266298 2550 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 23:40:44.271397 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.266302 2550 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 23:40:44.271397 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.266306 2550 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 23:40:44.271397 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.266310 2550 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 23:40:44.271397 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.266312 2550 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 23:40:44.271397 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.266315 2550 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 23:40:44.271397 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.266318 2550 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 23:40:44.271814 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.266320 2550 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 23:40:44.271814 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.266323 2550 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 23:40:44.271814 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.266327 2550 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 23:40:44.271814 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.268263 2550 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 23:40:44.271814 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.268274 2550 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 23:40:44.271814 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.268281 2550 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 23:40:44.271814 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.268287 2550 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 23:40:44.271814 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.268291 2550 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 23:40:44.271814 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.268294 2550 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 23:40:44.271814 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.268300 2550 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 23:40:44.271814 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.268305 2550 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 23:40:44.271814 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.268308 2550 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 23:40:44.271814 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.268311 2550 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 23:40:44.271814 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.268314 2550 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 23:40:44.271814 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.268318 2550 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 23:40:44.271814 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.268321 2550 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 23:40:44.271814 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.268324 2550 flags.go:64] FLAG: --cgroup-root="" Oct 30 23:40:44.271814 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.268327 2550 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 23:40:44.271814 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.268341 2550 flags.go:64] FLAG: --client-ca-file="" Oct 30 23:40:44.271814 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.268344 2550 flags.go:64] FLAG: --cloud-provider="external" Oct 30 23:40:44.271814 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.268348 2550 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 23:40:44.271814 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.268353 2550 flags.go:64] FLAG: --cluster-domain="" Oct 30 23:40:44.271814 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.268356 2550 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 23:40:44.271814 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.268360 2550 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 23:40:44.272387 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.268363 2550 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 23:40:44.272387 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.268366 2550 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 23:40:44.272387 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.268370 2550 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 23:40:44.272387 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.268374 2550 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 23:40:44.272387 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.268377 2550 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 23:40:44.272387 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.268380 2550 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 23:40:44.272387 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.268383 2550 flags.go:64] FLAG: --contention-profiling="false" Oct 30 23:40:44.272387 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.268386 2550 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 23:40:44.272387 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.268390 2550 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 23:40:44.272387 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.268393 2550 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 23:40:44.272387 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.268396 2550 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 23:40:44.272387 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.268401 2550 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 23:40:44.272387 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.268404 2550 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 23:40:44.272387 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.268408 2550 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 23:40:44.272387 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.268411 2550 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 23:40:44.272387 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.268415 2550 flags.go:64] FLAG: --enable-server="true" Oct 30 23:40:44.272387 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.268418 2550 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 23:40:44.272387 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.268423 2550 flags.go:64] FLAG: --event-burst="100" Oct 30 23:40:44.272387 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.268426 2550 flags.go:64] FLAG: --event-qps="50" Oct 30 23:40:44.272387 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.268429 2550 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 23:40:44.272387 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.268432 2550 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 23:40:44.272387 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.268435 2550 flags.go:64] FLAG: --eviction-hard="" Oct 30 23:40:44.272387 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.268440 2550 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 23:40:44.272387 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.268443 2550 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 23:40:44.273093 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.268447 2550 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 23:40:44.273093 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.268450 2550 flags.go:64] FLAG: --eviction-soft="" Oct 30 23:40:44.273093 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.268454 2550 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 23:40:44.273093 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.268457 2550 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 23:40:44.273093 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.268460 2550 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 23:40:44.273093 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.268463 2550 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 23:40:44.273093 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.268467 2550 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 23:40:44.273093 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.268470 2550 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 23:40:44.273093 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.268473 2550 flags.go:64] FLAG: --feature-gates="" Oct 30 23:40:44.273093 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.268477 2550 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 23:40:44.273093 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.268480 2550 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 23:40:44.273093 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.268484 2550 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 23:40:44.273093 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.268487 2550 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 23:40:44.273093 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.268490 2550 flags.go:64] FLAG: --healthz-port="10248" Oct 30 23:40:44.273093 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.268493 2550 flags.go:64] FLAG: --help="false" Oct 30 23:40:44.273093 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.268497 2550 flags.go:64] FLAG: --hostname-override="ip-10-0-141-180.us-west-2.compute.internal" Oct 30 23:40:44.273093 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.268500 2550 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 23:40:44.273093 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.268503 2550 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 23:40:44.273093 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.268506 2550 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 23:40:44.273093 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.268510 2550 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 23:40:44.273093 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.268513 2550 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 23:40:44.273093 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.268517 2550 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 23:40:44.273093 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.268520 2550 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 23:40:44.273660 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.268522 2550 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 23:40:44.273660 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.268526 2550 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 23:40:44.273660 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.268529 2550 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 23:40:44.273660 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.268532 2550 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 23:40:44.273660 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.268535 2550 flags.go:64] FLAG: --kube-reserved="" Oct 30 23:40:44.273660 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.268538 2550 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 23:40:44.273660 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.268541 2550 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 23:40:44.273660 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.268544 2550 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 23:40:44.273660 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.268547 2550 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 23:40:44.273660 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.268550 2550 flags.go:64] FLAG: --lock-file="" Oct 30 23:40:44.273660 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.268553 2550 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 23:40:44.273660 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.268556 2550 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 23:40:44.273660 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.268559 2550 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 23:40:44.273660 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.268565 2550 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 23:40:44.273660 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.268568 2550 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 23:40:44.273660 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.268571 2550 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 23:40:44.273660 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.268575 2550 flags.go:64] FLAG: --logging-format="text" Oct 30 23:40:44.273660 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.268578 2550 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 23:40:44.273660 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.268581 2550 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 23:40:44.273660 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.268584 2550 flags.go:64] FLAG: --manifest-url="" Oct 30 23:40:44.273660 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.268587 2550 flags.go:64] FLAG: --manifest-url-header="" Oct 30 23:40:44.273660 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.268592 2550 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 23:40:44.273660 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.268595 2550 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 23:40:44.273660 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.268599 2550 flags.go:64] FLAG: --max-pods="110" Oct 30 23:40:44.273660 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.268602 2550 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 23:40:44.274254 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.268605 2550 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 23:40:44.274254 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.268608 2550 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 23:40:44.274254 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.268611 2550 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 23:40:44.274254 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.268614 2550 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 23:40:44.274254 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.268617 2550 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 23:40:44.274254 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.268620 2550 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 23:40:44.274254 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.268628 2550 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 23:40:44.274254 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.268631 2550 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 23:40:44.274254 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.268634 2550 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 23:40:44.274254 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.268638 2550 flags.go:64] FLAG: --pod-cidr="" Oct 30 23:40:44.274254 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.268641 2550 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 23:40:44.274254 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.268644 2550 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 23:40:44.274254 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.268647 2550 flags.go:64] FLAG: --pods-per-core="0" Oct 30 23:40:44.274254 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.268650 2550 flags.go:64] FLAG: --port="10250" Oct 30 23:40:44.274254 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.268653 2550 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 23:40:44.274254 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.268656 2550 flags.go:64] FLAG: --provider-id="aws:///us-west-2a/i-01256209b5ec9c35f" Oct 30 23:40:44.274254 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.268659 2550 flags.go:64] FLAG: --qos-reserved="" Oct 30 23:40:44.274254 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.268662 2550 flags.go:64] FLAG: --read-only-port="10255" Oct 30 23:40:44.274254 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.268665 2550 flags.go:64] FLAG: --register-node="true" Oct 30 23:40:44.274254 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.268668 2550 flags.go:64] FLAG: --register-with-taints="" Oct 30 23:40:44.274254 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.268672 2550 flags.go:64] FLAG: --registry-burst="10" Oct 30 23:40:44.274254 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.268675 2550 flags.go:64] FLAG: --registry-qps="5" Oct 30 23:40:44.274254 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.268678 2550 flags.go:64] FLAG: --reserved-cpus="" Oct 30 23:40:44.274254 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.268681 2550 flags.go:64] FLAG: --reserved-memory="" Oct 30 23:40:44.274254 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.268685 2550 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 23:40:44.274254 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.268688 2550 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 23:40:44.274865 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.268691 2550 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 23:40:44.274865 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.268694 2550 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 23:40:44.274865 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.268697 2550 flags.go:64] FLAG: --runonce="false" Oct 30 23:40:44.274865 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.268700 2550 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 23:40:44.274865 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.268705 2550 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 23:40:44.274865 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.268708 2550 flags.go:64] FLAG: --seccomp-default="false" Oct 30 23:40:44.274865 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.268712 2550 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 23:40:44.274865 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.268715 2550 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 23:40:44.274865 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.268718 2550 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 23:40:44.274865 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.268721 2550 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 23:40:44.274865 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.268724 2550 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 23:40:44.274865 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.268728 2550 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 23:40:44.274865 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.268731 2550 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 23:40:44.274865 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.268734 2550 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 23:40:44.274865 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.268737 2550 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 23:40:44.274865 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.268741 2550 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 23:40:44.274865 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.268744 2550 flags.go:64] FLAG: --system-cgroups="" Oct 30 23:40:44.274865 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.268746 2550 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 23:40:44.274865 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.268752 2550 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 23:40:44.274865 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.268755 2550 flags.go:64] FLAG: --tls-cert-file="" Oct 30 23:40:44.274865 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.268758 2550 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 23:40:44.274865 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.268763 2550 flags.go:64] FLAG: --tls-min-version="" Oct 30 23:40:44.274865 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.268766 2550 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 23:40:44.274865 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.268769 2550 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 23:40:44.274865 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.268772 2550 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 23:40:44.275497 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.268775 2550 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 23:40:44.275497 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.268778 2550 flags.go:64] FLAG: --v="2" Oct 30 23:40:44.275497 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.268783 2550 flags.go:64] FLAG: --version="false" Oct 30 23:40:44.275497 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.268787 2550 flags.go:64] FLAG: --vmodule="" Oct 30 23:40:44.275497 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.268791 2550 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 23:40:44.275497 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.268795 2550 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 23:40:44.275497 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.268936 2550 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 23:40:44.275497 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.268940 2550 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 23:40:44.275497 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.268944 2550 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 23:40:44.275497 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.268947 2550 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 23:40:44.275497 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.268950 2550 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 23:40:44.275497 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.268955 2550 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 23:40:44.275497 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.268958 2550 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 23:40:44.275497 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.268961 2550 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 23:40:44.275497 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.268963 2550 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 23:40:44.275497 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.268966 2550 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 23:40:44.275497 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.268969 2550 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 23:40:44.275497 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.268972 2550 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 23:40:44.275497 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.268975 2550 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 23:40:44.275497 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.268978 2550 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 23:40:44.275497 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.268982 2550 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 23:40:44.275983 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.268985 2550 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 23:40:44.275983 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.268988 2550 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 23:40:44.275983 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.268990 2550 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 23:40:44.275983 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.268993 2550 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 23:40:44.275983 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.268996 2550 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 23:40:44.275983 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.268998 2550 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 23:40:44.275983 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.269001 2550 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 23:40:44.275983 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.269004 2550 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 23:40:44.275983 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.269007 2550 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 23:40:44.275983 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.269010 2550 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 23:40:44.275983 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.269012 2550 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 23:40:44.275983 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.269015 2550 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 23:40:44.275983 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.269017 2550 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 23:40:44.275983 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.269020 2550 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 23:40:44.275983 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.269023 2550 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 23:40:44.275983 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.269025 2550 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 23:40:44.275983 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.269028 2550 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 23:40:44.275983 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.269031 2550 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 23:40:44.275983 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.269034 2550 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 23:40:44.275983 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.269037 2550 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 23:40:44.276478 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.269040 2550 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 23:40:44.276478 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.269043 2550 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 23:40:44.276478 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.269046 2550 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 23:40:44.276478 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.269049 2550 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 23:40:44.276478 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.269052 2550 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 23:40:44.276478 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.269055 2550 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 23:40:44.276478 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.269058 2550 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 23:40:44.276478 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.269060 2550 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 23:40:44.276478 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.269063 2550 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 23:40:44.276478 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.269066 2550 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 23:40:44.276478 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.269069 2550 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 23:40:44.276478 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.269074 2550 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 23:40:44.276478 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.269077 2550 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 23:40:44.276478 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.269079 2550 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 23:40:44.276478 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.269082 2550 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 23:40:44.276478 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.269085 2550 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 23:40:44.276478 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.269088 2550 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 23:40:44.276478 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.269090 2550 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 23:40:44.276478 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.269093 2550 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 23:40:44.277073 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.269096 2550 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 23:40:44.277073 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.269098 2550 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 23:40:44.277073 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.269101 2550 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 23:40:44.277073 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.269104 2550 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 23:40:44.277073 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.269107 2550 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 23:40:44.277073 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.269109 2550 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 23:40:44.277073 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.269112 2550 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 23:40:44.277073 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.269115 2550 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 23:40:44.277073 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.269118 2550 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 23:40:44.277073 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.269120 2550 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 23:40:44.277073 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.269123 2550 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 23:40:44.277073 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.269127 2550 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 23:40:44.277073 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.269129 2550 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 23:40:44.277073 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.269132 2550 feature_gate.go:386] unrecognized feature gate: Example Oct 30 23:40:44.277073 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.269135 2550 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 23:40:44.277073 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.269139 2550 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 23:40:44.277073 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.269143 2550 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 23:40:44.277073 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.269146 2550 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 23:40:44.277073 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.269149 2550 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 23:40:44.277780 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.269152 2550 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 23:40:44.277780 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.269155 2550 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 23:40:44.277780 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.269158 2550 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 23:40:44.277780 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.269160 2550 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 23:40:44.277780 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.269163 2550 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 23:40:44.277780 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.269167 2550 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 23:40:44.277780 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.269170 2550 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 23:40:44.277780 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.269173 2550 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 23:40:44.277780 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.269175 2550 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 23:40:44.277780 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.269178 2550 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 23:40:44.277780 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.269181 2550 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 23:40:44.277780 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.269184 2550 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 23:40:44.277780 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.269186 2550 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 23:40:44.277780 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.269189 2550 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 23:40:44.277780 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.269192 2550 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 23:40:44.277780 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.269194 2550 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 23:40:44.277780 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.269197 2550 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 23:40:44.277780 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.269200 2550 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 23:40:44.277780 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.269202 2550 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 23:40:44.277780 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.269205 2550 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 23:40:44.278267 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.269208 2550 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 23:40:44.278267 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.269210 2550 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 23:40:44.278267 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.269213 2550 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 23:40:44.278267 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.269215 2550 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 23:40:44.278267 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.269220 2550 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 23:40:44.278267 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.269224 2550 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 23:40:44.278267 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.269228 2550 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 23:40:44.278267 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.269231 2550 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 23:40:44.278267 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.269235 2550 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 23:40:44.278267 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.269238 2550 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 23:40:44.278267 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.269240 2550 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 23:40:44.278267 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.269243 2550 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 23:40:44.278267 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.269246 2550 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 23:40:44.278267 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.269272 2550 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 23:40:44.278267 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.269307 2550 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 23:40:44.278687 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.269311 2550 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 23:40:44.278687 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.269314 2550 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 23:40:44.278687 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.269319 2550 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 23:40:44.278687 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.269325 2550 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 23:40:44.278687 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.269340 2550 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 23:40:44.278687 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.269344 2550 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 23:40:44.278687 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.269347 2550 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 23:40:44.278687 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.269350 2550 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 23:40:44.278687 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.269353 2550 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 23:40:44.278687 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.269356 2550 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 23:40:44.278687 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.269358 2550 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 23:40:44.278687 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.269362 2550 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 23:40:44.278687 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.269364 2550 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 23:40:44.278687 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.269367 2550 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 23:40:44.278687 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.269370 2550 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 23:40:44.278687 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.269373 2550 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 23:40:44.278687 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.269376 2550 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 23:40:44.278687 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.269378 2550 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 23:40:44.278687 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.269382 2550 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 23:40:44.279138 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.269384 2550 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 23:40:44.279138 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.269387 2550 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 23:40:44.279138 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.269390 2550 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 23:40:44.279138 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.269393 2550 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 23:40:44.279138 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.269396 2550 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 23:40:44.279138 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.269399 2550 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 23:40:44.279138 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.269401 2550 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 23:40:44.279138 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.269406 2550 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 23:40:44.279138 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.269409 2550 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 23:40:44.279138 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.269411 2550 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 23:40:44.279138 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.269414 2550 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 23:40:44.279138 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.269417 2550 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 23:40:44.279138 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.269420 2550 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 23:40:44.279138 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.269422 2550 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 23:40:44.279138 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.269425 2550 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 23:40:44.279138 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.269432 2550 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 23:40:44.279138 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.269435 2550 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 23:40:44.279138 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.269438 2550 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 23:40:44.279138 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.269440 2550 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 23:40:44.279138 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.269443 2550 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 23:40:44.279630 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.269446 2550 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 23:40:44.279630 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.269449 2550 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 23:40:44.279630 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.269451 2550 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 23:40:44.279630 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.269454 2550 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 23:40:44.279630 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.269457 2550 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 23:40:44.279630 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.269461 2550 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 23:40:44.279630 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.269463 2550 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 23:40:44.279630 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.269466 2550 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 23:40:44.279630 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.269469 2550 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 23:40:44.279630 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.269471 2550 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 23:40:44.279630 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.269474 2550 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 23:40:44.279630 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.269477 2550 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 23:40:44.279630 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.269479 2550 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 23:40:44.279630 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.269482 2550 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 23:40:44.279630 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.269485 2550 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 23:40:44.279630 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.269488 2550 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 23:40:44.279630 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.269491 2550 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 23:40:44.279630 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.269494 2550 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 23:40:44.279630 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.269497 2550 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 23:40:44.280085 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.269500 2550 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 23:40:44.280085 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.269503 2550 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 23:40:44.280085 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.269506 2550 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 23:40:44.280085 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.269508 2550 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 23:40:44.280085 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.269511 2550 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 23:40:44.280085 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.269514 2550 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 23:40:44.280085 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.269516 2550 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 23:40:44.280085 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.269519 2550 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 23:40:44.280085 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.269523 2550 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 23:40:44.280085 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.269526 2550 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 23:40:44.280085 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.269529 2550 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 23:40:44.280085 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.269531 2550 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 23:40:44.280085 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.269534 2550 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 23:40:44.280085 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.269536 2550 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 23:40:44.280085 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.269539 2550 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 23:40:44.280085 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.269542 2550 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 23:40:44.280085 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.269545 2550 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 23:40:44.280085 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.269548 2550 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 23:40:44.280085 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.269550 2550 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 23:40:44.280085 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.269553 2550 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 23:40:44.280566 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.269555 2550 feature_gate.go:386] unrecognized feature gate: Example Oct 30 23:40:44.280566 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.269558 2550 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 23:40:44.280566 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.269561 2550 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 23:40:44.280566 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.269564 2550 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 23:40:44.280566 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.269568 2550 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 23:40:44.280566 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.269571 2550 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 23:40:44.280566 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.269573 2550 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 23:40:44.280566 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.269577 2550 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 23:40:44.280566 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.269579 2550 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 23:40:44.280566 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.269582 2550 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 23:40:44.280566 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.269584 2550 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 23:40:44.280566 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.269587 2550 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 23:40:44.280566 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.269591 2550 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 23:40:44.280566 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.269594 2550 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 23:40:44.280566 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.269597 2550 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 23:40:44.280566 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.269599 2550 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 23:40:44.280566 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.269602 2550 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 23:40:44.280566 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.269604 2550 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 23:40:44.280566 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.269607 2550 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 23:40:44.280566 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.269610 2550 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 23:40:44.281033 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.269613 2550 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 23:40:44.281033 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.269616 2550 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 23:40:44.281033 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.269619 2550 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 23:40:44.281033 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.269621 2550 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 23:40:44.281033 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.269624 2550 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 23:40:44.281033 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.269626 2550 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 23:40:44.281033 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.269629 2550 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 23:40:44.281033 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.276995 2550 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 23:40:44.281033 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.277016 2550 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 23:40:44.281033 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.277760 2550 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 23:40:44.281033 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.277774 2550 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 23:40:44.281033 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.278024 2550 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 23:40:44.281033 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.278032 2550 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 23:40:44.281033 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.278036 2550 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 23:40:44.281033 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.278039 2550 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 23:40:44.281033 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.278042 2550 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 23:40:44.281033 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.278045 2550 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 23:40:44.281033 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.278048 2550 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 23:40:44.281033 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.278052 2550 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 23:40:44.281033 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.278056 2550 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 23:40:44.281521 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.278059 2550 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 23:40:44.281521 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.278062 2550 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 23:40:44.281521 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.278065 2550 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 23:40:44.281521 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.278067 2550 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 23:40:44.281521 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.278070 2550 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 23:40:44.281521 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.278073 2550 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 23:40:44.281521 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.278077 2550 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 23:40:44.281521 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.278079 2550 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 23:40:44.281521 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.278082 2550 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 23:40:44.281521 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.278085 2550 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 23:40:44.281521 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.278088 2550 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 23:40:44.281521 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.278091 2550 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 23:40:44.281521 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.278093 2550 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 23:40:44.281521 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.278096 2550 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 23:40:44.281521 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.278099 2550 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 23:40:44.281521 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.278102 2550 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 23:40:44.281521 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.278105 2550 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 23:40:44.281521 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.278107 2550 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 23:40:44.281521 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.278110 2550 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 23:40:44.281976 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.278113 2550 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 23:40:44.281976 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.278122 2550 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 23:40:44.281976 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.278125 2550 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 23:40:44.281976 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.278128 2550 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 23:40:44.281976 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.278132 2550 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 23:40:44.281976 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.278135 2550 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 23:40:44.281976 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.278138 2550 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 23:40:44.281976 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.278140 2550 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 23:40:44.281976 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.278143 2550 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 23:40:44.281976 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.278146 2550 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 23:40:44.281976 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.278148 2550 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 23:40:44.281976 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.278151 2550 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 23:40:44.281976 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.278155 2550 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 23:40:44.281976 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.278160 2550 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 23:40:44.281976 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.278162 2550 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 23:40:44.281976 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.278167 2550 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 23:40:44.281976 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.278170 2550 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 23:40:44.281976 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.278176 2550 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 23:40:44.281976 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.278183 2550 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 23:40:44.282442 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.278188 2550 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 23:40:44.282442 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.278191 2550 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 23:40:44.282442 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.278194 2550 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 23:40:44.282442 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.278197 2550 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 23:40:44.282442 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.278200 2550 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 23:40:44.282442 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.278202 2550 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 23:40:44.282442 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.278205 2550 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 23:40:44.282442 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.278208 2550 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 23:40:44.282442 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.278211 2550 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 23:40:44.282442 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.278214 2550 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 23:40:44.282442 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.278217 2550 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 23:40:44.282442 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.278220 2550 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 23:40:44.282442 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.278223 2550 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 23:40:44.282442 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.278226 2550 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 23:40:44.282442 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.278229 2550 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 23:40:44.282442 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.278231 2550 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 23:40:44.282442 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.278234 2550 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 23:40:44.282442 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.278238 2550 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 23:40:44.282442 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.278241 2550 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 23:40:44.282442 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.278244 2550 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 23:40:44.282912 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.278246 2550 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 23:40:44.282912 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.278249 2550 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 23:40:44.282912 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.278252 2550 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 23:40:44.282912 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.278255 2550 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 23:40:44.282912 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.278257 2550 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 23:40:44.282912 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.278260 2550 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 23:40:44.282912 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.278262 2550 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 23:40:44.282912 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.278265 2550 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 23:40:44.282912 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.278268 2550 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 23:40:44.282912 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.278271 2550 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 23:40:44.282912 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.278274 2550 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 23:40:44.282912 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.278277 2550 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 23:40:44.282912 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.278279 2550 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 23:40:44.282912 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.278282 2550 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 23:40:44.282912 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.278285 2550 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 23:40:44.282912 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.278289 2550 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 23:40:44.282912 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.278294 2550 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 23:40:44.282912 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.278297 2550 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 23:40:44.282912 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.278300 2550 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 23:40:44.283373 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.278303 2550 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 23:40:44.283373 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.278306 2550 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 23:40:44.283373 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.278308 2550 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 23:40:44.283373 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.278311 2550 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 23:40:44.283373 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.278314 2550 feature_gate.go:386] unrecognized feature gate: Example Oct 30 23:40:44.283373 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.278317 2550 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 23:40:44.283373 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.278319 2550 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 23:40:44.283373 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.278322 2550 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 23:40:44.283373 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.278324 2550 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 23:40:44.283373 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.278340 2550 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 23:40:44.283373 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.278344 2550 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 23:40:44.283373 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.278347 2550 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 23:40:44.283373 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.278350 2550 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 23:40:44.283373 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.278352 2550 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 23:40:44.283373 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.278355 2550 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 23:40:44.283373 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.278358 2550 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 23:40:44.283373 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.278360 2550 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 23:40:44.283373 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.278363 2550 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 23:40:44.283373 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.278366 2550 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 23:40:44.283373 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.278368 2550 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 23:40:44.283866 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.278400 2550 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 23:40:44.283866 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.278560 2550 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 23:40:44.283866 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.278566 2550 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 23:40:44.283866 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.278569 2550 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 23:40:44.283866 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.278571 2550 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 23:40:44.283866 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.278574 2550 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 23:40:44.283866 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.278582 2550 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 23:40:44.283866 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.278585 2550 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 23:40:44.283866 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.278588 2550 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 23:40:44.283866 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.278591 2550 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 23:40:44.283866 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.278593 2550 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 23:40:44.283866 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.278596 2550 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 23:40:44.283866 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.278599 2550 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 23:40:44.283866 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.278601 2550 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 23:40:44.283866 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.278604 2550 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 23:40:44.283866 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.278607 2550 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 23:40:44.284243 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.278609 2550 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 23:40:44.284243 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.278612 2550 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 23:40:44.284243 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.278615 2550 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 23:40:44.284243 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.278618 2550 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 23:40:44.284243 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.278620 2550 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 23:40:44.284243 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.278623 2550 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 23:40:44.284243 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.278626 2550 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 23:40:44.284243 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.278629 2550 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 23:40:44.284243 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.278631 2550 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 23:40:44.284243 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.278635 2550 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 23:40:44.284243 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.278640 2550 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 23:40:44.284243 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.278643 2550 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 23:40:44.284243 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.278646 2550 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 23:40:44.284243 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.278648 2550 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 23:40:44.284243 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.278651 2550 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 23:40:44.284243 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.278654 2550 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 23:40:44.284243 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.278657 2550 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 23:40:44.284243 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.278659 2550 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 23:40:44.284243 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.278662 2550 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 23:40:44.284701 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.278665 2550 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 23:40:44.284701 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.278667 2550 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 23:40:44.284701 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.278670 2550 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 23:40:44.284701 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.278673 2550 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 23:40:44.284701 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.278680 2550 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 23:40:44.284701 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.278683 2550 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 23:40:44.284701 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.278685 2550 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 23:40:44.284701 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.278688 2550 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 23:40:44.284701 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.278691 2550 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 23:40:44.284701 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.278694 2550 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 23:40:44.284701 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.278696 2550 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 23:40:44.284701 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.278699 2550 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 23:40:44.284701 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.278702 2550 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 23:40:44.284701 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.278705 2550 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 23:40:44.284701 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.278708 2550 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 23:40:44.284701 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.278710 2550 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 23:40:44.284701 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.278713 2550 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 23:40:44.284701 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.278716 2550 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 23:40:44.284701 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.278719 2550 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 23:40:44.284701 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.278722 2550 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 23:40:44.285165 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.278724 2550 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 23:40:44.285165 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.278727 2550 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 23:40:44.285165 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.278729 2550 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 23:40:44.285165 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.278732 2550 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 23:40:44.285165 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.278735 2550 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 23:40:44.285165 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.278737 2550 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 23:40:44.285165 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.278740 2550 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 23:40:44.285165 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.278742 2550 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 23:40:44.285165 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.278745 2550 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 23:40:44.285165 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.278748 2550 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 23:40:44.285165 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.278751 2550 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 23:40:44.285165 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.278753 2550 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 23:40:44.285165 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.278756 2550 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 23:40:44.285165 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.278759 2550 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 23:40:44.285165 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.278761 2550 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 23:40:44.285165 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.278764 2550 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 23:40:44.285165 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.278766 2550 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 23:40:44.285165 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.278775 2550 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 23:40:44.285165 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.278779 2550 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 23:40:44.285705 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.278781 2550 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 23:40:44.285705 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.278784 2550 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 23:40:44.285705 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.278786 2550 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 23:40:44.285705 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.278789 2550 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 23:40:44.285705 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.278791 2550 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 23:40:44.285705 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.278794 2550 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 23:40:44.285705 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.278797 2550 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 23:40:44.285705 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.278799 2550 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 23:40:44.285705 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.278802 2550 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 23:40:44.285705 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.278805 2550 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 23:40:44.285705 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.278808 2550 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 23:40:44.285705 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.278812 2550 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 23:40:44.285705 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.278815 2550 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 23:40:44.285705 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.278818 2550 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 23:40:44.285705 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.278821 2550 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 23:40:44.285705 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.278824 2550 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 23:40:44.285705 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.278827 2550 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 23:40:44.285705 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.278830 2550 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 23:40:44.285705 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.278832 2550 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 23:40:44.286155 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.278835 2550 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 23:40:44.286155 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.278838 2550 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 23:40:44.286155 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.278840 2550 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 23:40:44.286155 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.278843 2550 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 23:40:44.286155 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.278846 2550 feature_gate.go:386] unrecognized feature gate: Example Oct 30 23:40:44.286155 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.278848 2550 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 23:40:44.286155 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.278851 2550 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 23:40:44.286155 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.278854 2550 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 23:40:44.286155 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.278856 2550 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 23:40:44.286155 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.278859 2550 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 23:40:44.286155 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.278862 2550 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 23:40:44.286155 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.278864 2550 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 23:40:44.286155 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.278872 2550 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 23:40:44.286155 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.278875 2550 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 23:40:44.286155 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.278898 2550 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 23:40:44.286155 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.278934 2550 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 23:40:44.286541 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.278938 2550 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 23:40:44.286541 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.278941 2550 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 23:40:44.286541 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.278943 2550 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 23:40:44.286541 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.278947 2550 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 23:40:44.286541 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.278951 2550 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 23:40:44.286541 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.278954 2550 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 23:40:44.286541 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.278957 2550 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 23:40:44.286541 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.278960 2550 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 23:40:44.286541 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.278963 2550 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 23:40:44.286541 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.278965 2550 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 23:40:44.286541 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.278968 2550 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 23:40:44.286541 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.278971 2550 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 23:40:44.286541 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.278974 2550 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 23:40:44.286541 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.278976 2550 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 23:40:44.286541 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.278979 2550 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 23:40:44.286541 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.278982 2550 feature_gate.go:386] unrecognized feature gate: Example Oct 30 23:40:44.286541 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.278984 2550 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 23:40:44.286541 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.278987 2550 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 23:40:44.286541 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.278990 2550 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 23:40:44.286541 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.278992 2550 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 23:40:44.287009 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.278995 2550 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 23:40:44.287009 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.278998 2550 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 23:40:44.287009 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.279000 2550 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 23:40:44.287009 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.279003 2550 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 23:40:44.287009 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.279005 2550 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 23:40:44.287009 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.279008 2550 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 23:40:44.287009 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.279011 2550 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 23:40:44.287009 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.279013 2550 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 23:40:44.287009 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.279016 2550 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 23:40:44.287009 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.279024 2550 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 23:40:44.287009 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.279027 2550 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 23:40:44.287009 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.279029 2550 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 23:40:44.287009 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.279032 2550 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 23:40:44.287009 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.279035 2550 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 23:40:44.287009 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.279037 2550 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 23:40:44.287009 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.279040 2550 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 23:40:44.287009 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.279042 2550 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 23:40:44.287009 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.279045 2550 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 23:40:44.287009 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.279048 2550 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 23:40:44.287009 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.279050 2550 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 23:40:44.287637 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.279053 2550 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 23:40:44.287637 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.279056 2550 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 23:40:44.287637 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.279058 2550 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 23:40:44.287637 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.279061 2550 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 23:40:44.287637 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.279064 2550 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 23:40:44.287637 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.279066 2550 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 23:40:44.287637 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.279069 2550 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 23:40:44.287637 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.279072 2550 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 23:40:44.287637 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.279074 2550 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 23:40:44.287637 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.279078 2550 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 23:40:44.287637 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.279082 2550 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 23:40:44.287637 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.279084 2550 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 23:40:44.287637 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.279087 2550 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 23:40:44.287637 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.279090 2550 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 23:40:44.287637 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.279093 2550 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 23:40:44.287637 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.279095 2550 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 23:40:44.287637 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.279098 2550 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 23:40:44.287637 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.279101 2550 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 23:40:44.287637 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.279104 2550 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 23:40:44.288089 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.279106 2550 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 23:40:44.288089 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.279109 2550 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 23:40:44.288089 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.279112 2550 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 23:40:44.288089 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.279119 2550 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 23:40:44.288089 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.279125 2550 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 23:40:44.288089 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.279127 2550 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 23:40:44.288089 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.279130 2550 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 23:40:44.288089 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.279133 2550 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 23:40:44.288089 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.279136 2550 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 23:40:44.288089 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.279139 2550 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 23:40:44.288089 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.279141 2550 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 23:40:44.288089 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.279144 2550 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 23:40:44.288089 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.279147 2550 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 23:40:44.288089 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.279150 2550 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 23:40:44.288089 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.279153 2550 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 23:40:44.288089 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.279156 2550 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 23:40:44.288089 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.279159 2550 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 23:40:44.288089 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.279161 2550 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 23:40:44.288089 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.279164 2550 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 23:40:44.288089 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.279166 2550 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 23:40:44.288577 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.279169 2550 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 23:40:44.288577 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.279172 2550 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 23:40:44.288577 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.279175 2550 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 23:40:44.288577 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.279177 2550 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 23:40:44.288577 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.279180 2550 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 23:40:44.288577 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.279182 2550 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 23:40:44.288577 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.279185 2550 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 23:40:44.288577 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.279189 2550 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 23:40:44.288577 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.279192 2550 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 23:40:44.288577 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.279195 2550 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 23:40:44.288577 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.279198 2550 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 23:40:44.288577 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.279200 2550 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 23:40:44.288577 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.279204 2550 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 23:40:44.288577 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.279207 2550 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 23:40:44.288577 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.279209 2550 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 23:40:44.288577 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.279212 2550 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 23:40:44.288577 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.279220 2550 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 23:40:44.288577 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.279223 2550 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 23:40:44.288577 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.279226 2550 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 23:40:44.289027 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.279228 2550 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 23:40:44.289027 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.279231 2550 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 23:40:44.289027 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.279234 2550 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 23:40:44.289027 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.279236 2550 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 23:40:44.289027 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.279239 2550 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 23:40:44.289027 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.279241 2550 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 23:40:44.289027 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:44.279244 2550 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 23:40:44.289027 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.279324 2550 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 23:40:44.289027 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.281774 2550 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Oct 30 23:40:44.289027 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.282863 2550 server.go:1014] "Starting client certificate rotation" Oct 30 23:40:44.289027 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.282956 2550 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 23:40:44.289027 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.282994 2550 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 23:40:44.289027 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.285218 2550 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 23:40:44.289027 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.286361 2550 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 23:40:44.294033 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.294018 2550 log.go:25] "Validated CRI v1 runtime API" Oct 30 23:40:44.298047 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.298032 2550 log.go:25] "Validated CRI v1 image API" Oct 30 23:40:44.298967 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.298942 2550 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 23:40:44.302574 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.302553 2550 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 9dfc8826-3266-4e14-bd7f-ea674cb91201:/dev/nvme0n1p4 c78ae7d5-3ca7-44fa-b0a6-cd1ea243dc19:/dev/nvme0n1p3] Oct 30 23:40:44.302646 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.302574 2550 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Oct 30 23:40:44.309007 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.308887 2550 manager.go:217] Machine: {Timestamp:2026-10-30 23:40:44.307548116 +0000 UTC m=+0.365752023 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100238 MemoryCapacity:33164337152 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2acb4b4090c41d8b0bad096078018e SystemUUID:ec2acb4b-4090-c41d-8b0b-ad096078018e BootID:70c52820-3dd5-4ff4-ab01-ce728104e9a7 Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:06:ff:d9:eb:dd:69 Speed:0 Mtu:9001} {Name:ens5 MacAddress:06:ff:d9:eb:dd:69 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:56:a4:ef:fa:f3:68 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164337152 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 23:40:44.309007 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.309000 2550 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 23:40:44.309171 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.309110 2550 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 23:40:44.312727 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.312699 2550 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 23:40:44.312862 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.312730 2550 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-141-180.us-west-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 23:40:44.312904 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.312864 2550 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 23:40:44.312904 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.312872 2550 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 23:40:44.312904 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.312882 2550 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 23:40:44.313831 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.313820 2550 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 23:40:44.315091 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.315080 2550 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 23:40:44.319975 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.319960 2550 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 23:40:44.320086 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.320077 2550 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 23:40:44.320929 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.320919 2550 kubelet.go:493] "Attempting to sync node with API server" Oct 30 23:40:44.320962 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.320933 2550 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 23:40:44.320962 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.320944 2550 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 23:40:44.320962 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.320953 2550 kubelet.go:405] "Adding apiserver pod source" Oct 30 23:40:44.320962 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.320961 2550 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 23:40:44.321293 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.321282 2550 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 23:40:44.323871 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.323853 2550 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 23:40:44.331135 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.331121 2550 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 23:40:44.332086 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.332073 2550 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 23:40:44.332124 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.332104 2550 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 23:40:44.332255 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.332243 2550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 23:40:44.332293 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.332259 2550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 23:40:44.332293 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.332266 2550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 23:40:44.332293 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.332271 2550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 23:40:44.332293 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.332277 2550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 23:40:44.332293 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.332283 2550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 23:40:44.332979 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.332971 2550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 23:40:44.333013 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.332980 2550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 23:40:44.333013 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.332989 2550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 23:40:44.333013 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.332995 2550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 23:40:44.333013 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.333004 2550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 23:40:44.333013 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.333014 2550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 23:40:44.333868 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.333859 2550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 23:40:44.333868 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.333869 2550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 23:40:44.337773 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.337759 2550 server.go:1263] "Started kubelet" Oct 30 23:40:44.337902 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.337863 2550 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 23:40:44.337964 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.337891 2550 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 23:40:44.338013 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.337974 2550 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 23:40:44.338639 ip-10-0-141-180 systemd[1]: Started Kubernetes Kubelet. Oct 30 23:40:44.339405 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.339380 2550 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 23:40:44.341005 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.340990 2550 server.go:317] "Adding debug handlers to kubelet server" Oct 30 23:40:44.344913 ip-10-0-141-180 kubenswrapper[2550]: E1030 23:40:44.344890 2550 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Oct 30 23:40:44.348021 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.348005 2550 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 23:40:44.348713 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.348695 2550 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 23:40:44.348871 ip-10-0-141-180 kubenswrapper[2550]: E1030 23:40:44.348806 2550 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-180.us-west-2.compute.internal\" not found" Oct 30 23:40:44.349502 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.349467 2550 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 23:40:44.349502 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.349469 2550 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 23:40:44.349502 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.349500 2550 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 23:40:44.349678 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.349583 2550 reconstruct.go:97] "Volume reconstruction finished" Oct 30 23:40:44.349678 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.349593 2550 reconciler.go:29] "Reconciler: start to sync state" Oct 30 23:40:44.349678 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.349616 2550 factory.go:55] Registering systemd factory Oct 30 23:40:44.349678 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.349628 2550 factory.go:223] Registration of the systemd container factory successfully Oct 30 23:40:44.349862 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.349848 2550 factory.go:153] Registering CRI-O factory Oct 30 23:40:44.349862 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.349862 2550 factory.go:223] Registration of the crio container factory successfully Oct 30 23:40:44.349971 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.349913 2550 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 23:40:44.349971 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.349927 2550 factory.go:103] Registering Raw factory Oct 30 23:40:44.349971 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.349937 2550 manager.go:1199] Started watching for new ooms in manager Oct 30 23:40:44.351002 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.350988 2550 manager.go:319] Starting recovery of all containers Oct 30 23:40:44.363027 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.363005 2550 manager.go:324] Recovery completed Oct 30 23:40:44.368444 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.368429 2550 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 23:40:44.370619 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.370595 2550 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-180.us-west-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 23:40:44.370673 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.370629 2550 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-180.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 23:40:44.370673 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.370645 2550 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-180.us-west-2.compute.internal" event="NodeHasSufficientPID" Oct 30 23:40:44.371216 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.371200 2550 cpu_manager.go:226] "Starting" policy="none" Oct 30 23:40:44.371216 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.371207 2550 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 23:40:44.371216 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.371212 2550 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 23:40:44.371414 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.371235 2550 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 23:40:44.372066 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.372049 2550 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-180.us-west-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 23:40:44.372130 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.372076 2550 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-180.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 23:40:44.372130 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.372090 2550 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-180.us-west-2.compute.internal" event="NodeHasSufficientPID" Oct 30 23:40:44.373487 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.373469 2550 policy_none.go:50] "Start" Oct 30 23:40:44.373487 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.373486 2550 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 23:40:44.373632 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.373499 2550 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 23:40:44.375647 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.375632 2550 policy_none.go:44] "Start" Oct 30 23:40:44.431539 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.411855 2550 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 23:40:44.431539 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.414305 2550 manager.go:341] "Starting Device Plugin manager" Oct 30 23:40:44.431539 ip-10-0-141-180 kubenswrapper[2550]: E1030 23:40:44.414996 2550 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 23:40:44.431539 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.415020 2550 server.go:87] "Starting device plugin registration server" Oct 30 23:40:44.431539 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.415315 2550 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 23:40:44.431539 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.415348 2550 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 23:40:44.431539 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.415440 2550 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 23:40:44.431539 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.415526 2550 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 23:40:44.431539 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.415535 2550 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 23:40:44.431539 ip-10-0-141-180 kubenswrapper[2550]: E1030 23:40:44.416096 2550 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 23:40:44.431539 ip-10-0-141-180 kubenswrapper[2550]: E1030 23:40:44.416141 2550 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-141-180.us-west-2.compute.internal\" not found" Oct 30 23:40:44.431539 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.420846 2550 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 23:40:44.431539 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.421963 2550 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 23:40:44.431539 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.422013 2550 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 23:40:44.431539 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.422036 2550 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 23:40:44.431539 ip-10-0-141-180 kubenswrapper[2550]: E1030 23:40:44.422076 2550 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Oct 30 23:40:44.490789 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.490749 2550 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-141-180.us-west-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Oct 30 23:40:44.490789 ip-10-0-141-180 kubenswrapper[2550]: E1030 23:40:44.490748 2550 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-141-180.us-west-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Oct 30 23:40:44.491942 ip-10-0-141-180 kubenswrapper[2550]: E1030 23:40:44.490743 2550 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-180.us-west-2.compute.internal.18e3731cd7a44165 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-180.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-141-180.us-west-2.compute.internal,},FirstTimestamp:2026-10-30 23:40:44.337733989 +0000 UTC m=+0.395937899,LastTimestamp:2026-10-30 23:40:44.337733989 +0000 UTC m=+0.395937899,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-180.us-west-2.compute.internal,}" Oct 30 23:40:44.516249 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.516195 2550 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 23:40:44.517117 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.517100 2550 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-180.us-west-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 23:40:44.517186 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.517132 2550 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-180.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 23:40:44.517186 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.517147 2550 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-180.us-west-2.compute.internal" event="NodeHasSufficientPID" Oct 30 23:40:44.517252 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.517188 2550 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-141-180.us-west-2.compute.internal" Oct 30 23:40:44.523357 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.523325 2550 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-180.us-west-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-141-180.us-west-2.compute.internal"] Oct 30 23:40:44.523420 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.523409 2550 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 23:40:44.524273 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.524253 2550 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-180.us-west-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 23:40:44.524341 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.524287 2550 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-180.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 23:40:44.524341 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.524301 2550 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-180.us-west-2.compute.internal" event="NodeHasSufficientPID" Oct 30 23:40:44.526581 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.526569 2550 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 23:40:44.526754 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.526742 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-180.us-west-2.compute.internal" Oct 30 23:40:44.526786 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.526768 2550 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 23:40:44.527232 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.527216 2550 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-180.us-west-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 23:40:44.527284 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.527240 2550 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-180.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 23:40:44.527284 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.527240 2550 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-180.us-west-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 23:40:44.527284 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.527249 2550 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-180.us-west-2.compute.internal" event="NodeHasSufficientPID" Oct 30 23:40:44.527284 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.527263 2550 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-180.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 23:40:44.527284 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.527275 2550 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-180.us-west-2.compute.internal" event="NodeHasSufficientPID" Oct 30 23:40:44.529479 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.529455 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-180.us-west-2.compute.internal" Oct 30 23:40:44.529479 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.529479 2550 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 23:40:44.530275 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.530258 2550 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-180.us-west-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 23:40:44.530357 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.530285 2550 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-180.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 23:40:44.530357 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.530297 2550 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-180.us-west-2.compute.internal" event="NodeHasSufficientPID" Oct 30 23:40:44.551583 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.551409 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/fd028af9af4f41b9772b02bdb5343dc5-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-180.us-west-2.compute.internal\" (UID: \"fd028af9af4f41b9772b02bdb5343dc5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-180.us-west-2.compute.internal" Oct 30 23:40:44.551685 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.551600 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fd028af9af4f41b9772b02bdb5343dc5-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-180.us-west-2.compute.internal\" (UID: \"fd028af9af4f41b9772b02bdb5343dc5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-180.us-west-2.compute.internal" Oct 30 23:40:44.551685 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.551625 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2c1c7fe71dc5ea64e6fbc8dd666819fa-config\") pod \"kube-apiserver-proxy-ip-10-0-141-180.us-west-2.compute.internal\" (UID: \"2c1c7fe71dc5ea64e6fbc8dd666819fa\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-180.us-west-2.compute.internal" Oct 30 23:40:44.566411 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.566389 2550 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 23:40:44.566479 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.566445 2550 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-lzwzq" Oct 30 23:40:44.567759 ip-10-0-141-180 kubenswrapper[2550]: E1030 23:40:44.567742 2550 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-180.us-west-2.compute.internal\" not found" node="ip-10-0-141-180.us-west-2.compute.internal" Oct 30 23:40:44.569809 ip-10-0-141-180 kubenswrapper[2550]: E1030 23:40:44.569791 2550 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-180.us-west-2.compute.internal\" not found" node="ip-10-0-141-180.us-west-2.compute.internal" Oct 30 23:40:44.577421 ip-10-0-141-180 kubenswrapper[2550]: E1030 23:40:44.577355 2550 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-180.us-west-2.compute.internal.18e3731cd999ff69 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-180.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-141-180.us-west-2.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-141-180.us-west-2.compute.internal,},FirstTimestamp:2026-10-30 23:40:44.370616169 +0000 UTC m=+0.428820075,LastTimestamp:2026-10-30 23:40:44.370616169 +0000 UTC m=+0.428820075,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-180.us-west-2.compute.internal,}" Oct 30 23:40:44.577914 ip-10-0-141-180 kubenswrapper[2550]: E1030 23:40:44.577884 2550 reflector.go:204] "Failed to watch" err="failed to list *v1.RuntimeClass: runtimeclasses.node.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"runtimeclasses\" in API group \"node.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" Oct 30 23:40:44.577972 ip-10-0-141-180 kubenswrapper[2550]: E1030 23:40:44.577887 2550 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" Oct 30 23:40:44.643102 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.643074 2550 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 23:40:44.643292 ip-10-0-141-180 kubenswrapper[2550]: E1030 23:40:44.643273 2550 kubelet_node_status.go:115] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-141-180.us-west-2.compute.internal\" not found" node="ip-10-0-141-180.us-west-2.compute.internal" Oct 30 23:40:44.644630 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.644614 2550 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 23:40:44.651968 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.651950 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/fd028af9af4f41b9772b02bdb5343dc5-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-180.us-west-2.compute.internal\" (UID: \"fd028af9af4f41b9772b02bdb5343dc5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-180.us-west-2.compute.internal" Oct 30 23:40:44.652058 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.651981 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fd028af9af4f41b9772b02bdb5343dc5-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-180.us-west-2.compute.internal\" (UID: \"fd028af9af4f41b9772b02bdb5343dc5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-180.us-west-2.compute.internal" Oct 30 23:40:44.652058 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.651981 2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/fd028af9af4f41b9772b02bdb5343dc5-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-180.us-west-2.compute.internal\" (UID: \"fd028af9af4f41b9772b02bdb5343dc5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-180.us-west-2.compute.internal" Oct 30 23:40:44.652058 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.651999 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2c1c7fe71dc5ea64e6fbc8dd666819fa-config\") pod \"kube-apiserver-proxy-ip-10-0-141-180.us-west-2.compute.internal\" (UID: \"2c1c7fe71dc5ea64e6fbc8dd666819fa\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-180.us-west-2.compute.internal" Oct 30 23:40:44.652058 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.652036 2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2c1c7fe71dc5ea64e6fbc8dd666819fa-config\") pod \"kube-apiserver-proxy-ip-10-0-141-180.us-west-2.compute.internal\" (UID: \"2c1c7fe71dc5ea64e6fbc8dd666819fa\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-180.us-west-2.compute.internal" Oct 30 23:40:44.652230 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.652055 2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fd028af9af4f41b9772b02bdb5343dc5-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-180.us-west-2.compute.internal\" (UID: \"fd028af9af4f41b9772b02bdb5343dc5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-180.us-west-2.compute.internal" Oct 30 23:40:44.788648 ip-10-0-141-180 kubenswrapper[2550]: E1030 23:40:44.788573 2550 nodelease.go:50] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-141-180.us-west-2.compute.internal\" not found" node="ip-10-0-141-180.us-west-2.compute.internal" Oct 30 23:40:44.844393 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.844368 2550 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 23:40:44.845385 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.845367 2550 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-180.us-west-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 23:40:44.845481 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.845396 2550 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-180.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 23:40:44.845481 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.845414 2550 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-180.us-west-2.compute.internal" event="NodeHasSufficientPID" Oct 30 23:40:44.845481 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.845447 2550 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-141-180.us-west-2.compute.internal" Oct 30 23:40:44.872685 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.872669 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-180.us-west-2.compute.internal" Oct 30 23:40:44.874784 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.874771 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-180.us-west-2.compute.internal" Oct 30 23:40:44.898104 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.898083 2550 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-141-180.us-west-2.compute.internal" Oct 30 23:40:44.898104 ip-10-0-141-180 kubenswrapper[2550]: E1030 23:40:44.898104 2550 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-141-180.us-west-2.compute.internal\": node \"ip-10-0-141-180.us-west-2.compute.internal\" not found" Oct 30 23:40:44.949247 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:44.949229 2550 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-180.us-west-2.compute.internal" Oct 30 23:40:45.005269 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.005240 2550 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 23:40:45.006482 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.006467 2550 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-180.us-west-2.compute.internal" Oct 30 23:40:45.119505 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.119488 2550 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 23:40:45.283916 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.283890 2550 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Oct 30 23:40:45.284459 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.284060 2550 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 30 23:40:45.284459 ip-10-0-141-180 kubenswrapper[2550]: E1030 23:40:45.284133 2550 event.go:368] "Unable to write event (may retry after sleeping)" err="Patch \"https://ade377f6ef8974a73ad19fd7b7959d61-5b50151745721f56.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/default/events/ip-10-0-141-180.us-west-2.compute.internal.18e3731cd99a4e9b\": read tcp 10.0.141.180:56142->3.143.20.58:6443: use of closed network connection" event="&Event{ObjectMeta:{ip-10-0-141-180.us-west-2.compute.internal.18e3731cd99a4e9b default 4826 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-180.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-141-180.us-west-2.compute.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-141-180.us-west-2.compute.internal,},FirstTimestamp:2026-10-30 23:40:44 +0000 UTC,LastTimestamp:2026-10-30 23:40:44.524293584 +0000 UTC m=+0.582497491,Count:4,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-180.us-west-2.compute.internal,}" Oct 30 23:40:45.321697 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.321673 2550 apiserver.go:52] "Watching apiserver" Oct 30 23:40:45.349041 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.349023 2550 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Oct 30 23:40:45.573511 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.573431 2550 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 23:35:45 +0000 UTC" deadline="2028-04-06 07:16:44.700857179 +0000 UTC" Oct 30 23:40:45.573511 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.573459 2550 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12559h35m59.127400893s" Oct 30 23:40:45.636172 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.636141 2550 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 23:40:45.636491 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.636471 2550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/node-resolver-tfnr7","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-180.us-west-2.compute.internal","openshift-multus/multus-cb9vj","kube-system/kube-apiserver-proxy-ip-10-0-141-180.us-west-2.compute.internal","openshift-multus/network-metrics-daemon-pt58s","openshift-network-diagnostics/network-check-target-kgdzs","openshift-network-operator/iptables-alerter-57xrk","kube-system/global-pull-secret-syncer-7wrh7","kube-system/konnectivity-agent-9s2fx","openshift-image-registry/node-ca-7sbzf","openshift-multus/multus-additional-cni-plugins-7mp5r","openshift-ovn-kubernetes/ovnkube-node-26v6x","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j7n2l","openshift-cluster-node-tuning-operator/tuned-rsv7z"] Oct 30 23:40:45.639072 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.639050 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-57xrk" Oct 30 23:40:45.641233 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.641213 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j7n2l" Oct 30 23:40:45.643480 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.643466 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-cb9vj" Oct 30 23:40:45.645644 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.645630 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pt58s" Oct 30 23:40:45.645705 ip-10-0-141-180 kubenswrapper[2550]: E1030 23:40:45.645689 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pt58s" podUID="145dd323-9370-4b9d-a96e-8b93d0a47a7c" Oct 30 23:40:45.647803 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.647784 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kgdzs" Oct 30 23:40:45.648196 ip-10-0-141-180 kubenswrapper[2550]: E1030 23:40:45.648162 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kgdzs" podUID="7728abfe-1747-422e-b8e2-c91c9de83d09" Oct 30 23:40:45.650519 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.650496 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-9s2fx" Oct 30 23:40:45.652897 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.652882 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-tfnr7" Oct 30 23:40:45.655224 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.655204 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-rsv7z" Oct 30 23:40:45.657736 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.657711 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-7mp5r" Oct 30 23:40:45.659182 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.658901 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2fa7d24b-67b0-400f-94d4-903feb268782-multus-socket-dir-parent\") pod \"multus-cb9vj\" (UID: \"2fa7d24b-67b0-400f-94d4-903feb268782\") " pod="openshift-multus/multus-cb9vj" Oct 30 23:40:45.659182 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.658934 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ece07686-4d8e-4068-9fb2-5889046cb73b-host-slash\") pod \"iptables-alerter-57xrk\" (UID: \"ece07686-4d8e-4068-9fb2-5889046cb73b\") " pod="openshift-network-operator/iptables-alerter-57xrk" Oct 30 23:40:45.659182 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.658959 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0796261f-298e-4cbe-9b3e-b3dcf4528e07-sys-fs\") pod \"aws-ebs-csi-driver-node-j7n2l\" (UID: \"0796261f-298e-4cbe-9b3e-b3dcf4528e07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j7n2l" Oct 30 23:40:45.659182 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.658995 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2fa7d24b-67b0-400f-94d4-903feb268782-system-cni-dir\") pod \"multus-cb9vj\" (UID: \"2fa7d24b-67b0-400f-94d4-903feb268782\") " pod="openshift-multus/multus-cb9vj" Oct 30 23:40:45.659182 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.659019 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2fa7d24b-67b0-400f-94d4-903feb268782-multus-cni-dir\") pod \"multus-cb9vj\" (UID: \"2fa7d24b-67b0-400f-94d4-903feb268782\") " pod="openshift-multus/multus-cb9vj" Oct 30 23:40:45.659182 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.659046 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0796261f-298e-4cbe-9b3e-b3dcf4528e07-etc-selinux\") pod \"aws-ebs-csi-driver-node-j7n2l\" (UID: \"0796261f-298e-4cbe-9b3e-b3dcf4528e07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j7n2l" Oct 30 23:40:45.659182 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.659094 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a8e4c4db-197e-420c-9c60-4273cdf95377-agent-certs\") pod \"konnectivity-agent-9s2fx\" (UID: \"a8e4c4db-197e-420c-9c60-4273cdf95377\") " pod="kube-system/konnectivity-agent-9s2fx" Oct 30 23:40:45.659182 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.659135 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a8e4c4db-197e-420c-9c60-4273cdf95377-konnectivity-ca\") pod \"konnectivity-agent-9s2fx\" (UID: \"a8e4c4db-197e-420c-9c60-4273cdf95377\") " pod="kube-system/konnectivity-agent-9s2fx" Oct 30 23:40:45.659182 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.659168 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2fa7d24b-67b0-400f-94d4-903feb268782-cnibin\") pod \"multus-cb9vj\" (UID: \"2fa7d24b-67b0-400f-94d4-903feb268782\") " pod="openshift-multus/multus-cb9vj" Oct 30 23:40:45.659709 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.659196 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2fa7d24b-67b0-400f-94d4-903feb268782-host-run-k8s-cni-cncf-io\") pod \"multus-cb9vj\" (UID: \"2fa7d24b-67b0-400f-94d4-903feb268782\") " pod="openshift-multus/multus-cb9vj" Oct 30 23:40:45.659709 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.659232 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2fa7d24b-67b0-400f-94d4-903feb268782-hostroot\") pod \"multus-cb9vj\" (UID: \"2fa7d24b-67b0-400f-94d4-903feb268782\") " pod="openshift-multus/multus-cb9vj" Oct 30 23:40:45.659709 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.659271 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ece07686-4d8e-4068-9fb2-5889046cb73b-iptables-alerter-script\") pod \"iptables-alerter-57xrk\" (UID: \"ece07686-4d8e-4068-9fb2-5889046cb73b\") " pod="openshift-network-operator/iptables-alerter-57xrk" Oct 30 23:40:45.659709 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.659301 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0796261f-298e-4cbe-9b3e-b3dcf4528e07-kubelet-dir\") pod \"aws-ebs-csi-driver-node-j7n2l\" (UID: \"0796261f-298e-4cbe-9b3e-b3dcf4528e07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j7n2l" Oct 30 23:40:45.659709 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.659325 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-25rtq\" (UniqueName: \"kubernetes.io/projected/0796261f-298e-4cbe-9b3e-b3dcf4528e07-kube-api-access-25rtq\") pod \"aws-ebs-csi-driver-node-j7n2l\" (UID: \"0796261f-298e-4cbe-9b3e-b3dcf4528e07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j7n2l" Oct 30 23:40:45.659709 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.659375 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2fa7d24b-67b0-400f-94d4-903feb268782-host-var-lib-cni-multus\") pod \"multus-cb9vj\" (UID: \"2fa7d24b-67b0-400f-94d4-903feb268782\") " pod="openshift-multus/multus-cb9vj" Oct 30 23:40:45.659709 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.659401 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2fa7d24b-67b0-400f-94d4-903feb268782-multus-conf-dir\") pod \"multus-cb9vj\" (UID: \"2fa7d24b-67b0-400f-94d4-903feb268782\") " pod="openshift-multus/multus-cb9vj" Oct 30 23:40:45.659709 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.659441 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2fa7d24b-67b0-400f-94d4-903feb268782-host-run-multus-certs\") pod \"multus-cb9vj\" (UID: \"2fa7d24b-67b0-400f-94d4-903feb268782\") " pod="openshift-multus/multus-cb9vj" Oct 30 23:40:45.659709 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.659468 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/145dd323-9370-4b9d-a96e-8b93d0a47a7c-metrics-certs\") pod \"network-metrics-daemon-pt58s\" (UID: \"145dd323-9370-4b9d-a96e-8b93d0a47a7c\") " pod="openshift-multus/network-metrics-daemon-pt58s" Oct 30 23:40:45.659709 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.659494 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nsggg\" (UniqueName: \"kubernetes.io/projected/145dd323-9370-4b9d-a96e-8b93d0a47a7c-kube-api-access-nsggg\") pod \"network-metrics-daemon-pt58s\" (UID: \"145dd323-9370-4b9d-a96e-8b93d0a47a7c\") " pod="openshift-multus/network-metrics-daemon-pt58s" Oct 30 23:40:45.659709 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.659536 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0796261f-298e-4cbe-9b3e-b3dcf4528e07-socket-dir\") pod \"aws-ebs-csi-driver-node-j7n2l\" (UID: \"0796261f-298e-4cbe-9b3e-b3dcf4528e07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j7n2l" Oct 30 23:40:45.659709 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.659567 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0796261f-298e-4cbe-9b3e-b3dcf4528e07-device-dir\") pod \"aws-ebs-csi-driver-node-j7n2l\" (UID: \"0796261f-298e-4cbe-9b3e-b3dcf4528e07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j7n2l" Oct 30 23:40:45.659709 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.659646 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2fa7d24b-67b0-400f-94d4-903feb268782-host-run-netns\") pod \"multus-cb9vj\" (UID: \"2fa7d24b-67b0-400f-94d4-903feb268782\") " pod="openshift-multus/multus-cb9vj" Oct 30 23:40:45.659709 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.659675 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2fa7d24b-67b0-400f-94d4-903feb268782-multus-daemon-config\") pod \"multus-cb9vj\" (UID: \"2fa7d24b-67b0-400f-94d4-903feb268782\") " pod="openshift-multus/multus-cb9vj" Oct 30 23:40:45.659709 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.659709 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2fa7d24b-67b0-400f-94d4-903feb268782-cni-binary-copy\") pod \"multus-cb9vj\" (UID: \"2fa7d24b-67b0-400f-94d4-903feb268782\") " pod="openshift-multus/multus-cb9vj" Oct 30 23:40:45.660204 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.659740 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqmnp\" (UniqueName: \"kubernetes.io/projected/7728abfe-1747-422e-b8e2-c91c9de83d09-kube-api-access-jqmnp\") pod \"network-check-target-kgdzs\" (UID: \"7728abfe-1747-422e-b8e2-c91c9de83d09\") " pod="openshift-network-diagnostics/network-check-target-kgdzs" Oct 30 23:40:45.660204 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.659781 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2cxtv\" (UniqueName: \"kubernetes.io/projected/ece07686-4d8e-4068-9fb2-5889046cb73b-kube-api-access-2cxtv\") pod \"iptables-alerter-57xrk\" (UID: \"ece07686-4d8e-4068-9fb2-5889046cb73b\") " pod="openshift-network-operator/iptables-alerter-57xrk" Oct 30 23:40:45.660204 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.659815 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2fa7d24b-67b0-400f-94d4-903feb268782-etc-kubernetes\") pod \"multus-cb9vj\" (UID: \"2fa7d24b-67b0-400f-94d4-903feb268782\") " pod="openshift-multus/multus-cb9vj" Oct 30 23:40:45.660204 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.659862 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2fa7d24b-67b0-400f-94d4-903feb268782-host-var-lib-cni-bin\") pod \"multus-cb9vj\" (UID: \"2fa7d24b-67b0-400f-94d4-903feb268782\") " pod="openshift-multus/multus-cb9vj" Oct 30 23:40:45.660204 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.659880 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2fa7d24b-67b0-400f-94d4-903feb268782-host-var-lib-kubelet\") pod \"multus-cb9vj\" (UID: \"2fa7d24b-67b0-400f-94d4-903feb268782\") " pod="openshift-multus/multus-cb9vj" Oct 30 23:40:45.660204 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.659899 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ssd98\" (UniqueName: \"kubernetes.io/projected/2fa7d24b-67b0-400f-94d4-903feb268782-kube-api-access-ssd98\") pod \"multus-cb9vj\" (UID: \"2fa7d24b-67b0-400f-94d4-903feb268782\") " pod="openshift-multus/multus-cb9vj" Oct 30 23:40:45.660204 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.659923 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0796261f-298e-4cbe-9b3e-b3dcf4528e07-registration-dir\") pod \"aws-ebs-csi-driver-node-j7n2l\" (UID: \"0796261f-298e-4cbe-9b3e-b3dcf4528e07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j7n2l" Oct 30 23:40:45.660204 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.660153 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-26v6x" Oct 30 23:40:45.662680 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.662661 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7wrh7" Oct 30 23:40:45.662758 ip-10-0-141-180 kubenswrapper[2550]: E1030 23:40:45.662727 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7wrh7" podUID="97da2086-b00d-4f03-b502-56de5ee3d90d" Oct 30 23:40:45.665025 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.665007 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-7sbzf" Oct 30 23:40:45.697153 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.697133 2550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 23:40:45.697153 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.697133 2550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 23:40:45.697313 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.697189 2550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 23:40:45.697313 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.697280 2550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-tnlbs\"" Oct 30 23:40:45.698575 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.698549 2550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 23:40:45.698738 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.698719 2550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 23:40:45.698738 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.698732 2550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 23:40:45.698853 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.698723 2550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-v7rnf\"" Oct 30 23:40:45.701130 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.701113 2550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 23:40:45.701224 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.701113 2550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 23:40:45.701292 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.701222 2550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-xnb7d\"" Oct 30 23:40:45.701292 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.701246 2550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 23:40:45.701292 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.701239 2550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 23:40:45.708354 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.708334 2550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 23:40:45.708354 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.708351 2550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 23:40:45.708504 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.708411 2550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-hcpbw\"" Oct 30 23:40:45.710353 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.710329 2550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-ksfp2\"" Oct 30 23:40:45.710493 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.710480 2550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 23:40:45.710542 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.710483 2550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 23:40:45.712824 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.712810 2550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-72946\"" Oct 30 23:40:45.749136 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.749107 2550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 23:40:45.749136 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.749120 2550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 23:40:45.749432 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.749415 2550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 23:40:45.749512 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.749483 2550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 23:40:45.749576 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.749509 2550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-zg485\"" Oct 30 23:40:45.749576 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.749533 2550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-zwz57\"" Oct 30 23:40:45.749688 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.749623 2550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 23:40:45.749688 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.749633 2550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-hnph4\"" Oct 30 23:40:45.749688 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.749665 2550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 23:40:45.749688 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.749672 2550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 23:40:45.749864 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.749795 2550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 23:40:45.749864 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.749795 2550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 23:40:45.749864 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.749854 2550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 23:40:45.750016 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.749987 2550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 23:40:45.750443 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.750427 2550 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 23:40:45.752461 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.752444 2550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 23:40:45.752796 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.752772 2550 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 23:40:45.752877 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.752817 2550 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-v2vpw" Oct 30 23:40:45.753330 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.753317 2550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 23:40:45.761065 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.761044 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2fa7d24b-67b0-400f-94d4-903feb268782-multus-daemon-config\") pod \"multus-cb9vj\" (UID: \"2fa7d24b-67b0-400f-94d4-903feb268782\") " pod="openshift-multus/multus-cb9vj" Oct 30 23:40:45.761148 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.761073 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/97da2086-b00d-4f03-b502-56de5ee3d90d-dbus\") pod \"global-pull-secret-syncer-7wrh7\" (UID: \"97da2086-b00d-4f03-b502-56de5ee3d90d\") " pod="kube-system/global-pull-secret-syncer-7wrh7" Oct 30 23:40:45.761148 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.761092 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2fa7d24b-67b0-400f-94d4-903feb268782-cni-binary-copy\") pod \"multus-cb9vj\" (UID: \"2fa7d24b-67b0-400f-94d4-903feb268782\") " pod="openshift-multus/multus-cb9vj" Oct 30 23:40:45.761148 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.761111 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b22fe41c-cf52-4e97-81f1-2f64bcc43640-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-7mp5r\" (UID: \"b22fe41c-cf52-4e97-81f1-2f64bcc43640\") " pod="openshift-multus/multus-additional-cni-plugins-7mp5r" Oct 30 23:40:45.761148 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.761129 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2cxtv\" (UniqueName: \"kubernetes.io/projected/ece07686-4d8e-4068-9fb2-5889046cb73b-kube-api-access-2cxtv\") pod \"iptables-alerter-57xrk\" (UID: \"ece07686-4d8e-4068-9fb2-5889046cb73b\") " pod="openshift-network-operator/iptables-alerter-57xrk" Oct 30 23:40:45.761148 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.761145 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2fa7d24b-67b0-400f-94d4-903feb268782-etc-kubernetes\") pod \"multus-cb9vj\" (UID: \"2fa7d24b-67b0-400f-94d4-903feb268782\") " pod="openshift-multus/multus-cb9vj" Oct 30 23:40:45.761372 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.761206 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f8b33ecc-59a4-4680-9521-299648ea5055-hosts-file\") pod \"node-resolver-tfnr7\" (UID: \"f8b33ecc-59a4-4680-9521-299648ea5055\") " pod="openshift-dns/node-resolver-tfnr7" Oct 30 23:40:45.761372 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.761233 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/953b9446-c289-48ca-b854-a14c70e5971b-etc-sysctl-conf\") pod \"tuned-rsv7z\" (UID: \"953b9446-c289-48ca-b854-a14c70e5971b\") " pod="openshift-cluster-node-tuning-operator/tuned-rsv7z" Oct 30 23:40:45.761372 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.761250 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/953b9446-c289-48ca-b854-a14c70e5971b-etc-systemd\") pod \"tuned-rsv7z\" (UID: \"953b9446-c289-48ca-b854-a14c70e5971b\") " pod="openshift-cluster-node-tuning-operator/tuned-rsv7z" Oct 30 23:40:45.761372 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.761251 2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2fa7d24b-67b0-400f-94d4-903feb268782-etc-kubernetes\") pod \"multus-cb9vj\" (UID: \"2fa7d24b-67b0-400f-94d4-903feb268782\") " pod="openshift-multus/multus-cb9vj" Oct 30 23:40:45.761372 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.761265 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/953b9446-c289-48ca-b854-a14c70e5971b-sys\") pod \"tuned-rsv7z\" (UID: \"953b9446-c289-48ca-b854-a14c70e5971b\") " pod="openshift-cluster-node-tuning-operator/tuned-rsv7z" Oct 30 23:40:45.761372 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.761288 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/99f800b6-a38c-46ca-8234-0dbd3610c8fd-host\") pod \"node-ca-7sbzf\" (UID: \"99f800b6-a38c-46ca-8234-0dbd3610c8fd\") " pod="openshift-image-registry/node-ca-7sbzf" Oct 30 23:40:45.761372 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.761310 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2fa7d24b-67b0-400f-94d4-903feb268782-host-var-lib-kubelet\") pod \"multus-cb9vj\" (UID: \"2fa7d24b-67b0-400f-94d4-903feb268782\") " pod="openshift-multus/multus-cb9vj" Oct 30 23:40:45.761372 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.761341 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8b33ecc-59a4-4680-9521-299648ea5055-tmp-dir\") pod \"node-resolver-tfnr7\" (UID: \"f8b33ecc-59a4-4680-9521-299648ea5055\") " pod="openshift-dns/node-resolver-tfnr7" Oct 30 23:40:45.761646 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.761403 2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2fa7d24b-67b0-400f-94d4-903feb268782-host-var-lib-kubelet\") pod \"multus-cb9vj\" (UID: \"2fa7d24b-67b0-400f-94d4-903feb268782\") " pod="openshift-multus/multus-cb9vj" Oct 30 23:40:45.761646 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.761450 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/953b9446-c289-48ca-b854-a14c70e5971b-etc-tuned\") pod \"tuned-rsv7z\" (UID: \"953b9446-c289-48ca-b854-a14c70e5971b\") " pod="openshift-cluster-node-tuning-operator/tuned-rsv7z" Oct 30 23:40:45.761646 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.761485 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b22fe41c-cf52-4e97-81f1-2f64bcc43640-cni-binary-copy\") pod \"multus-additional-cni-plugins-7mp5r\" (UID: \"b22fe41c-cf52-4e97-81f1-2f64bcc43640\") " pod="openshift-multus/multus-additional-cni-plugins-7mp5r" Oct 30 23:40:45.761646 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.761521 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0d4810de-8bd2-4270-b907-341d9f840bb8-host-run-netns\") pod \"ovnkube-node-26v6x\" (UID: \"0d4810de-8bd2-4270-b907-341d9f840bb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-26v6x" Oct 30 23:40:45.761646 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.761546 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0d4810de-8bd2-4270-b907-341d9f840bb8-env-overrides\") pod \"ovnkube-node-26v6x\" (UID: \"0d4810de-8bd2-4270-b907-341d9f840bb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-26v6x" Oct 30 23:40:45.761646 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.761563 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0796261f-298e-4cbe-9b3e-b3dcf4528e07-registration-dir\") pod \"aws-ebs-csi-driver-node-j7n2l\" (UID: \"0796261f-298e-4cbe-9b3e-b3dcf4528e07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j7n2l" Oct 30 23:40:45.761646 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.761579 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2fa7d24b-67b0-400f-94d4-903feb268782-multus-socket-dir-parent\") pod \"multus-cb9vj\" (UID: \"2fa7d24b-67b0-400f-94d4-903feb268782\") " pod="openshift-multus/multus-cb9vj" Oct 30 23:40:45.761646 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.761600 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/953b9446-c289-48ca-b854-a14c70e5971b-etc-sysconfig\") pod \"tuned-rsv7z\" (UID: \"953b9446-c289-48ca-b854-a14c70e5971b\") " pod="openshift-cluster-node-tuning-operator/tuned-rsv7z" Oct 30 23:40:45.761646 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.761625 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0d4810de-8bd2-4270-b907-341d9f840bb8-run-systemd\") pod \"ovnkube-node-26v6x\" (UID: \"0d4810de-8bd2-4270-b907-341d9f840bb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-26v6x" Oct 30 23:40:45.761646 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.761634 2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0796261f-298e-4cbe-9b3e-b3dcf4528e07-registration-dir\") pod \"aws-ebs-csi-driver-node-j7n2l\" (UID: \"0796261f-298e-4cbe-9b3e-b3dcf4528e07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j7n2l" Oct 30 23:40:45.761646 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.761650 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0d4810de-8bd2-4270-b907-341d9f840bb8-etc-openvswitch\") pod \"ovnkube-node-26v6x\" (UID: \"0d4810de-8bd2-4270-b907-341d9f840bb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-26v6x" Oct 30 23:40:45.762100 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.761713 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0d4810de-8bd2-4270-b907-341d9f840bb8-log-socket\") pod \"ovnkube-node-26v6x\" (UID: \"0d4810de-8bd2-4270-b907-341d9f840bb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-26v6x" Oct 30 23:40:45.762100 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.761726 2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2fa7d24b-67b0-400f-94d4-903feb268782-multus-socket-dir-parent\") pod \"multus-cb9vj\" (UID: \"2fa7d24b-67b0-400f-94d4-903feb268782\") " pod="openshift-multus/multus-cb9vj" Oct 30 23:40:45.762100 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.761778 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ece07686-4d8e-4068-9fb2-5889046cb73b-host-slash\") pod \"iptables-alerter-57xrk\" (UID: \"ece07686-4d8e-4068-9fb2-5889046cb73b\") " pod="openshift-network-operator/iptables-alerter-57xrk" Oct 30 23:40:45.762100 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.761805 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0796261f-298e-4cbe-9b3e-b3dcf4528e07-sys-fs\") pod \"aws-ebs-csi-driver-node-j7n2l\" (UID: \"0796261f-298e-4cbe-9b3e-b3dcf4528e07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j7n2l" Oct 30 23:40:45.762100 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.761822 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2fa7d24b-67b0-400f-94d4-903feb268782-system-cni-dir\") pod \"multus-cb9vj\" (UID: \"2fa7d24b-67b0-400f-94d4-903feb268782\") " pod="openshift-multus/multus-cb9vj" Oct 30 23:40:45.762100 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.761836 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/953b9446-c289-48ca-b854-a14c70e5971b-run\") pod \"tuned-rsv7z\" (UID: \"953b9446-c289-48ca-b854-a14c70e5971b\") " pod="openshift-cluster-node-tuning-operator/tuned-rsv7z" Oct 30 23:40:45.762100 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.761852 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a8e4c4db-197e-420c-9c60-4273cdf95377-agent-certs\") pod \"konnectivity-agent-9s2fx\" (UID: \"a8e4c4db-197e-420c-9c60-4273cdf95377\") " pod="kube-system/konnectivity-agent-9s2fx" Oct 30 23:40:45.762100 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.761877 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a8e4c4db-197e-420c-9c60-4273cdf95377-konnectivity-ca\") pod \"konnectivity-agent-9s2fx\" (UID: \"a8e4c4db-197e-420c-9c60-4273cdf95377\") " pod="kube-system/konnectivity-agent-9s2fx" Oct 30 23:40:45.762100 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.761888 2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2fa7d24b-67b0-400f-94d4-903feb268782-system-cni-dir\") pod \"multus-cb9vj\" (UID: \"2fa7d24b-67b0-400f-94d4-903feb268782\") " pod="openshift-multus/multus-cb9vj" Oct 30 23:40:45.762100 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.761902 2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ece07686-4d8e-4068-9fb2-5889046cb73b-host-slash\") pod \"iptables-alerter-57xrk\" (UID: \"ece07686-4d8e-4068-9fb2-5889046cb73b\") " pod="openshift-network-operator/iptables-alerter-57xrk" Oct 30 23:40:45.762100 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.761903 2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0796261f-298e-4cbe-9b3e-b3dcf4528e07-sys-fs\") pod \"aws-ebs-csi-driver-node-j7n2l\" (UID: \"0796261f-298e-4cbe-9b3e-b3dcf4528e07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j7n2l" Oct 30 23:40:45.762100 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.761918 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2fa7d24b-67b0-400f-94d4-903feb268782-cnibin\") pod \"multus-cb9vj\" (UID: \"2fa7d24b-67b0-400f-94d4-903feb268782\") " pod="openshift-multus/multus-cb9vj" Oct 30 23:40:45.762576 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.762017 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2fa7d24b-67b0-400f-94d4-903feb268782-host-run-k8s-cni-cncf-io\") pod \"multus-cb9vj\" (UID: \"2fa7d24b-67b0-400f-94d4-903feb268782\") " pod="openshift-multus/multus-cb9vj" Oct 30 23:40:45.762576 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.762409 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/953b9446-c289-48ca-b854-a14c70e5971b-var-lib-kubelet\") pod \"tuned-rsv7z\" (UID: \"953b9446-c289-48ca-b854-a14c70e5971b\") " pod="openshift-cluster-node-tuning-operator/tuned-rsv7z" Oct 30 23:40:45.762576 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.762451 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/953b9446-c289-48ca-b854-a14c70e5971b-tmp\") pod \"tuned-rsv7z\" (UID: \"953b9446-c289-48ca-b854-a14c70e5971b\") " pod="openshift-cluster-node-tuning-operator/tuned-rsv7z" Oct 30 23:40:45.762576 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.762440 2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2fa7d24b-67b0-400f-94d4-903feb268782-cnibin\") pod \"multus-cb9vj\" (UID: \"2fa7d24b-67b0-400f-94d4-903feb268782\") " pod="openshift-multus/multus-cb9vj" Oct 30 23:40:45.762576 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.762485 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0d4810de-8bd2-4270-b907-341d9f840bb8-host-kubelet\") pod \"ovnkube-node-26v6x\" (UID: \"0d4810de-8bd2-4270-b907-341d9f840bb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-26v6x" Oct 30 23:40:45.762876 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.762576 2550 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 23:40:45.762876 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.762619 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0d4810de-8bd2-4270-b907-341d9f840bb8-run-openvswitch\") pod \"ovnkube-node-26v6x\" (UID: \"0d4810de-8bd2-4270-b907-341d9f840bb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-26v6x" Oct 30 23:40:45.762876 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.762653 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ece07686-4d8e-4068-9fb2-5889046cb73b-iptables-alerter-script\") pod \"iptables-alerter-57xrk\" (UID: \"ece07686-4d8e-4068-9fb2-5889046cb73b\") " pod="openshift-network-operator/iptables-alerter-57xrk" Oct 30 23:40:45.762876 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.762689 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25rtq\" (UniqueName: \"kubernetes.io/projected/0796261f-298e-4cbe-9b3e-b3dcf4528e07-kube-api-access-25rtq\") pod \"aws-ebs-csi-driver-node-j7n2l\" (UID: \"0796261f-298e-4cbe-9b3e-b3dcf4528e07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j7n2l" Oct 30 23:40:45.762876 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.762723 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2fa7d24b-67b0-400f-94d4-903feb268782-host-run-multus-certs\") pod \"multus-cb9vj\" (UID: \"2fa7d24b-67b0-400f-94d4-903feb268782\") " pod="openshift-multus/multus-cb9vj" Oct 30 23:40:45.762876 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.762759 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nsggg\" (UniqueName: \"kubernetes.io/projected/145dd323-9370-4b9d-a96e-8b93d0a47a7c-kube-api-access-nsggg\") pod \"network-metrics-daemon-pt58s\" (UID: \"145dd323-9370-4b9d-a96e-8b93d0a47a7c\") " pod="openshift-multus/network-metrics-daemon-pt58s" Oct 30 23:40:45.762876 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.762796 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/953b9446-c289-48ca-b854-a14c70e5971b-etc-modprobe-d\") pod \"tuned-rsv7z\" (UID: \"953b9446-c289-48ca-b854-a14c70e5971b\") " pod="openshift-cluster-node-tuning-operator/tuned-rsv7z" Oct 30 23:40:45.762876 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.762823 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/953b9446-c289-48ca-b854-a14c70e5971b-lib-modules\") pod \"tuned-rsv7z\" (UID: \"953b9446-c289-48ca-b854-a14c70e5971b\") " pod="openshift-cluster-node-tuning-operator/tuned-rsv7z" Oct 30 23:40:45.762876 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.762841 2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2fa7d24b-67b0-400f-94d4-903feb268782-cni-binary-copy\") pod \"multus-cb9vj\" (UID: \"2fa7d24b-67b0-400f-94d4-903feb268782\") " pod="openshift-multus/multus-cb9vj" Oct 30 23:40:45.762876 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.762862 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0d4810de-8bd2-4270-b907-341d9f840bb8-host-cni-bin\") pod \"ovnkube-node-26v6x\" (UID: \"0d4810de-8bd2-4270-b907-341d9f840bb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-26v6x" Oct 30 23:40:45.763375 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.762926 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48m7s\" (UniqueName: \"kubernetes.io/projected/0d4810de-8bd2-4270-b907-341d9f840bb8-kube-api-access-48m7s\") pod \"ovnkube-node-26v6x\" (UID: \"0d4810de-8bd2-4270-b907-341d9f840bb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-26v6x" Oct 30 23:40:45.763375 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.763020 2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a8e4c4db-197e-420c-9c60-4273cdf95377-konnectivity-ca\") pod \"konnectivity-agent-9s2fx\" (UID: \"a8e4c4db-197e-420c-9c60-4273cdf95377\") " pod="kube-system/konnectivity-agent-9s2fx" Oct 30 23:40:45.763375 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.763012 2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2fa7d24b-67b0-400f-94d4-903feb268782-multus-daemon-config\") pod \"multus-cb9vj\" (UID: \"2fa7d24b-67b0-400f-94d4-903feb268782\") " pod="openshift-multus/multus-cb9vj" Oct 30 23:40:45.763375 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.762112 2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2fa7d24b-67b0-400f-94d4-903feb268782-host-run-k8s-cni-cncf-io\") pod \"multus-cb9vj\" (UID: \"2fa7d24b-67b0-400f-94d4-903feb268782\") " pod="openshift-multus/multus-cb9vj" Oct 30 23:40:45.763375 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.763070 2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2fa7d24b-67b0-400f-94d4-903feb268782-host-run-multus-certs\") pod \"multus-cb9vj\" (UID: \"2fa7d24b-67b0-400f-94d4-903feb268782\") " pod="openshift-multus/multus-cb9vj" Oct 30 23:40:45.763375 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.763085 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2fa7d24b-67b0-400f-94d4-903feb268782-host-run-netns\") pod \"multus-cb9vj\" (UID: \"2fa7d24b-67b0-400f-94d4-903feb268782\") " pod="openshift-multus/multus-cb9vj" Oct 30 23:40:45.763375 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.763122 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0d4810de-8bd2-4270-b907-341d9f840bb8-run-ovn\") pod \"ovnkube-node-26v6x\" (UID: \"0d4810de-8bd2-4270-b907-341d9f840bb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-26v6x" Oct 30 23:40:45.763771 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.763385 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0d4810de-8bd2-4270-b907-341d9f840bb8-host-cni-netd\") pod \"ovnkube-node-26v6x\" (UID: \"0d4810de-8bd2-4270-b907-341d9f840bb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-26v6x" Oct 30 23:40:45.763771 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.763424 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b22fe41c-cf52-4e97-81f1-2f64bcc43640-tuning-conf-dir\") pod \"multus-additional-cni-plugins-7mp5r\" (UID: \"b22fe41c-cf52-4e97-81f1-2f64bcc43640\") " pod="openshift-multus/multus-additional-cni-plugins-7mp5r" Oct 30 23:40:45.763771 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.763457 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0d4810de-8bd2-4270-b907-341d9f840bb8-node-log\") pod \"ovnkube-node-26v6x\" (UID: \"0d4810de-8bd2-4270-b907-341d9f840bb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-26v6x" Oct 30 23:40:45.763771 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.763504 2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2fa7d24b-67b0-400f-94d4-903feb268782-host-run-netns\") pod \"multus-cb9vj\" (UID: \"2fa7d24b-67b0-400f-94d4-903feb268782\") " pod="openshift-multus/multus-cb9vj" Oct 30 23:40:45.763771 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.763508 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqmnp\" (UniqueName: \"kubernetes.io/projected/7728abfe-1747-422e-b8e2-c91c9de83d09-kube-api-access-jqmnp\") pod \"network-check-target-kgdzs\" (UID: \"7728abfe-1747-422e-b8e2-c91c9de83d09\") " pod="openshift-network-diagnostics/network-check-target-kgdzs" Oct 30 23:40:45.763771 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.763560 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gtt94\" (UniqueName: \"kubernetes.io/projected/953b9446-c289-48ca-b854-a14c70e5971b-kube-api-access-gtt94\") pod \"tuned-rsv7z\" (UID: \"953b9446-c289-48ca-b854-a14c70e5971b\") " pod="openshift-cluster-node-tuning-operator/tuned-rsv7z" Oct 30 23:40:45.763771 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.763688 2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ece07686-4d8e-4068-9fb2-5889046cb73b-iptables-alerter-script\") pod \"iptables-alerter-57xrk\" (UID: \"ece07686-4d8e-4068-9fb2-5889046cb73b\") " pod="openshift-network-operator/iptables-alerter-57xrk" Oct 30 23:40:45.763771 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.763756 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0d4810de-8bd2-4270-b907-341d9f840bb8-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-26v6x\" (UID: \"0d4810de-8bd2-4270-b907-341d9f840bb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-26v6x" Oct 30 23:40:45.764138 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.763834 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2fa7d24b-67b0-400f-94d4-903feb268782-host-var-lib-cni-bin\") pod \"multus-cb9vj\" (UID: \"2fa7d24b-67b0-400f-94d4-903feb268782\") " pod="openshift-multus/multus-cb9vj" Oct 30 23:40:45.764138 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.763893 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ssd98\" (UniqueName: \"kubernetes.io/projected/2fa7d24b-67b0-400f-94d4-903feb268782-kube-api-access-ssd98\") pod \"multus-cb9vj\" (UID: \"2fa7d24b-67b0-400f-94d4-903feb268782\") " pod="openshift-multus/multus-cb9vj" Oct 30 23:40:45.764138 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.763929 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/99f800b6-a38c-46ca-8234-0dbd3610c8fd-serviceca\") pod \"node-ca-7sbzf\" (UID: \"99f800b6-a38c-46ca-8234-0dbd3610c8fd\") " pod="openshift-image-registry/node-ca-7sbzf" Oct 30 23:40:45.764138 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.763976 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mlhgb\" (UniqueName: \"kubernetes.io/projected/99f800b6-a38c-46ca-8234-0dbd3610c8fd-kube-api-access-mlhgb\") pod \"node-ca-7sbzf\" (UID: \"99f800b6-a38c-46ca-8234-0dbd3610c8fd\") " pod="openshift-image-registry/node-ca-7sbzf" Oct 30 23:40:45.764138 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.764009 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b22fe41c-cf52-4e97-81f1-2f64bcc43640-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-7mp5r\" (UID: \"b22fe41c-cf52-4e97-81f1-2f64bcc43640\") " pod="openshift-multus/multus-additional-cni-plugins-7mp5r" Oct 30 23:40:45.764138 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.764039 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0d4810de-8bd2-4270-b907-341d9f840bb8-ovnkube-config\") pod \"ovnkube-node-26v6x\" (UID: \"0d4810de-8bd2-4270-b907-341d9f840bb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-26v6x" Oct 30 23:40:45.764138 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.764069 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b22fe41c-cf52-4e97-81f1-2f64bcc43640-cnibin\") pod \"multus-additional-cni-plugins-7mp5r\" (UID: \"b22fe41c-cf52-4e97-81f1-2f64bcc43640\") " pod="openshift-multus/multus-additional-cni-plugins-7mp5r" Oct 30 23:40:45.764138 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.764109 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/953b9446-c289-48ca-b854-a14c70e5971b-etc-kubernetes\") pod \"tuned-rsv7z\" (UID: \"953b9446-c289-48ca-b854-a14c70e5971b\") " pod="openshift-cluster-node-tuning-operator/tuned-rsv7z" Oct 30 23:40:45.764138 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.764139 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/953b9446-c289-48ca-b854-a14c70e5971b-etc-sysctl-d\") pod \"tuned-rsv7z\" (UID: \"953b9446-c289-48ca-b854-a14c70e5971b\") " pod="openshift-cluster-node-tuning-operator/tuned-rsv7z" Oct 30 23:40:45.764575 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.764169 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b22fe41c-cf52-4e97-81f1-2f64bcc43640-system-cni-dir\") pod \"multus-additional-cni-plugins-7mp5r\" (UID: \"b22fe41c-cf52-4e97-81f1-2f64bcc43640\") " pod="openshift-multus/multus-additional-cni-plugins-7mp5r" Oct 30 23:40:45.764575 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.764220 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0d4810de-8bd2-4270-b907-341d9f840bb8-host-run-ovn-kubernetes\") pod \"ovnkube-node-26v6x\" (UID: \"0d4810de-8bd2-4270-b907-341d9f840bb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-26v6x" Oct 30 23:40:45.764575 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.764246 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0d4810de-8bd2-4270-b907-341d9f840bb8-ovnkube-script-lib\") pod \"ovnkube-node-26v6x\" (UID: \"0d4810de-8bd2-4270-b907-341d9f840bb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-26v6x" Oct 30 23:40:45.764575 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.764279 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2fa7d24b-67b0-400f-94d4-903feb268782-multus-cni-dir\") pod \"multus-cb9vj\" (UID: \"2fa7d24b-67b0-400f-94d4-903feb268782\") " pod="openshift-multus/multus-cb9vj" Oct 30 23:40:45.764575 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.764309 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-btgs8\" (UniqueName: \"kubernetes.io/projected/b22fe41c-cf52-4e97-81f1-2f64bcc43640-kube-api-access-btgs8\") pod \"multus-additional-cni-plugins-7mp5r\" (UID: \"b22fe41c-cf52-4e97-81f1-2f64bcc43640\") " pod="openshift-multus/multus-additional-cni-plugins-7mp5r" Oct 30 23:40:45.764575 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.764344 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/97da2086-b00d-4f03-b502-56de5ee3d90d-original-pull-secret\") pod \"global-pull-secret-syncer-7wrh7\" (UID: \"97da2086-b00d-4f03-b502-56de5ee3d90d\") " pod="kube-system/global-pull-secret-syncer-7wrh7" Oct 30 23:40:45.764575 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.764443 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0796261f-298e-4cbe-9b3e-b3dcf4528e07-etc-selinux\") pod \"aws-ebs-csi-driver-node-j7n2l\" (UID: \"0796261f-298e-4cbe-9b3e-b3dcf4528e07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j7n2l" Oct 30 23:40:45.764575 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.764476 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2fa7d24b-67b0-400f-94d4-903feb268782-hostroot\") pod \"multus-cb9vj\" (UID: \"2fa7d24b-67b0-400f-94d4-903feb268782\") " pod="openshift-multus/multus-cb9vj" Oct 30 23:40:45.764575 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.764510 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/953b9446-c289-48ca-b854-a14c70e5971b-host\") pod \"tuned-rsv7z\" (UID: \"953b9446-c289-48ca-b854-a14c70e5971b\") " pod="openshift-cluster-node-tuning-operator/tuned-rsv7z" Oct 30 23:40:45.764575 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.764547 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0d4810de-8bd2-4270-b907-341d9f840bb8-host-slash\") pod \"ovnkube-node-26v6x\" (UID: \"0d4810de-8bd2-4270-b907-341d9f840bb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-26v6x" Oct 30 23:40:45.765028 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.764585 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0d4810de-8bd2-4270-b907-341d9f840bb8-ovn-node-metrics-cert\") pod \"ovnkube-node-26v6x\" (UID: \"0d4810de-8bd2-4270-b907-341d9f840bb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-26v6x" Oct 30 23:40:45.765028 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.764611 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0796261f-298e-4cbe-9b3e-b3dcf4528e07-kubelet-dir\") pod \"aws-ebs-csi-driver-node-j7n2l\" (UID: \"0796261f-298e-4cbe-9b3e-b3dcf4528e07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j7n2l" Oct 30 23:40:45.765028 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.764644 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2fa7d24b-67b0-400f-94d4-903feb268782-host-var-lib-cni-multus\") pod \"multus-cb9vj\" (UID: \"2fa7d24b-67b0-400f-94d4-903feb268782\") " pod="openshift-multus/multus-cb9vj" Oct 30 23:40:45.765028 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.764675 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2fa7d24b-67b0-400f-94d4-903feb268782-multus-conf-dir\") pod \"multus-cb9vj\" (UID: \"2fa7d24b-67b0-400f-94d4-903feb268782\") " pod="openshift-multus/multus-cb9vj" Oct 30 23:40:45.765028 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.764705 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/145dd323-9370-4b9d-a96e-8b93d0a47a7c-metrics-certs\") pod \"network-metrics-daemon-pt58s\" (UID: \"145dd323-9370-4b9d-a96e-8b93d0a47a7c\") " pod="openshift-multus/network-metrics-daemon-pt58s" Oct 30 23:40:45.765028 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.764732 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7p9rj\" (UniqueName: \"kubernetes.io/projected/f8b33ecc-59a4-4680-9521-299648ea5055-kube-api-access-7p9rj\") pod \"node-resolver-tfnr7\" (UID: \"f8b33ecc-59a4-4680-9521-299648ea5055\") " pod="openshift-dns/node-resolver-tfnr7" Oct 30 23:40:45.765028 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.764771 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0d4810de-8bd2-4270-b907-341d9f840bb8-systemd-units\") pod \"ovnkube-node-26v6x\" (UID: \"0d4810de-8bd2-4270-b907-341d9f840bb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-26v6x" Oct 30 23:40:45.765028 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.764803 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0d4810de-8bd2-4270-b907-341d9f840bb8-var-lib-openvswitch\") pod \"ovnkube-node-26v6x\" (UID: \"0d4810de-8bd2-4270-b907-341d9f840bb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-26v6x" Oct 30 23:40:45.765028 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.764836 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/97da2086-b00d-4f03-b502-56de5ee3d90d-kubelet-config\") pod \"global-pull-secret-syncer-7wrh7\" (UID: \"97da2086-b00d-4f03-b502-56de5ee3d90d\") " pod="kube-system/global-pull-secret-syncer-7wrh7" Oct 30 23:40:45.765028 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.764871 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0796261f-298e-4cbe-9b3e-b3dcf4528e07-socket-dir\") pod \"aws-ebs-csi-driver-node-j7n2l\" (UID: \"0796261f-298e-4cbe-9b3e-b3dcf4528e07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j7n2l" Oct 30 23:40:45.765028 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.764903 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0796261f-298e-4cbe-9b3e-b3dcf4528e07-device-dir\") pod \"aws-ebs-csi-driver-node-j7n2l\" (UID: \"0796261f-298e-4cbe-9b3e-b3dcf4528e07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j7n2l" Oct 30 23:40:45.765531 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.765159 2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0796261f-298e-4cbe-9b3e-b3dcf4528e07-device-dir\") pod \"aws-ebs-csi-driver-node-j7n2l\" (UID: \"0796261f-298e-4cbe-9b3e-b3dcf4528e07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j7n2l" Oct 30 23:40:45.765531 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.765205 2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2fa7d24b-67b0-400f-94d4-903feb268782-host-var-lib-cni-bin\") pod \"multus-cb9vj\" (UID: \"2fa7d24b-67b0-400f-94d4-903feb268782\") " pod="openshift-multus/multus-cb9vj" Oct 30 23:40:45.765531 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.765472 2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2fa7d24b-67b0-400f-94d4-903feb268782-multus-cni-dir\") pod \"multus-cb9vj\" (UID: \"2fa7d24b-67b0-400f-94d4-903feb268782\") " pod="openshift-multus/multus-cb9vj" Oct 30 23:40:45.765674 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.765530 2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0796261f-298e-4cbe-9b3e-b3dcf4528e07-etc-selinux\") pod \"aws-ebs-csi-driver-node-j7n2l\" (UID: \"0796261f-298e-4cbe-9b3e-b3dcf4528e07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j7n2l" Oct 30 23:40:45.765674 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.765570 2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2fa7d24b-67b0-400f-94d4-903feb268782-hostroot\") pod \"multus-cb9vj\" (UID: \"2fa7d24b-67b0-400f-94d4-903feb268782\") " pod="openshift-multus/multus-cb9vj" Oct 30 23:40:45.765674 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.765630 2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0796261f-298e-4cbe-9b3e-b3dcf4528e07-kubelet-dir\") pod \"aws-ebs-csi-driver-node-j7n2l\" (UID: \"0796261f-298e-4cbe-9b3e-b3dcf4528e07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j7n2l" Oct 30 23:40:45.765674 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.765665 2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2fa7d24b-67b0-400f-94d4-903feb268782-host-var-lib-cni-multus\") pod \"multus-cb9vj\" (UID: \"2fa7d24b-67b0-400f-94d4-903feb268782\") " pod="openshift-multus/multus-cb9vj" Oct 30 23:40:45.765877 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.765706 2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2fa7d24b-67b0-400f-94d4-903feb268782-multus-conf-dir\") pod \"multus-cb9vj\" (UID: \"2fa7d24b-67b0-400f-94d4-903feb268782\") " pod="openshift-multus/multus-cb9vj" Oct 30 23:40:45.765877 ip-10-0-141-180 kubenswrapper[2550]: E1030 23:40:45.765789 2550 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 23:40:45.765877 ip-10-0-141-180 kubenswrapper[2550]: E1030 23:40:45.765857 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/145dd323-9370-4b9d-a96e-8b93d0a47a7c-metrics-certs podName:145dd323-9370-4b9d-a96e-8b93d0a47a7c nodeName:}" failed. No retries permitted until 2026-10-30 23:40:46.265828757 +0000 UTC m=+2.324032658 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/145dd323-9370-4b9d-a96e-8b93d0a47a7c-metrics-certs") pod "network-metrics-daemon-pt58s" (UID: "145dd323-9370-4b9d-a96e-8b93d0a47a7c") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 23:40:45.766044 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.765991 2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0796261f-298e-4cbe-9b3e-b3dcf4528e07-socket-dir\") pod \"aws-ebs-csi-driver-node-j7n2l\" (UID: \"0796261f-298e-4cbe-9b3e-b3dcf4528e07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j7n2l" Oct 30 23:40:45.767359 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.767338 2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a8e4c4db-197e-420c-9c60-4273cdf95377-agent-certs\") pod \"konnectivity-agent-9s2fx\" (UID: \"a8e4c4db-197e-420c-9c60-4273cdf95377\") " pod="kube-system/konnectivity-agent-9s2fx" Oct 30 23:40:45.828170 ip-10-0-141-180 kubenswrapper[2550]: E1030 23:40:45.828113 2550 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 23:40:45.828170 ip-10-0-141-180 kubenswrapper[2550]: E1030 23:40:45.828138 2550 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 23:40:45.828170 ip-10-0-141-180 kubenswrapper[2550]: E1030 23:40:45.828149 2550 projected.go:196] Error preparing data for projected volume kube-api-access-jqmnp for pod openshift-network-diagnostics/network-check-target-kgdzs: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 23:40:45.828363 ip-10-0-141-180 kubenswrapper[2550]: E1030 23:40:45.828231 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7728abfe-1747-422e-b8e2-c91c9de83d09-kube-api-access-jqmnp podName:7728abfe-1747-422e-b8e2-c91c9de83d09 nodeName:}" failed. No retries permitted until 2026-10-30 23:40:46.328215521 +0000 UTC m=+2.386419431 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-jqmnp" (UniqueName: "kubernetes.io/projected/7728abfe-1747-422e-b8e2-c91c9de83d09-kube-api-access-jqmnp") pod "network-check-target-kgdzs" (UID: "7728abfe-1747-422e-b8e2-c91c9de83d09") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 23:40:45.830496 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.830471 2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2cxtv\" (UniqueName: \"kubernetes.io/projected/ece07686-4d8e-4068-9fb2-5889046cb73b-kube-api-access-2cxtv\") pod \"iptables-alerter-57xrk\" (UID: \"ece07686-4d8e-4068-9fb2-5889046cb73b\") " pod="openshift-network-operator/iptables-alerter-57xrk" Oct 30 23:40:45.830581 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.830515 2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nsggg\" (UniqueName: \"kubernetes.io/projected/145dd323-9370-4b9d-a96e-8b93d0a47a7c-kube-api-access-nsggg\") pod \"network-metrics-daemon-pt58s\" (UID: \"145dd323-9370-4b9d-a96e-8b93d0a47a7c\") " pod="openshift-multus/network-metrics-daemon-pt58s" Oct 30 23:40:45.841536 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.841510 2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ssd98\" (UniqueName: \"kubernetes.io/projected/2fa7d24b-67b0-400f-94d4-903feb268782-kube-api-access-ssd98\") pod \"multus-cb9vj\" (UID: \"2fa7d24b-67b0-400f-94d4-903feb268782\") " pod="openshift-multus/multus-cb9vj" Oct 30 23:40:45.841620 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.841540 2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-25rtq\" (UniqueName: \"kubernetes.io/projected/0796261f-298e-4cbe-9b3e-b3dcf4528e07-kube-api-access-25rtq\") pod \"aws-ebs-csi-driver-node-j7n2l\" (UID: \"0796261f-298e-4cbe-9b3e-b3dcf4528e07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j7n2l" Oct 30 23:40:45.861913 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.861894 2550 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 23:40:45.866304 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.866284 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/953b9446-c289-48ca-b854-a14c70e5971b-etc-systemd\") pod \"tuned-rsv7z\" (UID: \"953b9446-c289-48ca-b854-a14c70e5971b\") " pod="openshift-cluster-node-tuning-operator/tuned-rsv7z" Oct 30 23:40:45.866357 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.866312 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/953b9446-c289-48ca-b854-a14c70e5971b-sys\") pod \"tuned-rsv7z\" (UID: \"953b9446-c289-48ca-b854-a14c70e5971b\") " pod="openshift-cluster-node-tuning-operator/tuned-rsv7z" Oct 30 23:40:45.866357 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.866329 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/99f800b6-a38c-46ca-8234-0dbd3610c8fd-host\") pod \"node-ca-7sbzf\" (UID: \"99f800b6-a38c-46ca-8234-0dbd3610c8fd\") " pod="openshift-image-registry/node-ca-7sbzf" Oct 30 23:40:45.866452 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.866360 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8b33ecc-59a4-4680-9521-299648ea5055-tmp-dir\") pod \"node-resolver-tfnr7\" (UID: \"f8b33ecc-59a4-4680-9521-299648ea5055\") " pod="openshift-dns/node-resolver-tfnr7" Oct 30 23:40:45.866452 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.866376 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/953b9446-c289-48ca-b854-a14c70e5971b-etc-tuned\") pod \"tuned-rsv7z\" (UID: \"953b9446-c289-48ca-b854-a14c70e5971b\") " pod="openshift-cluster-node-tuning-operator/tuned-rsv7z" Oct 30 23:40:45.866452 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.866393 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b22fe41c-cf52-4e97-81f1-2f64bcc43640-cni-binary-copy\") pod \"multus-additional-cni-plugins-7mp5r\" (UID: \"b22fe41c-cf52-4e97-81f1-2f64bcc43640\") " pod="openshift-multus/multus-additional-cni-plugins-7mp5r" Oct 30 23:40:45.866452 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.866412 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0d4810de-8bd2-4270-b907-341d9f840bb8-host-run-netns\") pod \"ovnkube-node-26v6x\" (UID: \"0d4810de-8bd2-4270-b907-341d9f840bb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-26v6x" Oct 30 23:40:45.866452 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.866436 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0d4810de-8bd2-4270-b907-341d9f840bb8-env-overrides\") pod \"ovnkube-node-26v6x\" (UID: \"0d4810de-8bd2-4270-b907-341d9f840bb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-26v6x" Oct 30 23:40:45.866695 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.866462 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/953b9446-c289-48ca-b854-a14c70e5971b-etc-sysconfig\") pod \"tuned-rsv7z\" (UID: \"953b9446-c289-48ca-b854-a14c70e5971b\") " pod="openshift-cluster-node-tuning-operator/tuned-rsv7z" Oct 30 23:40:45.866695 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.866487 2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/953b9446-c289-48ca-b854-a14c70e5971b-etc-systemd\") pod \"tuned-rsv7z\" (UID: \"953b9446-c289-48ca-b854-a14c70e5971b\") " pod="openshift-cluster-node-tuning-operator/tuned-rsv7z" Oct 30 23:40:45.866695 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.866495 2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/953b9446-c289-48ca-b854-a14c70e5971b-sys\") pod \"tuned-rsv7z\" (UID: \"953b9446-c289-48ca-b854-a14c70e5971b\") " pod="openshift-cluster-node-tuning-operator/tuned-rsv7z" Oct 30 23:40:45.866695 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.866511 2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/953b9446-c289-48ca-b854-a14c70e5971b-etc-sysconfig\") pod \"tuned-rsv7z\" (UID: \"953b9446-c289-48ca-b854-a14c70e5971b\") " pod="openshift-cluster-node-tuning-operator/tuned-rsv7z" Oct 30 23:40:45.866695 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.866551 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0d4810de-8bd2-4270-b907-341d9f840bb8-run-systemd\") pod \"ovnkube-node-26v6x\" (UID: \"0d4810de-8bd2-4270-b907-341d9f840bb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-26v6x" Oct 30 23:40:45.866695 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.866568 2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0d4810de-8bd2-4270-b907-341d9f840bb8-host-run-netns\") pod \"ovnkube-node-26v6x\" (UID: \"0d4810de-8bd2-4270-b907-341d9f840bb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-26v6x" Oct 30 23:40:45.866695 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.866580 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0d4810de-8bd2-4270-b907-341d9f840bb8-etc-openvswitch\") pod \"ovnkube-node-26v6x\" (UID: \"0d4810de-8bd2-4270-b907-341d9f840bb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-26v6x" Oct 30 23:40:45.866695 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.866606 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0d4810de-8bd2-4270-b907-341d9f840bb8-log-socket\") pod \"ovnkube-node-26v6x\" (UID: \"0d4810de-8bd2-4270-b907-341d9f840bb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-26v6x" Oct 30 23:40:45.866695 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.866626 2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0d4810de-8bd2-4270-b907-341d9f840bb8-run-systemd\") pod \"ovnkube-node-26v6x\" (UID: \"0d4810de-8bd2-4270-b907-341d9f840bb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-26v6x" Oct 30 23:40:45.866695 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.866651 2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0d4810de-8bd2-4270-b907-341d9f840bb8-log-socket\") pod \"ovnkube-node-26v6x\" (UID: \"0d4810de-8bd2-4270-b907-341d9f840bb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-26v6x" Oct 30 23:40:45.866695 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.866651 2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0d4810de-8bd2-4270-b907-341d9f840bb8-etc-openvswitch\") pod \"ovnkube-node-26v6x\" (UID: \"0d4810de-8bd2-4270-b907-341d9f840bb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-26v6x" Oct 30 23:40:45.866695 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.866670 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/953b9446-c289-48ca-b854-a14c70e5971b-run\") pod \"tuned-rsv7z\" (UID: \"953b9446-c289-48ca-b854-a14c70e5971b\") " pod="openshift-cluster-node-tuning-operator/tuned-rsv7z" Oct 30 23:40:45.866695 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.866703 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/953b9446-c289-48ca-b854-a14c70e5971b-var-lib-kubelet\") pod \"tuned-rsv7z\" (UID: \"953b9446-c289-48ca-b854-a14c70e5971b\") " pod="openshift-cluster-node-tuning-operator/tuned-rsv7z" Oct 30 23:40:45.867277 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.866708 2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/953b9446-c289-48ca-b854-a14c70e5971b-run\") pod \"tuned-rsv7z\" (UID: \"953b9446-c289-48ca-b854-a14c70e5971b\") " pod="openshift-cluster-node-tuning-operator/tuned-rsv7z" Oct 30 23:40:45.867277 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.866729 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/953b9446-c289-48ca-b854-a14c70e5971b-tmp\") pod \"tuned-rsv7z\" (UID: \"953b9446-c289-48ca-b854-a14c70e5971b\") " pod="openshift-cluster-node-tuning-operator/tuned-rsv7z" Oct 30 23:40:45.867277 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.866728 2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/99f800b6-a38c-46ca-8234-0dbd3610c8fd-host\") pod \"node-ca-7sbzf\" (UID: \"99f800b6-a38c-46ca-8234-0dbd3610c8fd\") " pod="openshift-image-registry/node-ca-7sbzf" Oct 30 23:40:45.867277 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.866785 2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/953b9446-c289-48ca-b854-a14c70e5971b-var-lib-kubelet\") pod \"tuned-rsv7z\" (UID: \"953b9446-c289-48ca-b854-a14c70e5971b\") " pod="openshift-cluster-node-tuning-operator/tuned-rsv7z" Oct 30 23:40:45.867277 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.866768 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0d4810de-8bd2-4270-b907-341d9f840bb8-host-kubelet\") pod \"ovnkube-node-26v6x\" (UID: \"0d4810de-8bd2-4270-b907-341d9f840bb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-26v6x" Oct 30 23:40:45.867277 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.866829 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0d4810de-8bd2-4270-b907-341d9f840bb8-run-openvswitch\") pod \"ovnkube-node-26v6x\" (UID: \"0d4810de-8bd2-4270-b907-341d9f840bb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-26v6x" Oct 30 23:40:45.867277 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.866853 2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0d4810de-8bd2-4270-b907-341d9f840bb8-run-openvswitch\") pod \"ovnkube-node-26v6x\" (UID: \"0d4810de-8bd2-4270-b907-341d9f840bb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-26v6x" Oct 30 23:40:45.867277 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.866828 2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0d4810de-8bd2-4270-b907-341d9f840bb8-host-kubelet\") pod \"ovnkube-node-26v6x\" (UID: \"0d4810de-8bd2-4270-b907-341d9f840bb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-26v6x" Oct 30 23:40:45.867277 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.866881 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/953b9446-c289-48ca-b854-a14c70e5971b-etc-modprobe-d\") pod \"tuned-rsv7z\" (UID: \"953b9446-c289-48ca-b854-a14c70e5971b\") " pod="openshift-cluster-node-tuning-operator/tuned-rsv7z" Oct 30 23:40:45.867277 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.866899 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/953b9446-c289-48ca-b854-a14c70e5971b-lib-modules\") pod \"tuned-rsv7z\" (UID: \"953b9446-c289-48ca-b854-a14c70e5971b\") " pod="openshift-cluster-node-tuning-operator/tuned-rsv7z" Oct 30 23:40:45.867277 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.866915 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0d4810de-8bd2-4270-b907-341d9f840bb8-host-cni-bin\") pod \"ovnkube-node-26v6x\" (UID: \"0d4810de-8bd2-4270-b907-341d9f840bb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-26v6x" Oct 30 23:40:45.867277 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.866931 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48m7s\" (UniqueName: \"kubernetes.io/projected/0d4810de-8bd2-4270-b907-341d9f840bb8-kube-api-access-48m7s\") pod \"ovnkube-node-26v6x\" (UID: \"0d4810de-8bd2-4270-b907-341d9f840bb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-26v6x" Oct 30 23:40:45.867277 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.866968 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0d4810de-8bd2-4270-b907-341d9f840bb8-run-ovn\") pod \"ovnkube-node-26v6x\" (UID: \"0d4810de-8bd2-4270-b907-341d9f840bb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-26v6x" Oct 30 23:40:45.867277 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.866993 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0d4810de-8bd2-4270-b907-341d9f840bb8-host-cni-netd\") pod \"ovnkube-node-26v6x\" (UID: \"0d4810de-8bd2-4270-b907-341d9f840bb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-26v6x" Oct 30 23:40:45.867277 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.867004 2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0d4810de-8bd2-4270-b907-341d9f840bb8-host-cni-bin\") pod \"ovnkube-node-26v6x\" (UID: \"0d4810de-8bd2-4270-b907-341d9f840bb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-26v6x" Oct 30 23:40:45.867277 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.867020 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b22fe41c-cf52-4e97-81f1-2f64bcc43640-tuning-conf-dir\") pod \"multus-additional-cni-plugins-7mp5r\" (UID: \"b22fe41c-cf52-4e97-81f1-2f64bcc43640\") " pod="openshift-multus/multus-additional-cni-plugins-7mp5r" Oct 30 23:40:45.867277 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.867041 2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/953b9446-c289-48ca-b854-a14c70e5971b-lib-modules\") pod \"tuned-rsv7z\" (UID: \"953b9446-c289-48ca-b854-a14c70e5971b\") " pod="openshift-cluster-node-tuning-operator/tuned-rsv7z" Oct 30 23:40:45.867277 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.867046 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0d4810de-8bd2-4270-b907-341d9f840bb8-node-log\") pod \"ovnkube-node-26v6x\" (UID: \"0d4810de-8bd2-4270-b907-341d9f840bb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-26v6x" Oct 30 23:40:45.868072 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.867055 2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0d4810de-8bd2-4270-b907-341d9f840bb8-run-ovn\") pod \"ovnkube-node-26v6x\" (UID: \"0d4810de-8bd2-4270-b907-341d9f840bb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-26v6x" Oct 30 23:40:45.868072 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.867091 2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0d4810de-8bd2-4270-b907-341d9f840bb8-node-log\") pod \"ovnkube-node-26v6x\" (UID: \"0d4810de-8bd2-4270-b907-341d9f840bb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-26v6x" Oct 30 23:40:45.868072 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.867112 2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/953b9446-c289-48ca-b854-a14c70e5971b-etc-modprobe-d\") pod \"tuned-rsv7z\" (UID: \"953b9446-c289-48ca-b854-a14c70e5971b\") " pod="openshift-cluster-node-tuning-operator/tuned-rsv7z" Oct 30 23:40:45.868072 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.867111 2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0d4810de-8bd2-4270-b907-341d9f840bb8-host-cni-netd\") pod \"ovnkube-node-26v6x\" (UID: \"0d4810de-8bd2-4270-b907-341d9f840bb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-26v6x" Oct 30 23:40:45.868072 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.867166 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gtt94\" (UniqueName: \"kubernetes.io/projected/953b9446-c289-48ca-b854-a14c70e5971b-kube-api-access-gtt94\") pod \"tuned-rsv7z\" (UID: \"953b9446-c289-48ca-b854-a14c70e5971b\") " pod="openshift-cluster-node-tuning-operator/tuned-rsv7z" Oct 30 23:40:45.868072 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.867223 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0d4810de-8bd2-4270-b907-341d9f840bb8-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-26v6x\" (UID: \"0d4810de-8bd2-4270-b907-341d9f840bb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-26v6x" Oct 30 23:40:45.868072 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.867257 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/99f800b6-a38c-46ca-8234-0dbd3610c8fd-serviceca\") pod \"node-ca-7sbzf\" (UID: \"99f800b6-a38c-46ca-8234-0dbd3610c8fd\") " pod="openshift-image-registry/node-ca-7sbzf" Oct 30 23:40:45.868072 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.867284 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlhgb\" (UniqueName: \"kubernetes.io/projected/99f800b6-a38c-46ca-8234-0dbd3610c8fd-kube-api-access-mlhgb\") pod \"node-ca-7sbzf\" (UID: \"99f800b6-a38c-46ca-8234-0dbd3610c8fd\") " pod="openshift-image-registry/node-ca-7sbzf" Oct 30 23:40:45.868072 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.867299 2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0d4810de-8bd2-4270-b907-341d9f840bb8-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-26v6x\" (UID: \"0d4810de-8bd2-4270-b907-341d9f840bb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-26v6x" Oct 30 23:40:45.868072 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.867325 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b22fe41c-cf52-4e97-81f1-2f64bcc43640-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-7mp5r\" (UID: \"b22fe41c-cf52-4e97-81f1-2f64bcc43640\") " pod="openshift-multus/multus-additional-cni-plugins-7mp5r" Oct 30 23:40:45.868072 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.867485 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0d4810de-8bd2-4270-b907-341d9f840bb8-ovnkube-config\") pod \"ovnkube-node-26v6x\" (UID: \"0d4810de-8bd2-4270-b907-341d9f840bb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-26v6x" Oct 30 23:40:45.868072 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.867518 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b22fe41c-cf52-4e97-81f1-2f64bcc43640-cnibin\") pod \"multus-additional-cni-plugins-7mp5r\" (UID: \"b22fe41c-cf52-4e97-81f1-2f64bcc43640\") " pod="openshift-multus/multus-additional-cni-plugins-7mp5r" Oct 30 23:40:45.868072 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.867545 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/953b9446-c289-48ca-b854-a14c70e5971b-etc-kubernetes\") pod \"tuned-rsv7z\" (UID: \"953b9446-c289-48ca-b854-a14c70e5971b\") " pod="openshift-cluster-node-tuning-operator/tuned-rsv7z" Oct 30 23:40:45.868072 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.867572 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/953b9446-c289-48ca-b854-a14c70e5971b-etc-sysctl-d\") pod \"tuned-rsv7z\" (UID: \"953b9446-c289-48ca-b854-a14c70e5971b\") " pod="openshift-cluster-node-tuning-operator/tuned-rsv7z" Oct 30 23:40:45.868072 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.867598 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b22fe41c-cf52-4e97-81f1-2f64bcc43640-system-cni-dir\") pod \"multus-additional-cni-plugins-7mp5r\" (UID: \"b22fe41c-cf52-4e97-81f1-2f64bcc43640\") " pod="openshift-multus/multus-additional-cni-plugins-7mp5r" Oct 30 23:40:45.868072 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.867621 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0d4810de-8bd2-4270-b907-341d9f840bb8-host-run-ovn-kubernetes\") pod \"ovnkube-node-26v6x\" (UID: \"0d4810de-8bd2-4270-b907-341d9f840bb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-26v6x" Oct 30 23:40:45.868072 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.867639 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0d4810de-8bd2-4270-b907-341d9f840bb8-ovnkube-script-lib\") pod \"ovnkube-node-26v6x\" (UID: \"0d4810de-8bd2-4270-b907-341d9f840bb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-26v6x" Oct 30 23:40:45.868680 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.867642 2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8b33ecc-59a4-4680-9521-299648ea5055-tmp-dir\") pod \"node-resolver-tfnr7\" (UID: \"f8b33ecc-59a4-4680-9521-299648ea5055\") " pod="openshift-dns/node-resolver-tfnr7" Oct 30 23:40:45.868680 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.867669 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btgs8\" (UniqueName: \"kubernetes.io/projected/b22fe41c-cf52-4e97-81f1-2f64bcc43640-kube-api-access-btgs8\") pod \"multus-additional-cni-plugins-7mp5r\" (UID: \"b22fe41c-cf52-4e97-81f1-2f64bcc43640\") " pod="openshift-multus/multus-additional-cni-plugins-7mp5r" Oct 30 23:40:45.868680 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.867642 2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b22fe41c-cf52-4e97-81f1-2f64bcc43640-cnibin\") pod \"multus-additional-cni-plugins-7mp5r\" (UID: \"b22fe41c-cf52-4e97-81f1-2f64bcc43640\") " pod="openshift-multus/multus-additional-cni-plugins-7mp5r" Oct 30 23:40:45.868680 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.867702 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/97da2086-b00d-4f03-b502-56de5ee3d90d-original-pull-secret\") pod \"global-pull-secret-syncer-7wrh7\" (UID: \"97da2086-b00d-4f03-b502-56de5ee3d90d\") " pod="kube-system/global-pull-secret-syncer-7wrh7" Oct 30 23:40:45.868680 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.867715 2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b22fe41c-cf52-4e97-81f1-2f64bcc43640-system-cni-dir\") pod \"multus-additional-cni-plugins-7mp5r\" (UID: \"b22fe41c-cf52-4e97-81f1-2f64bcc43640\") " pod="openshift-multus/multus-additional-cni-plugins-7mp5r" Oct 30 23:40:45.868680 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.867730 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/953b9446-c289-48ca-b854-a14c70e5971b-host\") pod \"tuned-rsv7z\" (UID: \"953b9446-c289-48ca-b854-a14c70e5971b\") " pod="openshift-cluster-node-tuning-operator/tuned-rsv7z" Oct 30 23:40:45.868680 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.867756 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0d4810de-8bd2-4270-b907-341d9f840bb8-host-slash\") pod \"ovnkube-node-26v6x\" (UID: \"0d4810de-8bd2-4270-b907-341d9f840bb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-26v6x" Oct 30 23:40:45.868680 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.867776 2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/953b9446-c289-48ca-b854-a14c70e5971b-etc-kubernetes\") pod \"tuned-rsv7z\" (UID: \"953b9446-c289-48ca-b854-a14c70e5971b\") " pod="openshift-cluster-node-tuning-operator/tuned-rsv7z" Oct 30 23:40:45.868680 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.867791 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0d4810de-8bd2-4270-b907-341d9f840bb8-ovn-node-metrics-cert\") pod \"ovnkube-node-26v6x\" (UID: \"0d4810de-8bd2-4270-b907-341d9f840bb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-26v6x" Oct 30 23:40:45.868680 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.867831 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7p9rj\" (UniqueName: \"kubernetes.io/projected/f8b33ecc-59a4-4680-9521-299648ea5055-kube-api-access-7p9rj\") pod \"node-resolver-tfnr7\" (UID: \"f8b33ecc-59a4-4680-9521-299648ea5055\") " pod="openshift-dns/node-resolver-tfnr7" Oct 30 23:40:45.868680 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.867895 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0d4810de-8bd2-4270-b907-341d9f840bb8-systemd-units\") pod \"ovnkube-node-26v6x\" (UID: \"0d4810de-8bd2-4270-b907-341d9f840bb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-26v6x" Oct 30 23:40:45.868680 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.867903 2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/953b9446-c289-48ca-b854-a14c70e5971b-etc-sysctl-d\") pod \"tuned-rsv7z\" (UID: \"953b9446-c289-48ca-b854-a14c70e5971b\") " pod="openshift-cluster-node-tuning-operator/tuned-rsv7z" Oct 30 23:40:45.868680 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.867923 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0d4810de-8bd2-4270-b907-341d9f840bb8-var-lib-openvswitch\") pod \"ovnkube-node-26v6x\" (UID: \"0d4810de-8bd2-4270-b907-341d9f840bb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-26v6x" Oct 30 23:40:45.868680 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.867941 2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0d4810de-8bd2-4270-b907-341d9f840bb8-host-run-ovn-kubernetes\") pod \"ovnkube-node-26v6x\" (UID: \"0d4810de-8bd2-4270-b907-341d9f840bb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-26v6x" Oct 30 23:40:45.868680 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.867949 2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0d4810de-8bd2-4270-b907-341d9f840bb8-host-slash\") pod \"ovnkube-node-26v6x\" (UID: \"0d4810de-8bd2-4270-b907-341d9f840bb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-26v6x" Oct 30 23:40:45.868680 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.867951 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/97da2086-b00d-4f03-b502-56de5ee3d90d-kubelet-config\") pod \"global-pull-secret-syncer-7wrh7\" (UID: \"97da2086-b00d-4f03-b502-56de5ee3d90d\") " pod="kube-system/global-pull-secret-syncer-7wrh7" Oct 30 23:40:45.868680 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.867994 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/97da2086-b00d-4f03-b502-56de5ee3d90d-dbus\") pod \"global-pull-secret-syncer-7wrh7\" (UID: \"97da2086-b00d-4f03-b502-56de5ee3d90d\") " pod="kube-system/global-pull-secret-syncer-7wrh7" Oct 30 23:40:45.869195 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.868000 2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/97da2086-b00d-4f03-b502-56de5ee3d90d-kubelet-config\") pod \"global-pull-secret-syncer-7wrh7\" (UID: \"97da2086-b00d-4f03-b502-56de5ee3d90d\") " pod="kube-system/global-pull-secret-syncer-7wrh7" Oct 30 23:40:45.869195 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.868024 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b22fe41c-cf52-4e97-81f1-2f64bcc43640-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-7mp5r\" (UID: \"b22fe41c-cf52-4e97-81f1-2f64bcc43640\") " pod="openshift-multus/multus-additional-cni-plugins-7mp5r" Oct 30 23:40:45.869195 ip-10-0-141-180 kubenswrapper[2550]: E1030 23:40:45.868037 2550 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 23:40:45.869195 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.868045 2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0d4810de-8bd2-4270-b907-341d9f840bb8-systemd-units\") pod \"ovnkube-node-26v6x\" (UID: \"0d4810de-8bd2-4270-b907-341d9f840bb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-26v6x" Oct 30 23:40:45.869195 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.868058 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f8b33ecc-59a4-4680-9521-299648ea5055-hosts-file\") pod \"node-resolver-tfnr7\" (UID: \"f8b33ecc-59a4-4680-9521-299648ea5055\") " pod="openshift-dns/node-resolver-tfnr7" Oct 30 23:40:45.869195 ip-10-0-141-180 kubenswrapper[2550]: E1030 23:40:45.868084 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/97da2086-b00d-4f03-b502-56de5ee3d90d-original-pull-secret podName:97da2086-b00d-4f03-b502-56de5ee3d90d nodeName:}" failed. No retries permitted until 2026-10-30 23:40:46.368071353 +0000 UTC m=+2.426275249 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/97da2086-b00d-4f03-b502-56de5ee3d90d-original-pull-secret") pod "global-pull-secret-syncer-7wrh7" (UID: "97da2086-b00d-4f03-b502-56de5ee3d90d") : object "kube-system"/"original-pull-secret" not registered Oct 30 23:40:45.869195 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.868087 2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0d4810de-8bd2-4270-b907-341d9f840bb8-var-lib-openvswitch\") pod \"ovnkube-node-26v6x\" (UID: \"0d4810de-8bd2-4270-b907-341d9f840bb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-26v6x" Oct 30 23:40:45.869195 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.868101 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/953b9446-c289-48ca-b854-a14c70e5971b-etc-sysctl-conf\") pod \"tuned-rsv7z\" (UID: \"953b9446-c289-48ca-b854-a14c70e5971b\") " pod="openshift-cluster-node-tuning-operator/tuned-rsv7z" Oct 30 23:40:45.869195 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.868135 2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/953b9446-c289-48ca-b854-a14c70e5971b-host\") pod \"tuned-rsv7z\" (UID: \"953b9446-c289-48ca-b854-a14c70e5971b\") " pod="openshift-cluster-node-tuning-operator/tuned-rsv7z" Oct 30 23:40:45.869195 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.868175 2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f8b33ecc-59a4-4680-9521-299648ea5055-hosts-file\") pod \"node-resolver-tfnr7\" (UID: \"f8b33ecc-59a4-4680-9521-299648ea5055\") " pod="openshift-dns/node-resolver-tfnr7" Oct 30 23:40:45.869195 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.868229 2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/97da2086-b00d-4f03-b502-56de5ee3d90d-dbus\") pod \"global-pull-secret-syncer-7wrh7\" (UID: \"97da2086-b00d-4f03-b502-56de5ee3d90d\") " pod="kube-system/global-pull-secret-syncer-7wrh7" Oct 30 23:40:45.869195 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.868243 2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/953b9446-c289-48ca-b854-a14c70e5971b-etc-sysctl-conf\") pod \"tuned-rsv7z\" (UID: \"953b9446-c289-48ca-b854-a14c70e5971b\") " pod="openshift-cluster-node-tuning-operator/tuned-rsv7z" Oct 30 23:40:45.869808 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.869383 2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0d4810de-8bd2-4270-b907-341d9f840bb8-env-overrides\") pod \"ovnkube-node-26v6x\" (UID: \"0d4810de-8bd2-4270-b907-341d9f840bb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-26v6x" Oct 30 23:40:45.869808 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.869383 2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b22fe41c-cf52-4e97-81f1-2f64bcc43640-cni-binary-copy\") pod \"multus-additional-cni-plugins-7mp5r\" (UID: \"b22fe41c-cf52-4e97-81f1-2f64bcc43640\") " pod="openshift-multus/multus-additional-cni-plugins-7mp5r" Oct 30 23:40:45.869808 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.869553 2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/953b9446-c289-48ca-b854-a14c70e5971b-etc-tuned\") pod \"tuned-rsv7z\" (UID: \"953b9446-c289-48ca-b854-a14c70e5971b\") " pod="openshift-cluster-node-tuning-operator/tuned-rsv7z" Oct 30 23:40:45.869808 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.869682 2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/99f800b6-a38c-46ca-8234-0dbd3610c8fd-serviceca\") pod \"node-ca-7sbzf\" (UID: \"99f800b6-a38c-46ca-8234-0dbd3610c8fd\") " pod="openshift-image-registry/node-ca-7sbzf" Oct 30 23:40:45.869808 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.869703 2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b22fe41c-cf52-4e97-81f1-2f64bcc43640-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-7mp5r\" (UID: \"b22fe41c-cf52-4e97-81f1-2f64bcc43640\") " pod="openshift-multus/multus-additional-cni-plugins-7mp5r" Oct 30 23:40:45.869808 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.869715 2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0d4810de-8bd2-4270-b907-341d9f840bb8-ovnkube-config\") pod \"ovnkube-node-26v6x\" (UID: \"0d4810de-8bd2-4270-b907-341d9f840bb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-26v6x" Oct 30 23:40:45.869808 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.869743 2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b22fe41c-cf52-4e97-81f1-2f64bcc43640-tuning-conf-dir\") pod \"multus-additional-cni-plugins-7mp5r\" (UID: \"b22fe41c-cf52-4e97-81f1-2f64bcc43640\") " pod="openshift-multus/multus-additional-cni-plugins-7mp5r" Oct 30 23:40:45.870039 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.870010 2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b22fe41c-cf52-4e97-81f1-2f64bcc43640-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-7mp5r\" (UID: \"b22fe41c-cf52-4e97-81f1-2f64bcc43640\") " pod="openshift-multus/multus-additional-cni-plugins-7mp5r" Oct 30 23:40:45.870368 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.870340 2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0d4810de-8bd2-4270-b907-341d9f840bb8-ovnkube-script-lib\") pod \"ovnkube-node-26v6x\" (UID: \"0d4810de-8bd2-4270-b907-341d9f840bb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-26v6x" Oct 30 23:40:45.871077 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.871055 2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/953b9446-c289-48ca-b854-a14c70e5971b-tmp\") pod \"tuned-rsv7z\" (UID: \"953b9446-c289-48ca-b854-a14c70e5971b\") " pod="openshift-cluster-node-tuning-operator/tuned-rsv7z" Oct 30 23:40:45.871897 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.871877 2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0d4810de-8bd2-4270-b907-341d9f840bb8-ovn-node-metrics-cert\") pod \"ovnkube-node-26v6x\" (UID: \"0d4810de-8bd2-4270-b907-341d9f840bb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-26v6x" Oct 30 23:40:45.934581 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.934554 2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-btgs8\" (UniqueName: \"kubernetes.io/projected/b22fe41c-cf52-4e97-81f1-2f64bcc43640-kube-api-access-btgs8\") pod \"multus-additional-cni-plugins-7mp5r\" (UID: \"b22fe41c-cf52-4e97-81f1-2f64bcc43640\") " pod="openshift-multus/multus-additional-cni-plugins-7mp5r" Oct 30 23:40:45.934731 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.934593 2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7p9rj\" (UniqueName: \"kubernetes.io/projected/f8b33ecc-59a4-4680-9521-299648ea5055-kube-api-access-7p9rj\") pod \"node-resolver-tfnr7\" (UID: \"f8b33ecc-59a4-4680-9521-299648ea5055\") " pod="openshift-dns/node-resolver-tfnr7" Oct 30 23:40:45.934731 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.934693 2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mlhgb\" (UniqueName: \"kubernetes.io/projected/99f800b6-a38c-46ca-8234-0dbd3610c8fd-kube-api-access-mlhgb\") pod \"node-ca-7sbzf\" (UID: \"99f800b6-a38c-46ca-8234-0dbd3610c8fd\") " pod="openshift-image-registry/node-ca-7sbzf" Oct 30 23:40:45.935291 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.935268 2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-48m7s\" (UniqueName: \"kubernetes.io/projected/0d4810de-8bd2-4270-b907-341d9f840bb8-kube-api-access-48m7s\") pod \"ovnkube-node-26v6x\" (UID: \"0d4810de-8bd2-4270-b907-341d9f840bb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-26v6x" Oct 30 23:40:45.935379 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.935311 2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gtt94\" (UniqueName: \"kubernetes.io/projected/953b9446-c289-48ca-b854-a14c70e5971b-kube-api-access-gtt94\") pod \"tuned-rsv7z\" (UID: \"953b9446-c289-48ca-b854-a14c70e5971b\") " pod="openshift-cluster-node-tuning-operator/tuned-rsv7z" Oct 30 23:40:45.972929 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.972893 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-57xrk" Oct 30 23:40:45.996367 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.996341 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j7n2l" Oct 30 23:40:45.997502 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:45.997482 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-cb9vj" Oct 30 23:40:46.027141 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:46.027119 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-9s2fx" Oct 30 23:40:46.036506 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:46.036489 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-tfnr7" Oct 30 23:40:46.042420 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:46.042390 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-rsv7z" Oct 30 23:40:46.083585 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:46.083555 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-7mp5r" Oct 30 23:40:46.085270 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:46.085242 2550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod953b9446_c289_48ca_b854_a14c70e5971b.slice/crio-29f7c6a8c313b55f51d0728424ff0d32219fce78a38d354850bf97e539815d8e WatchSource:0}: Error finding container 29f7c6a8c313b55f51d0728424ff0d32219fce78a38d354850bf97e539815d8e: Status 404 returned error can't find the container with id 29f7c6a8c313b55f51d0728424ff0d32219fce78a38d354850bf97e539815d8e Oct 30 23:40:46.086623 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:46.086592 2550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd028af9af4f41b9772b02bdb5343dc5.slice/crio-4357c8b1b1cd7c31d12ef70138214a17b84944e2afc3d6d78932c37e9743b036 WatchSource:0}: Error finding container 4357c8b1b1cd7c31d12ef70138214a17b84944e2afc3d6d78932c37e9743b036: Status 404 returned error can't find the container with id 4357c8b1b1cd7c31d12ef70138214a17b84944e2afc3d6d78932c37e9743b036 Oct 30 23:40:46.086922 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:46.086904 2550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c1c7fe71dc5ea64e6fbc8dd666819fa.slice/crio-154e4407f570053087b1f80f38a62a9607d8146fbbadeb07629573256de48e7e WatchSource:0}: Error finding container 154e4407f570053087b1f80f38a62a9607d8146fbbadeb07629573256de48e7e: Status 404 returned error can't find the container with id 154e4407f570053087b1f80f38a62a9607d8146fbbadeb07629573256de48e7e Oct 30 23:40:46.087526 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:46.087502 2550 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 23:40:46.087974 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:46.087951 2550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8b33ecc_59a4_4680_9521_299648ea5055.slice/crio-a85a7532289c5eee82ff2b7f93b6834d6ceba975a039ef17c247d87a33699c05 WatchSource:0}: Error finding container a85a7532289c5eee82ff2b7f93b6834d6ceba975a039ef17c247d87a33699c05: Status 404 returned error can't find the container with id a85a7532289c5eee82ff2b7f93b6834d6ceba975a039ef17c247d87a33699c05 Oct 30 23:40:46.091738 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:46.091721 2550 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 23:40:46.093973 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:46.093938 2550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb22fe41c_cf52_4e97_81f1_2f64bcc43640.slice/crio-c374b94e0a85533ff7652b5def95a433a127c7f26a0f314e854e8962450dae40 WatchSource:0}: Error finding container c374b94e0a85533ff7652b5def95a433a127c7f26a0f314e854e8962450dae40: Status 404 returned error can't find the container with id c374b94e0a85533ff7652b5def95a433a127c7f26a0f314e854e8962450dae40 Oct 30 23:40:46.100071 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:46.100053 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-7sbzf" Oct 30 23:40:46.106345 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:46.106327 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-26v6x" Oct 30 23:40:46.108945 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:46.108924 2550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99f800b6_a38c_46ca_8234_0dbd3610c8fd.slice/crio-d353d4d879e678a52dad33597146b7e44151f5f2886f5c2c28a1cdddea04459b WatchSource:0}: Error finding container d353d4d879e678a52dad33597146b7e44151f5f2886f5c2c28a1cdddea04459b: Status 404 returned error can't find the container with id d353d4d879e678a52dad33597146b7e44151f5f2886f5c2c28a1cdddea04459b Oct 30 23:40:46.114496 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:46.114464 2550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d4810de_8bd2_4270_b907_341d9f840bb8.slice/crio-c80c7448baf9962aeb89efcadbfe35a76110a83280e99ce6a7d38c1eb89cac18 WatchSource:0}: Error finding container c80c7448baf9962aeb89efcadbfe35a76110a83280e99ce6a7d38c1eb89cac18: Status 404 returned error can't find the container with id c80c7448baf9962aeb89efcadbfe35a76110a83280e99ce6a7d38c1eb89cac18 Oct 30 23:40:46.271156 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:46.271124 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/145dd323-9370-4b9d-a96e-8b93d0a47a7c-metrics-certs\") pod \"network-metrics-daemon-pt58s\" (UID: \"145dd323-9370-4b9d-a96e-8b93d0a47a7c\") " pod="openshift-multus/network-metrics-daemon-pt58s" Oct 30 23:40:46.271322 ip-10-0-141-180 kubenswrapper[2550]: E1030 23:40:46.271234 2550 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 23:40:46.271322 ip-10-0-141-180 kubenswrapper[2550]: E1030 23:40:46.271293 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/145dd323-9370-4b9d-a96e-8b93d0a47a7c-metrics-certs podName:145dd323-9370-4b9d-a96e-8b93d0a47a7c nodeName:}" failed. No retries permitted until 2026-10-30 23:40:47.271268469 +0000 UTC m=+3.329472364 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/145dd323-9370-4b9d-a96e-8b93d0a47a7c-metrics-certs") pod "network-metrics-daemon-pt58s" (UID: "145dd323-9370-4b9d-a96e-8b93d0a47a7c") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 23:40:46.372241 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:46.372156 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/97da2086-b00d-4f03-b502-56de5ee3d90d-original-pull-secret\") pod \"global-pull-secret-syncer-7wrh7\" (UID: \"97da2086-b00d-4f03-b502-56de5ee3d90d\") " pod="kube-system/global-pull-secret-syncer-7wrh7" Oct 30 23:40:46.372241 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:46.372226 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqmnp\" (UniqueName: \"kubernetes.io/projected/7728abfe-1747-422e-b8e2-c91c9de83d09-kube-api-access-jqmnp\") pod \"network-check-target-kgdzs\" (UID: \"7728abfe-1747-422e-b8e2-c91c9de83d09\") " pod="openshift-network-diagnostics/network-check-target-kgdzs" Oct 30 23:40:46.372899 ip-10-0-141-180 kubenswrapper[2550]: E1030 23:40:46.372330 2550 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 23:40:46.372899 ip-10-0-141-180 kubenswrapper[2550]: E1030 23:40:46.372436 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/97da2086-b00d-4f03-b502-56de5ee3d90d-original-pull-secret podName:97da2086-b00d-4f03-b502-56de5ee3d90d nodeName:}" failed. No retries permitted until 2026-10-30 23:40:47.372406517 +0000 UTC m=+3.430610426 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/97da2086-b00d-4f03-b502-56de5ee3d90d-original-pull-secret") pod "global-pull-secret-syncer-7wrh7" (UID: "97da2086-b00d-4f03-b502-56de5ee3d90d") : object "kube-system"/"original-pull-secret" not registered Oct 30 23:40:46.372899 ip-10-0-141-180 kubenswrapper[2550]: E1030 23:40:46.372344 2550 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 23:40:46.372899 ip-10-0-141-180 kubenswrapper[2550]: E1030 23:40:46.372474 2550 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 23:40:46.372899 ip-10-0-141-180 kubenswrapper[2550]: E1030 23:40:46.372486 2550 projected.go:196] Error preparing data for projected volume kube-api-access-jqmnp for pod openshift-network-diagnostics/network-check-target-kgdzs: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 23:40:46.372899 ip-10-0-141-180 kubenswrapper[2550]: E1030 23:40:46.372533 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7728abfe-1747-422e-b8e2-c91c9de83d09-kube-api-access-jqmnp podName:7728abfe-1747-422e-b8e2-c91c9de83d09 nodeName:}" failed. No retries permitted until 2026-10-30 23:40:47.372522539 +0000 UTC m=+3.430726433 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-jqmnp" (UniqueName: "kubernetes.io/projected/7728abfe-1747-422e-b8e2-c91c9de83d09-kube-api-access-jqmnp") pod "network-check-target-kgdzs" (UID: "7728abfe-1747-422e-b8e2-c91c9de83d09") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 23:40:46.426796 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:46.426715 2550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-tfnr7" event={"ID":"f8b33ecc-59a4-4680-9521-299648ea5055","Type":"ContainerStarted","Data":"a85a7532289c5eee82ff2b7f93b6834d6ceba975a039ef17c247d87a33699c05"} Oct 30 23:40:46.427847 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:46.427814 2550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-26v6x" event={"ID":"0d4810de-8bd2-4270-b907-341d9f840bb8","Type":"ContainerStarted","Data":"c80c7448baf9962aeb89efcadbfe35a76110a83280e99ce6a7d38c1eb89cac18"} Oct 30 23:40:46.428937 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:46.428914 2550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-7sbzf" event={"ID":"99f800b6-a38c-46ca-8234-0dbd3610c8fd","Type":"ContainerStarted","Data":"d353d4d879e678a52dad33597146b7e44151f5f2886f5c2c28a1cdddea04459b"} Oct 30 23:40:46.430042 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:46.430022 2550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7mp5r" event={"ID":"b22fe41c-cf52-4e97-81f1-2f64bcc43640","Type":"ContainerStarted","Data":"c374b94e0a85533ff7652b5def95a433a127c7f26a0f314e854e8962450dae40"} Oct 30 23:40:46.431095 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:46.431074 2550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-180.us-west-2.compute.internal" event={"ID":"fd028af9af4f41b9772b02bdb5343dc5","Type":"ContainerStarted","Data":"4357c8b1b1cd7c31d12ef70138214a17b84944e2afc3d6d78932c37e9743b036"} Oct 30 23:40:46.432166 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:46.432125 2550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-180.us-west-2.compute.internal" event={"ID":"2c1c7fe71dc5ea64e6fbc8dd666819fa","Type":"ContainerStarted","Data":"154e4407f570053087b1f80f38a62a9607d8146fbbadeb07629573256de48e7e"} Oct 30 23:40:46.433050 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:46.433031 2550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-rsv7z" event={"ID":"953b9446-c289-48ca-b854-a14c70e5971b","Type":"ContainerStarted","Data":"29f7c6a8c313b55f51d0728424ff0d32219fce78a38d354850bf97e539815d8e"} Oct 30 23:40:46.478634 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:46.478609 2550 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 23:40:46.755964 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:46.755853 2550 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 23:35:46 +0000 UTC" deadline="2028-04-27 06:10:40.26374921 +0000 UTC" Oct 30 23:40:46.755964 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:46.755902 2550 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13062h29m53.507849834s" Oct 30 23:40:46.868719 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:46.868680 2550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podece07686_4d8e_4068_9fb2_5889046cb73b.slice/crio-0a3626cfde5478d63a59c0432c8cac480a568eb5bcbdf0fcbd33e342c40e75dd WatchSource:0}: Error finding container 0a3626cfde5478d63a59c0432c8cac480a568eb5bcbdf0fcbd33e342c40e75dd: Status 404 returned error can't find the container with id 0a3626cfde5478d63a59c0432c8cac480a568eb5bcbdf0fcbd33e342c40e75dd Oct 30 23:40:46.936124 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:46.936096 2550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fa7d24b_67b0_400f_94d4_903feb268782.slice/crio-020dec11112038ed834b49f31a2d35496a872cbc95e5b9692e8bf311a59c5260 WatchSource:0}: Error finding container 020dec11112038ed834b49f31a2d35496a872cbc95e5b9692e8bf311a59c5260: Status 404 returned error can't find the container with id 020dec11112038ed834b49f31a2d35496a872cbc95e5b9692e8bf311a59c5260 Oct 30 23:40:46.998522 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:46.998486 2550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8e4c4db_197e_420c_9c60_4273cdf95377.slice/crio-69619f4a1d594665a6b7373c4f0480dec83db882660369456758935c0ee3aebd WatchSource:0}: Error finding container 69619f4a1d594665a6b7373c4f0480dec83db882660369456758935c0ee3aebd: Status 404 returned error can't find the container with id 69619f4a1d594665a6b7373c4f0480dec83db882660369456758935c0ee3aebd Oct 30 23:40:47.038668 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:40:47.038623 2550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0796261f_298e_4cbe_9b3e_b3dcf4528e07.slice/crio-1dda77b536c5cf8fe095e91e3a2da765111f2116770dcb972dc5b0b76134b542 WatchSource:0}: Error finding container 1dda77b536c5cf8fe095e91e3a2da765111f2116770dcb972dc5b0b76134b542: Status 404 returned error can't find the container with id 1dda77b536c5cf8fe095e91e3a2da765111f2116770dcb972dc5b0b76134b542 Oct 30 23:40:47.281307 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:47.281263 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/145dd323-9370-4b9d-a96e-8b93d0a47a7c-metrics-certs\") pod \"network-metrics-daemon-pt58s\" (UID: \"145dd323-9370-4b9d-a96e-8b93d0a47a7c\") " pod="openshift-multus/network-metrics-daemon-pt58s" Oct 30 23:40:47.281569 ip-10-0-141-180 kubenswrapper[2550]: E1030 23:40:47.281456 2550 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 23:40:47.281569 ip-10-0-141-180 kubenswrapper[2550]: E1030 23:40:47.281538 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/145dd323-9370-4b9d-a96e-8b93d0a47a7c-metrics-certs podName:145dd323-9370-4b9d-a96e-8b93d0a47a7c nodeName:}" failed. No retries permitted until 2026-10-30 23:40:49.281516092 +0000 UTC m=+5.339720003 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/145dd323-9370-4b9d-a96e-8b93d0a47a7c-metrics-certs") pod "network-metrics-daemon-pt58s" (UID: "145dd323-9370-4b9d-a96e-8b93d0a47a7c") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 23:40:47.383824 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:47.383733 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqmnp\" (UniqueName: \"kubernetes.io/projected/7728abfe-1747-422e-b8e2-c91c9de83d09-kube-api-access-jqmnp\") pod \"network-check-target-kgdzs\" (UID: \"7728abfe-1747-422e-b8e2-c91c9de83d09\") " pod="openshift-network-diagnostics/network-check-target-kgdzs" Oct 30 23:40:47.383824 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:47.383803 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/97da2086-b00d-4f03-b502-56de5ee3d90d-original-pull-secret\") pod \"global-pull-secret-syncer-7wrh7\" (UID: \"97da2086-b00d-4f03-b502-56de5ee3d90d\") " pod="kube-system/global-pull-secret-syncer-7wrh7" Oct 30 23:40:47.384553 ip-10-0-141-180 kubenswrapper[2550]: E1030 23:40:47.383975 2550 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 23:40:47.384553 ip-10-0-141-180 kubenswrapper[2550]: E1030 23:40:47.384038 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/97da2086-b00d-4f03-b502-56de5ee3d90d-original-pull-secret podName:97da2086-b00d-4f03-b502-56de5ee3d90d nodeName:}" failed. No retries permitted until 2026-10-30 23:40:49.384020376 +0000 UTC m=+5.442224274 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/97da2086-b00d-4f03-b502-56de5ee3d90d-original-pull-secret") pod "global-pull-secret-syncer-7wrh7" (UID: "97da2086-b00d-4f03-b502-56de5ee3d90d") : object "kube-system"/"original-pull-secret" not registered Oct 30 23:40:47.384553 ip-10-0-141-180 kubenswrapper[2550]: E1030 23:40:47.384127 2550 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 23:40:47.384553 ip-10-0-141-180 kubenswrapper[2550]: E1030 23:40:47.384151 2550 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 23:40:47.384553 ip-10-0-141-180 kubenswrapper[2550]: E1030 23:40:47.384166 2550 projected.go:196] Error preparing data for projected volume kube-api-access-jqmnp for pod openshift-network-diagnostics/network-check-target-kgdzs: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 23:40:47.384553 ip-10-0-141-180 kubenswrapper[2550]: E1030 23:40:47.384203 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7728abfe-1747-422e-b8e2-c91c9de83d09-kube-api-access-jqmnp podName:7728abfe-1747-422e-b8e2-c91c9de83d09 nodeName:}" failed. No retries permitted until 2026-10-30 23:40:49.384191954 +0000 UTC m=+5.442395854 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-jqmnp" (UniqueName: "kubernetes.io/projected/7728abfe-1747-422e-b8e2-c91c9de83d09-kube-api-access-jqmnp") pod "network-check-target-kgdzs" (UID: "7728abfe-1747-422e-b8e2-c91c9de83d09") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 23:40:47.423089 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:47.423053 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7wrh7" Oct 30 23:40:47.423270 ip-10-0-141-180 kubenswrapper[2550]: E1030 23:40:47.423198 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7wrh7" podUID="97da2086-b00d-4f03-b502-56de5ee3d90d" Oct 30 23:40:47.423383 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:47.423291 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pt58s" Oct 30 23:40:47.423473 ip-10-0-141-180 kubenswrapper[2550]: E1030 23:40:47.423434 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pt58s" podUID="145dd323-9370-4b9d-a96e-8b93d0a47a7c" Oct 30 23:40:47.423541 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:47.423533 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kgdzs" Oct 30 23:40:47.423645 ip-10-0-141-180 kubenswrapper[2550]: E1030 23:40:47.423608 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kgdzs" podUID="7728abfe-1747-422e-b8e2-c91c9de83d09" Oct 30 23:40:47.451574 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:47.451537 2550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-9s2fx" event={"ID":"a8e4c4db-197e-420c-9c60-4273cdf95377","Type":"ContainerStarted","Data":"69619f4a1d594665a6b7373c4f0480dec83db882660369456758935c0ee3aebd"} Oct 30 23:40:47.458103 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:47.457861 2550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j7n2l" event={"ID":"0796261f-298e-4cbe-9b3e-b3dcf4528e07","Type":"ContainerStarted","Data":"1dda77b536c5cf8fe095e91e3a2da765111f2116770dcb972dc5b0b76134b542"} Oct 30 23:40:47.461615 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:47.460911 2550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-cb9vj" event={"ID":"2fa7d24b-67b0-400f-94d4-903feb268782","Type":"ContainerStarted","Data":"020dec11112038ed834b49f31a2d35496a872cbc95e5b9692e8bf311a59c5260"} Oct 30 23:40:47.463438 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:47.462342 2550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-57xrk" event={"ID":"ece07686-4d8e-4068-9fb2-5889046cb73b","Type":"ContainerStarted","Data":"0a3626cfde5478d63a59c0432c8cac480a568eb5bcbdf0fcbd33e342c40e75dd"} Oct 30 23:40:47.756938 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:47.756894 2550 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 23:35:46 +0000 UTC" deadline="2028-07-22 01:45:35.188788136 +0000 UTC" Oct 30 23:40:47.756938 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:47.756938 2550 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15122h4m47.4318561s" Oct 30 23:40:49.301425 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:49.300936 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/145dd323-9370-4b9d-a96e-8b93d0a47a7c-metrics-certs\") pod \"network-metrics-daemon-pt58s\" (UID: \"145dd323-9370-4b9d-a96e-8b93d0a47a7c\") " pod="openshift-multus/network-metrics-daemon-pt58s" Oct 30 23:40:49.301425 ip-10-0-141-180 kubenswrapper[2550]: E1030 23:40:49.301079 2550 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 23:40:49.301425 ip-10-0-141-180 kubenswrapper[2550]: E1030 23:40:49.301132 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/145dd323-9370-4b9d-a96e-8b93d0a47a7c-metrics-certs podName:145dd323-9370-4b9d-a96e-8b93d0a47a7c nodeName:}" failed. No retries permitted until 2026-10-30 23:40:53.301118415 +0000 UTC m=+9.359322323 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/145dd323-9370-4b9d-a96e-8b93d0a47a7c-metrics-certs") pod "network-metrics-daemon-pt58s" (UID: "145dd323-9370-4b9d-a96e-8b93d0a47a7c") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 23:40:49.402247 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:49.402195 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqmnp\" (UniqueName: \"kubernetes.io/projected/7728abfe-1747-422e-b8e2-c91c9de83d09-kube-api-access-jqmnp\") pod \"network-check-target-kgdzs\" (UID: \"7728abfe-1747-422e-b8e2-c91c9de83d09\") " pod="openshift-network-diagnostics/network-check-target-kgdzs" Oct 30 23:40:49.402467 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:49.402268 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/97da2086-b00d-4f03-b502-56de5ee3d90d-original-pull-secret\") pod \"global-pull-secret-syncer-7wrh7\" (UID: \"97da2086-b00d-4f03-b502-56de5ee3d90d\") " pod="kube-system/global-pull-secret-syncer-7wrh7" Oct 30 23:40:49.402467 ip-10-0-141-180 kubenswrapper[2550]: E1030 23:40:49.402449 2550 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 23:40:49.402600 ip-10-0-141-180 kubenswrapper[2550]: E1030 23:40:49.402523 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/97da2086-b00d-4f03-b502-56de5ee3d90d-original-pull-secret podName:97da2086-b00d-4f03-b502-56de5ee3d90d nodeName:}" failed. No retries permitted until 2026-10-30 23:40:53.402501634 +0000 UTC m=+9.460705536 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/97da2086-b00d-4f03-b502-56de5ee3d90d-original-pull-secret") pod "global-pull-secret-syncer-7wrh7" (UID: "97da2086-b00d-4f03-b502-56de5ee3d90d") : object "kube-system"/"original-pull-secret" not registered Oct 30 23:40:49.402673 ip-10-0-141-180 kubenswrapper[2550]: E1030 23:40:49.402633 2550 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 23:40:49.402673 ip-10-0-141-180 kubenswrapper[2550]: E1030 23:40:49.402652 2550 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 23:40:49.402673 ip-10-0-141-180 kubenswrapper[2550]: E1030 23:40:49.402665 2550 projected.go:196] Error preparing data for projected volume kube-api-access-jqmnp for pod openshift-network-diagnostics/network-check-target-kgdzs: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 23:40:49.402820 ip-10-0-141-180 kubenswrapper[2550]: E1030 23:40:49.402711 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7728abfe-1747-422e-b8e2-c91c9de83d09-kube-api-access-jqmnp podName:7728abfe-1747-422e-b8e2-c91c9de83d09 nodeName:}" failed. No retries permitted until 2026-10-30 23:40:53.402697615 +0000 UTC m=+9.460901512 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-jqmnp" (UniqueName: "kubernetes.io/projected/7728abfe-1747-422e-b8e2-c91c9de83d09-kube-api-access-jqmnp") pod "network-check-target-kgdzs" (UID: "7728abfe-1747-422e-b8e2-c91c9de83d09") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 23:40:49.423815 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:49.423116 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pt58s" Oct 30 23:40:49.423815 ip-10-0-141-180 kubenswrapper[2550]: E1030 23:40:49.423261 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pt58s" podUID="145dd323-9370-4b9d-a96e-8b93d0a47a7c" Oct 30 23:40:49.423815 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:49.423352 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kgdzs" Oct 30 23:40:49.423815 ip-10-0-141-180 kubenswrapper[2550]: E1030 23:40:49.423452 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kgdzs" podUID="7728abfe-1747-422e-b8e2-c91c9de83d09" Oct 30 23:40:49.423815 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:49.423506 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7wrh7" Oct 30 23:40:49.423815 ip-10-0-141-180 kubenswrapper[2550]: E1030 23:40:49.423584 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7wrh7" podUID="97da2086-b00d-4f03-b502-56de5ee3d90d" Oct 30 23:40:51.423731 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:51.423012 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pt58s" Oct 30 23:40:51.423731 ip-10-0-141-180 kubenswrapper[2550]: E1030 23:40:51.423421 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pt58s" podUID="145dd323-9370-4b9d-a96e-8b93d0a47a7c" Oct 30 23:40:51.423731 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:51.423500 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kgdzs" Oct 30 23:40:51.423731 ip-10-0-141-180 kubenswrapper[2550]: E1030 23:40:51.423577 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kgdzs" podUID="7728abfe-1747-422e-b8e2-c91c9de83d09" Oct 30 23:40:51.423731 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:51.423623 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7wrh7" Oct 30 23:40:51.423731 ip-10-0-141-180 kubenswrapper[2550]: E1030 23:40:51.423693 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7wrh7" podUID="97da2086-b00d-4f03-b502-56de5ee3d90d" Oct 30 23:40:51.500275 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:51.499507 2550 generic.go:356] "Generic (PLEG): container finished" podID="fd028af9af4f41b9772b02bdb5343dc5" containerID="0cbd49cfc73086c0ebe144eeead70ae2fe4039fe472d088d347917b95fa867ea" exitCode=0 Oct 30 23:40:51.500275 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:51.499582 2550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-180.us-west-2.compute.internal" event={"ID":"fd028af9af4f41b9772b02bdb5343dc5","Type":"ContainerDied","Data":"0cbd49cfc73086c0ebe144eeead70ae2fe4039fe472d088d347917b95fa867ea"} Oct 30 23:40:51.507164 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:51.506536 2550 generic.go:356] "Generic (PLEG): container finished" podID="b22fe41c-cf52-4e97-81f1-2f64bcc43640" containerID="ce628f435799ce877d768d21adb4750878b8e18381849400b2afe380bb635067" exitCode=0 Oct 30 23:40:51.507164 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:51.506577 2550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7mp5r" event={"ID":"b22fe41c-cf52-4e97-81f1-2f64bcc43640","Type":"ContainerDied","Data":"ce628f435799ce877d768d21adb4750878b8e18381849400b2afe380bb635067"} Oct 30 23:40:52.516000 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:52.515118 2550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-180.us-west-2.compute.internal" event={"ID":"fd028af9af4f41b9772b02bdb5343dc5","Type":"ContainerStarted","Data":"45a922e83b9d3d59575a0f5f53ff8c6543356a7dd0694833ed3ce55429beb9fe"} Oct 30 23:40:52.519531 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:52.518901 2550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-9s2fx" event={"ID":"a8e4c4db-197e-420c-9c60-4273cdf95377","Type":"ContainerStarted","Data":"24ce9bd768757c2e96a5e192b4caa99339ba91a537c22c5236c578cb61f0287f"} Oct 30 23:40:52.521754 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:52.521728 2550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-180.us-west-2.compute.internal" event={"ID":"2c1c7fe71dc5ea64e6fbc8dd666819fa","Type":"ContainerStarted","Data":"2043c7d133b2def891e736545ab2908e1466262942abb4176193031621dd73b7"} Oct 30 23:40:52.769224 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:52.769119 2550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-180.us-west-2.compute.internal" podStartSLOduration=7.769097109 podStartE2EDuration="7.769097109s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 23:40:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:40:52.643484327 +0000 UTC m=+8.701688247" watchObservedRunningTime="2026-10-30 23:40:52.769097109 +0000 UTC m=+8.827301029" Oct 30 23:40:52.897338 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:52.897275 2550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-180.us-west-2.compute.internal" podStartSLOduration=7.8972510289999995 podStartE2EDuration="7.897251029s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 23:40:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:40:52.76998224 +0000 UTC m=+8.828186162" watchObservedRunningTime="2026-10-30 23:40:52.897251029 +0000 UTC m=+8.955454948" Oct 30 23:40:53.338065 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:53.338030 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/145dd323-9370-4b9d-a96e-8b93d0a47a7c-metrics-certs\") pod \"network-metrics-daemon-pt58s\" (UID: \"145dd323-9370-4b9d-a96e-8b93d0a47a7c\") " pod="openshift-multus/network-metrics-daemon-pt58s" Oct 30 23:40:53.338233 ip-10-0-141-180 kubenswrapper[2550]: E1030 23:40:53.338198 2550 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 23:40:53.338293 ip-10-0-141-180 kubenswrapper[2550]: E1030 23:40:53.338269 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/145dd323-9370-4b9d-a96e-8b93d0a47a7c-metrics-certs podName:145dd323-9370-4b9d-a96e-8b93d0a47a7c nodeName:}" failed. No retries permitted until 2026-10-30 23:41:01.338250406 +0000 UTC m=+17.396454300 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/145dd323-9370-4b9d-a96e-8b93d0a47a7c-metrics-certs") pod "network-metrics-daemon-pt58s" (UID: "145dd323-9370-4b9d-a96e-8b93d0a47a7c") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 23:40:53.422842 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:53.422806 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pt58s" Oct 30 23:40:53.423005 ip-10-0-141-180 kubenswrapper[2550]: E1030 23:40:53.422973 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pt58s" podUID="145dd323-9370-4b9d-a96e-8b93d0a47a7c" Oct 30 23:40:53.423085 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:53.423057 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kgdzs" Oct 30 23:40:53.423137 ip-10-0-141-180 kubenswrapper[2550]: E1030 23:40:53.423115 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kgdzs" podUID="7728abfe-1747-422e-b8e2-c91c9de83d09" Oct 30 23:40:53.423181 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:53.423165 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7wrh7" Oct 30 23:40:53.423227 ip-10-0-141-180 kubenswrapper[2550]: E1030 23:40:53.423212 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7wrh7" podUID="97da2086-b00d-4f03-b502-56de5ee3d90d" Oct 30 23:40:53.439198 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:53.439166 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqmnp\" (UniqueName: \"kubernetes.io/projected/7728abfe-1747-422e-b8e2-c91c9de83d09-kube-api-access-jqmnp\") pod \"network-check-target-kgdzs\" (UID: \"7728abfe-1747-422e-b8e2-c91c9de83d09\") " pod="openshift-network-diagnostics/network-check-target-kgdzs" Oct 30 23:40:53.439348 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:53.439224 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/97da2086-b00d-4f03-b502-56de5ee3d90d-original-pull-secret\") pod \"global-pull-secret-syncer-7wrh7\" (UID: \"97da2086-b00d-4f03-b502-56de5ee3d90d\") " pod="kube-system/global-pull-secret-syncer-7wrh7" Oct 30 23:40:53.439434 ip-10-0-141-180 kubenswrapper[2550]: E1030 23:40:53.439410 2550 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 23:40:53.439497 ip-10-0-141-180 kubenswrapper[2550]: E1030 23:40:53.439469 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/97da2086-b00d-4f03-b502-56de5ee3d90d-original-pull-secret podName:97da2086-b00d-4f03-b502-56de5ee3d90d nodeName:}" failed. No retries permitted until 2026-10-30 23:41:01.439451856 +0000 UTC m=+17.497655767 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/97da2086-b00d-4f03-b502-56de5ee3d90d-original-pull-secret") pod "global-pull-secret-syncer-7wrh7" (UID: "97da2086-b00d-4f03-b502-56de5ee3d90d") : object "kube-system"/"original-pull-secret" not registered Oct 30 23:40:53.439646 ip-10-0-141-180 kubenswrapper[2550]: E1030 23:40:53.439576 2550 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 23:40:53.439646 ip-10-0-141-180 kubenswrapper[2550]: E1030 23:40:53.439594 2550 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 23:40:53.439646 ip-10-0-141-180 kubenswrapper[2550]: E1030 23:40:53.439607 2550 projected.go:196] Error preparing data for projected volume kube-api-access-jqmnp for pod openshift-network-diagnostics/network-check-target-kgdzs: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 23:40:53.439646 ip-10-0-141-180 kubenswrapper[2550]: E1030 23:40:53.439644 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7728abfe-1747-422e-b8e2-c91c9de83d09-kube-api-access-jqmnp podName:7728abfe-1747-422e-b8e2-c91c9de83d09 nodeName:}" failed. No retries permitted until 2026-10-30 23:41:01.439631901 +0000 UTC m=+17.497835816 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-jqmnp" (UniqueName: "kubernetes.io/projected/7728abfe-1747-422e-b8e2-c91c9de83d09-kube-api-access-jqmnp") pod "network-check-target-kgdzs" (UID: "7728abfe-1747-422e-b8e2-c91c9de83d09") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 23:40:53.527145 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:53.526752 2550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-7sbzf" event={"ID":"99f800b6-a38c-46ca-8234-0dbd3610c8fd","Type":"ContainerStarted","Data":"bdaaaad57aee155da719d2ac3522ddef57c7ecf2d6818f831498b2cfacb1a89c"} Oct 30 23:40:53.654661 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:53.654554 2550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-9s2fx" podStartSLOduration=3.468673393 podStartE2EDuration="8.654530825s" totalImagesPullingTime="5.185857432s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:40:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:40:52.897719939 +0000 UTC m=+8.955923860" watchObservedRunningTime="2026-10-30 23:40:53.654530825 +0000 UTC m=+9.712734745" Oct 30 23:40:53.654840 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:53.654678 2550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-7sbzf" podStartSLOduration=1.329827077 podStartE2EDuration="7.654667329s" totalImagesPullingTime="6.324840252s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 23:40:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:40:53.65408228 +0000 UTC m=+9.712286200" watchObservedRunningTime="2026-10-30 23:40:53.654667329 +0000 UTC m=+9.712871248" Oct 30 23:40:54.531870 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:54.531403 2550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j7n2l" event={"ID":"0796261f-298e-4cbe-9b3e-b3dcf4528e07","Type":"ContainerStarted","Data":"05ebc0a7fe6821a5d43ee9d2bceba9bb9faadb2f3f582bae0c8a86a97fd3934f"} Oct 30 23:40:55.422251 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:55.422219 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pt58s" Oct 30 23:40:55.422433 ip-10-0-141-180 kubenswrapper[2550]: E1030 23:40:55.422372 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pt58s" podUID="145dd323-9370-4b9d-a96e-8b93d0a47a7c" Oct 30 23:40:55.422872 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:55.422721 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kgdzs" Oct 30 23:40:55.422872 ip-10-0-141-180 kubenswrapper[2550]: E1030 23:40:55.422842 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kgdzs" podUID="7728abfe-1747-422e-b8e2-c91c9de83d09" Oct 30 23:40:55.423028 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:55.422895 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7wrh7" Oct 30 23:40:55.423028 ip-10-0-141-180 kubenswrapper[2550]: E1030 23:40:55.422968 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7wrh7" podUID="97da2086-b00d-4f03-b502-56de5ee3d90d" Oct 30 23:40:55.523515 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:55.522859 2550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-9s2fx" Oct 30 23:40:55.523940 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:55.523913 2550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-9s2fx" Oct 30 23:40:56.267813 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:56.267678 2550 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 23:40:56.422675 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:56.422392 2550 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T23:40:56.267697414Z","UUID":"4c7adcf4-1747-450e-80d6-72a53eac62f2","Handler":null,"Name":"","Endpoint":""} Oct 30 23:40:56.424439 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:56.424422 2550 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 23:40:56.424525 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:56.424445 2550 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 23:40:56.539021 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:56.538983 2550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-tfnr7" event={"ID":"f8b33ecc-59a4-4680-9521-299648ea5055","Type":"ContainerStarted","Data":"d72d1630f805555ceed311c56e4a03a9a53bc4bf1ccc731f94f5bd2de577d430"} Oct 30 23:40:56.543191 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:56.543166 2550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7mp5r" event={"ID":"b22fe41c-cf52-4e97-81f1-2f64bcc43640","Type":"ContainerStarted","Data":"b5ae94dd9747f2578c6cfa7d12200dece2c3865629b1717ad2c2e93e7a9595c7"} Oct 30 23:40:56.545951 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:56.545931 2550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j7n2l" event={"ID":"0796261f-298e-4cbe-9b3e-b3dcf4528e07","Type":"ContainerStarted","Data":"346e629e951958e902fd609ff05d92a00878fcb1d3b8bf34abc50895ce7c6996"} Oct 30 23:40:56.548010 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:56.547983 2550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-cb9vj" event={"ID":"2fa7d24b-67b0-400f-94d4-903feb268782","Type":"ContainerStarted","Data":"ef2c3f5124f3df8f561227fce96f304d87e575e770bc51491a9d3b0654e3960e"} Oct 30 23:40:56.655994 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:56.655910 2550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-tfnr7" podStartSLOduration=2.167365505 podStartE2EDuration="11.655892888s" totalImagesPullingTime="9.488527383s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 23:40:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:40:56.655739367 +0000 UTC m=+12.713943283" watchObservedRunningTime="2026-10-30 23:40:56.655892888 +0000 UTC m=+12.714096804" Oct 30 23:40:56.899594 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:56.899512 2550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-cb9vj" podStartSLOduration=3.040324924 podStartE2EDuration="11.899495657s" totalImagesPullingTime="8.859170733s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 23:40:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:40:56.77587383 +0000 UTC m=+12.834077745" watchObservedRunningTime="2026-10-30 23:40:56.899495657 +0000 UTC m=+12.957699572" Oct 30 23:40:57.423239 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:57.423205 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kgdzs" Oct 30 23:40:57.423726 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:57.423244 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7wrh7" Oct 30 23:40:57.423726 ip-10-0-141-180 kubenswrapper[2550]: E1030 23:40:57.423369 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kgdzs" podUID="7728abfe-1747-422e-b8e2-c91c9de83d09" Oct 30 23:40:57.423726 ip-10-0-141-180 kubenswrapper[2550]: E1030 23:40:57.423435 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7wrh7" podUID="97da2086-b00d-4f03-b502-56de5ee3d90d" Oct 30 23:40:57.423726 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:57.423538 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pt58s" Oct 30 23:40:57.423726 ip-10-0-141-180 kubenswrapper[2550]: E1030 23:40:57.423625 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pt58s" podUID="145dd323-9370-4b9d-a96e-8b93d0a47a7c" Oct 30 23:40:57.552781 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:57.552744 2550 generic.go:356] "Generic (PLEG): container finished" podID="b22fe41c-cf52-4e97-81f1-2f64bcc43640" containerID="b5ae94dd9747f2578c6cfa7d12200dece2c3865629b1717ad2c2e93e7a9595c7" exitCode=0 Oct 30 23:40:57.552982 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:57.552845 2550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7mp5r" event={"ID":"b22fe41c-cf52-4e97-81f1-2f64bcc43640","Type":"ContainerDied","Data":"b5ae94dd9747f2578c6cfa7d12200dece2c3865629b1717ad2c2e93e7a9595c7"} Oct 30 23:40:58.556991 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:58.556938 2550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-rsv7z" event={"ID":"953b9446-c289-48ca-b854-a14c70e5971b","Type":"ContainerStarted","Data":"ebda0e1155206a15df1a37d36d6971d9a9d3c634475773838b437804431bd9bb"} Oct 30 23:40:58.559496 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:58.559471 2550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j7n2l" event={"ID":"0796261f-298e-4cbe-9b3e-b3dcf4528e07","Type":"ContainerStarted","Data":"af209089ce103ba0ec13d50f8afde5ef0aa6430a9f226e0e41a07815abb92ec9"} Oct 30 23:40:58.561126 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:58.561099 2550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-57xrk" event={"ID":"ece07686-4d8e-4068-9fb2-5889046cb73b","Type":"ContainerStarted","Data":"09dbb610ea6bed5261a12735a421eaac8c853bfa0c44df30b988efcf92770756"} Oct 30 23:40:58.677123 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:58.677063 2550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-rsv7z" podStartSLOduration=1.674723382 podStartE2EDuration="13.677049414s" totalImagesPullingTime="12.002326032s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 23:40:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:40:58.676720727 +0000 UTC m=+14.734924621" watchObservedRunningTime="2026-10-30 23:40:58.677049414 +0000 UTC m=+14.735253328" Oct 30 23:40:58.798477 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:58.798425 2550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j7n2l" podStartSLOduration=2.846064316 podStartE2EDuration="13.798410976s" totalImagesPullingTime="10.95234666s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 23:40:45 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:40:58.797965983 +0000 UTC m=+14.856169897" watchObservedRunningTime="2026-10-30 23:40:58.798410976 +0000 UTC m=+14.856614917" Oct 30 23:40:58.914466 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:58.914405 2550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-57xrk" podStartSLOduration=4.691389398 podStartE2EDuration="13.914384513s" totalImagesPullingTime="9.222995115s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 23:40:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:40:58.914371281 +0000 UTC m=+14.972575200" watchObservedRunningTime="2026-10-30 23:40:58.914384513 +0000 UTC m=+14.972588429" Oct 30 23:40:59.422876 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:59.422849 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pt58s" Oct 30 23:40:59.423008 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:59.422885 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7wrh7" Oct 30 23:40:59.423008 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:59.422884 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kgdzs" Oct 30 23:40:59.423008 ip-10-0-141-180 kubenswrapper[2550]: E1030 23:40:59.422957 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pt58s" podUID="145dd323-9370-4b9d-a96e-8b93d0a47a7c" Oct 30 23:40:59.423145 ip-10-0-141-180 kubenswrapper[2550]: E1030 23:40:59.423047 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7wrh7" podUID="97da2086-b00d-4f03-b502-56de5ee3d90d" Oct 30 23:40:59.423145 ip-10-0-141-180 kubenswrapper[2550]: E1030 23:40:59.423090 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kgdzs" podUID="7728abfe-1747-422e-b8e2-c91c9de83d09" Oct 30 23:40:59.565719 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:59.565677 2550 generic.go:356] "Generic (PLEG): container finished" podID="b22fe41c-cf52-4e97-81f1-2f64bcc43640" containerID="bfdcd4ef9e891b5bd8f68623867a8a250524ed0d5cb742b332ed7f5916830147" exitCode=0 Oct 30 23:40:59.566307 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:40:59.565745 2550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7mp5r" event={"ID":"b22fe41c-cf52-4e97-81f1-2f64bcc43640","Type":"ContainerDied","Data":"bfdcd4ef9e891b5bd8f68623867a8a250524ed0d5cb742b332ed7f5916830147"} Oct 30 23:41:01.407113 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:01.406812 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/145dd323-9370-4b9d-a96e-8b93d0a47a7c-metrics-certs\") pod \"network-metrics-daemon-pt58s\" (UID: \"145dd323-9370-4b9d-a96e-8b93d0a47a7c\") " pod="openshift-multus/network-metrics-daemon-pt58s" Oct 30 23:41:01.407113 ip-10-0-141-180 kubenswrapper[2550]: E1030 23:41:01.406962 2550 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 23:41:01.407755 ip-10-0-141-180 kubenswrapper[2550]: E1030 23:41:01.407122 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/145dd323-9370-4b9d-a96e-8b93d0a47a7c-metrics-certs podName:145dd323-9370-4b9d-a96e-8b93d0a47a7c nodeName:}" failed. No retries permitted until 2026-10-30 23:41:17.407106933 +0000 UTC m=+33.465310841 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/145dd323-9370-4b9d-a96e-8b93d0a47a7c-metrics-certs") pod "network-metrics-daemon-pt58s" (UID: "145dd323-9370-4b9d-a96e-8b93d0a47a7c") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 23:41:01.423050 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:01.423014 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kgdzs" Oct 30 23:41:01.423202 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:01.423134 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7wrh7" Oct 30 23:41:01.423202 ip-10-0-141-180 kubenswrapper[2550]: E1030 23:41:01.423149 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kgdzs" podUID="7728abfe-1747-422e-b8e2-c91c9de83d09" Oct 30 23:41:01.423202 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:01.423164 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pt58s" Oct 30 23:41:01.423341 ip-10-0-141-180 kubenswrapper[2550]: E1030 23:41:01.423252 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7wrh7" podUID="97da2086-b00d-4f03-b502-56de5ee3d90d" Oct 30 23:41:01.423378 ip-10-0-141-180 kubenswrapper[2550]: E1030 23:41:01.423325 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pt58s" podUID="145dd323-9370-4b9d-a96e-8b93d0a47a7c" Oct 30 23:41:01.507977 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:01.507940 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/97da2086-b00d-4f03-b502-56de5ee3d90d-original-pull-secret\") pod \"global-pull-secret-syncer-7wrh7\" (UID: \"97da2086-b00d-4f03-b502-56de5ee3d90d\") " pod="kube-system/global-pull-secret-syncer-7wrh7" Oct 30 23:41:01.508165 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:01.508042 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqmnp\" (UniqueName: \"kubernetes.io/projected/7728abfe-1747-422e-b8e2-c91c9de83d09-kube-api-access-jqmnp\") pod \"network-check-target-kgdzs\" (UID: \"7728abfe-1747-422e-b8e2-c91c9de83d09\") " pod="openshift-network-diagnostics/network-check-target-kgdzs" Oct 30 23:41:01.508229 ip-10-0-141-180 kubenswrapper[2550]: E1030 23:41:01.508172 2550 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 23:41:01.508269 ip-10-0-141-180 kubenswrapper[2550]: E1030 23:41:01.508253 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/97da2086-b00d-4f03-b502-56de5ee3d90d-original-pull-secret podName:97da2086-b00d-4f03-b502-56de5ee3d90d nodeName:}" failed. No retries permitted until 2026-10-30 23:41:17.508232026 +0000 UTC m=+33.566435962 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/97da2086-b00d-4f03-b502-56de5ee3d90d-original-pull-secret") pod "global-pull-secret-syncer-7wrh7" (UID: "97da2086-b00d-4f03-b502-56de5ee3d90d") : object "kube-system"/"original-pull-secret" not registered Oct 30 23:41:01.508269 ip-10-0-141-180 kubenswrapper[2550]: E1030 23:41:01.508253 2550 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 23:41:01.508756 ip-10-0-141-180 kubenswrapper[2550]: E1030 23:41:01.508283 2550 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 23:41:01.508756 ip-10-0-141-180 kubenswrapper[2550]: E1030 23:41:01.508296 2550 projected.go:196] Error preparing data for projected volume kube-api-access-jqmnp for pod openshift-network-diagnostics/network-check-target-kgdzs: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 23:41:01.508756 ip-10-0-141-180 kubenswrapper[2550]: E1030 23:41:01.508350 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7728abfe-1747-422e-b8e2-c91c9de83d09-kube-api-access-jqmnp podName:7728abfe-1747-422e-b8e2-c91c9de83d09 nodeName:}" failed. No retries permitted until 2026-10-30 23:41:17.508326002 +0000 UTC m=+33.566529927 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-jqmnp" (UniqueName: "kubernetes.io/projected/7728abfe-1747-422e-b8e2-c91c9de83d09-kube-api-access-jqmnp") pod "network-check-target-kgdzs" (UID: "7728abfe-1747-422e-b8e2-c91c9de83d09") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 23:41:01.573806 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:01.573769 2550 generic.go:356] "Generic (PLEG): container finished" podID="b22fe41c-cf52-4e97-81f1-2f64bcc43640" containerID="e158c325258c29c4ef0643a4493be3843d84484b3dc8daf0da769ecc29722de9" exitCode=0 Oct 30 23:41:01.573968 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:01.573829 2550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7mp5r" event={"ID":"b22fe41c-cf52-4e97-81f1-2f64bcc43640","Type":"ContainerDied","Data":"e158c325258c29c4ef0643a4493be3843d84484b3dc8daf0da769ecc29722de9"} Oct 30 23:41:02.425063 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:02.425035 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7wrh7" Oct 30 23:41:02.425450 ip-10-0-141-180 kubenswrapper[2550]: E1030 23:41:02.425139 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7wrh7" podUID="97da2086-b00d-4f03-b502-56de5ee3d90d" Oct 30 23:41:03.423276 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:03.423087 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kgdzs" Oct 30 23:41:03.423460 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:03.423118 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pt58s" Oct 30 23:41:03.423460 ip-10-0-141-180 kubenswrapper[2550]: E1030 23:41:03.423393 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kgdzs" podUID="7728abfe-1747-422e-b8e2-c91c9de83d09" Oct 30 23:41:03.423460 ip-10-0-141-180 kubenswrapper[2550]: E1030 23:41:03.423450 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pt58s" podUID="145dd323-9370-4b9d-a96e-8b93d0a47a7c" Oct 30 23:41:03.582044 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:03.582008 2550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-26v6x" event={"ID":"0d4810de-8bd2-4270-b907-341d9f840bb8","Type":"ContainerStarted","Data":"0069f540d93757ddcd2cd62ac2dc3d4d0aa48c125911d9adf364d26853164030"} Oct 30 23:41:03.582449 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:03.582099 2550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-26v6x" event={"ID":"0d4810de-8bd2-4270-b907-341d9f840bb8","Type":"ContainerStarted","Data":"d35b02338232ad1da1eb3b9893db686943060ec8060820c589953be2aecdc15f"} Oct 30 23:41:03.582449 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:03.582148 2550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-26v6x" event={"ID":"0d4810de-8bd2-4270-b907-341d9f840bb8","Type":"ContainerStarted","Data":"b6dcdcea4569e1b56e9a11be889c15b0be58524567ce238e1abb96e054061ed4"} Oct 30 23:41:03.582449 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:03.582220 2550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-26v6x" event={"ID":"0d4810de-8bd2-4270-b907-341d9f840bb8","Type":"ContainerStarted","Data":"d7ab90662b43a73bc7c2d7138c62fd089dd9999380f9a84bc69eb3752245fb47"} Oct 30 23:41:03.582449 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:03.582268 2550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-26v6x" event={"ID":"0d4810de-8bd2-4270-b907-341d9f840bb8","Type":"ContainerStarted","Data":"775d1d0b195d060359d2b6e42b76d9369781808658ab3f1e0b9a7fec36cbcc3c"} Oct 30 23:41:03.582449 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:03.582312 2550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-26v6x" event={"ID":"0d4810de-8bd2-4270-b907-341d9f840bb8","Type":"ContainerStarted","Data":"3320455dd63857a2f467588e8b95bb8b5137259b4c25e762f4c1e1cab56df887"} Oct 30 23:41:04.425756 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:04.425722 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7wrh7" Oct 30 23:41:04.425890 ip-10-0-141-180 kubenswrapper[2550]: E1030 23:41:04.425829 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7wrh7" podUID="97da2086-b00d-4f03-b502-56de5ee3d90d" Oct 30 23:41:04.586130 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:04.586096 2550 generic.go:356] "Generic (PLEG): container finished" podID="b22fe41c-cf52-4e97-81f1-2f64bcc43640" containerID="5f51926d68e2a0e635eb974f550ce17ad576034eef7ad09e8250de50c547cf3a" exitCode=0 Oct 30 23:41:04.586655 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:04.586150 2550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7mp5r" event={"ID":"b22fe41c-cf52-4e97-81f1-2f64bcc43640","Type":"ContainerDied","Data":"5f51926d68e2a0e635eb974f550ce17ad576034eef7ad09e8250de50c547cf3a"} Oct 30 23:41:05.422284 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:05.422253 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pt58s" Oct 30 23:41:05.422284 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:05.422280 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kgdzs" Oct 30 23:41:05.422457 ip-10-0-141-180 kubenswrapper[2550]: E1030 23:41:05.422375 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pt58s" podUID="145dd323-9370-4b9d-a96e-8b93d0a47a7c" Oct 30 23:41:05.422549 ip-10-0-141-180 kubenswrapper[2550]: E1030 23:41:05.422489 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kgdzs" podUID="7728abfe-1747-422e-b8e2-c91c9de83d09" Oct 30 23:41:05.590692 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:05.590613 2550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-26v6x" event={"ID":"0d4810de-8bd2-4270-b907-341d9f840bb8","Type":"ContainerStarted","Data":"f6cfd71947983c9a8c990efb9a166392749cf88979cd1a1a63c32b3d855ed83c"} Oct 30 23:41:05.593437 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:05.593413 2550 generic.go:356] "Generic (PLEG): container finished" podID="b22fe41c-cf52-4e97-81f1-2f64bcc43640" containerID="b8c54e9dd60ac3d6b707158cdbb1f65969985423d3907afdd4ac51290047332f" exitCode=0 Oct 30 23:41:05.593550 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:05.593453 2550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7mp5r" event={"ID":"b22fe41c-cf52-4e97-81f1-2f64bcc43640","Type":"ContainerDied","Data":"b8c54e9dd60ac3d6b707158cdbb1f65969985423d3907afdd4ac51290047332f"} Oct 30 23:41:06.422945 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:06.422909 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7wrh7" Oct 30 23:41:06.423116 ip-10-0-141-180 kubenswrapper[2550]: E1030 23:41:06.423009 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7wrh7" podUID="97da2086-b00d-4f03-b502-56de5ee3d90d" Oct 30 23:41:06.598609 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:06.598575 2550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7mp5r" event={"ID":"b22fe41c-cf52-4e97-81f1-2f64bcc43640","Type":"ContainerStarted","Data":"dbdc1926c314df200c6f71880f8174bd01f3f20e3bafb53f5bf00d400a02cd2f"} Oct 30 23:41:06.773751 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:06.773656 2550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-7mp5r" podStartSLOduration=7.258831025 podStartE2EDuration="21.773642404s" totalImagesPullingTime="14.514811379s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 23:40:45 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:41:06.773273082 +0000 UTC m=+22.831476998" watchObservedRunningTime="2026-10-30 23:41:06.773642404 +0000 UTC m=+22.831846319" Oct 30 23:41:07.423297 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:07.423136 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kgdzs" Oct 30 23:41:07.423467 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:07.423207 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pt58s" Oct 30 23:41:07.423467 ip-10-0-141-180 kubenswrapper[2550]: E1030 23:41:07.423417 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kgdzs" podUID="7728abfe-1747-422e-b8e2-c91c9de83d09" Oct 30 23:41:07.423576 ip-10-0-141-180 kubenswrapper[2550]: E1030 23:41:07.423552 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pt58s" podUID="145dd323-9370-4b9d-a96e-8b93d0a47a7c" Oct 30 23:41:07.603929 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:07.603821 2550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-26v6x" event={"ID":"0d4810de-8bd2-4270-b907-341d9f840bb8","Type":"ContainerStarted","Data":"b62b91483f78247e66cac2494f272e32ea73d818563449f7793dee2dd33ea73f"} Oct 30 23:41:07.889147 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:07.889035 2550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-26v6x" podStartSLOduration=6.21384741 podStartE2EDuration="22.88902145s" totalImagesPullingTime="16.67517404s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 23:40:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:41:07.88884662 +0000 UTC m=+23.947050535" watchObservedRunningTime="2026-10-30 23:41:07.88902145 +0000 UTC m=+23.947225364" Oct 30 23:41:08.425840 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:08.425815 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7wrh7" Oct 30 23:41:08.426060 ip-10-0-141-180 kubenswrapper[2550]: E1030 23:41:08.425905 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7wrh7" podUID="97da2086-b00d-4f03-b502-56de5ee3d90d" Oct 30 23:41:08.606699 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:08.606661 2550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-26v6x" Oct 30 23:41:08.607096 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:08.606748 2550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-26v6x" Oct 30 23:41:08.607096 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:08.606803 2550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-26v6x" Oct 30 23:41:08.621550 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:08.621522 2550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-26v6x" Oct 30 23:41:08.621687 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:08.621649 2550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-26v6x" Oct 30 23:41:09.422697 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:09.422666 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pt58s" Oct 30 23:41:09.422697 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:09.422694 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kgdzs" Oct 30 23:41:09.422884 ip-10-0-141-180 kubenswrapper[2550]: E1030 23:41:09.422770 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pt58s" podUID="145dd323-9370-4b9d-a96e-8b93d0a47a7c" Oct 30 23:41:09.423035 ip-10-0-141-180 kubenswrapper[2550]: E1030 23:41:09.423009 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kgdzs" podUID="7728abfe-1747-422e-b8e2-c91c9de83d09" Oct 30 23:41:09.870438 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:09.870374 2550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-7wrh7"] Oct 30 23:41:09.870763 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:09.870481 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7wrh7" Oct 30 23:41:09.870763 ip-10-0-141-180 kubenswrapper[2550]: E1030 23:41:09.870558 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7wrh7" podUID="97da2086-b00d-4f03-b502-56de5ee3d90d" Oct 30 23:41:09.873463 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:09.873442 2550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-pt58s"] Oct 30 23:41:09.873571 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:09.873523 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pt58s" Oct 30 23:41:09.873620 ip-10-0-141-180 kubenswrapper[2550]: E1030 23:41:09.873605 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pt58s" podUID="145dd323-9370-4b9d-a96e-8b93d0a47a7c" Oct 30 23:41:09.876082 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:09.876060 2550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-kgdzs"] Oct 30 23:41:09.876168 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:09.876146 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kgdzs" Oct 30 23:41:09.876254 ip-10-0-141-180 kubenswrapper[2550]: E1030 23:41:09.876236 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kgdzs" podUID="7728abfe-1747-422e-b8e2-c91c9de83d09" Oct 30 23:41:11.422273 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:11.422242 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7wrh7" Oct 30 23:41:11.422676 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:11.422277 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pt58s" Oct 30 23:41:11.422676 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:11.422284 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kgdzs" Oct 30 23:41:11.422676 ip-10-0-141-180 kubenswrapper[2550]: E1030 23:41:11.422387 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7wrh7" podUID="97da2086-b00d-4f03-b502-56de5ee3d90d" Oct 30 23:41:11.422676 ip-10-0-141-180 kubenswrapper[2550]: E1030 23:41:11.422478 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pt58s" podUID="145dd323-9370-4b9d-a96e-8b93d0a47a7c" Oct 30 23:41:11.422676 ip-10-0-141-180 kubenswrapper[2550]: E1030 23:41:11.422567 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kgdzs" podUID="7728abfe-1747-422e-b8e2-c91c9de83d09" Oct 30 23:41:13.423239 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:13.423210 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7wrh7" Oct 30 23:41:13.423679 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:13.423246 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pt58s" Oct 30 23:41:13.423679 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:13.423310 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kgdzs" Oct 30 23:41:13.423679 ip-10-0-141-180 kubenswrapper[2550]: E1030 23:41:13.423325 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7wrh7" podUID="97da2086-b00d-4f03-b502-56de5ee3d90d" Oct 30 23:41:13.423679 ip-10-0-141-180 kubenswrapper[2550]: E1030 23:41:13.423396 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kgdzs" podUID="7728abfe-1747-422e-b8e2-c91c9de83d09" Oct 30 23:41:13.423679 ip-10-0-141-180 kubenswrapper[2550]: E1030 23:41:13.423462 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pt58s" podUID="145dd323-9370-4b9d-a96e-8b93d0a47a7c" Oct 30 23:41:15.423247 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:15.423220 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pt58s" Oct 30 23:41:15.423621 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:15.423255 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7wrh7" Oct 30 23:41:15.423621 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:15.423253 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kgdzs" Oct 30 23:41:15.423621 ip-10-0-141-180 kubenswrapper[2550]: E1030 23:41:15.423358 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pt58s" podUID="145dd323-9370-4b9d-a96e-8b93d0a47a7c" Oct 30 23:41:15.423621 ip-10-0-141-180 kubenswrapper[2550]: E1030 23:41:15.423473 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7wrh7" podUID="97da2086-b00d-4f03-b502-56de5ee3d90d" Oct 30 23:41:15.423621 ip-10-0-141-180 kubenswrapper[2550]: E1030 23:41:15.423532 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kgdzs" podUID="7728abfe-1747-422e-b8e2-c91c9de83d09" Oct 30 23:41:15.504441 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:15.504421 2550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-9s2fx" Oct 30 23:41:15.504551 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:15.504539 2550 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 23:41:15.505093 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:15.505066 2550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-9s2fx" Oct 30 23:41:15.768967 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:15.768942 2550 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-180.us-west-2.compute.internal" event="NodeReady" Oct 30 23:41:15.769128 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:15.769044 2550 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Oct 30 23:41:15.861044 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:15.861012 2550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-f7zjp"] Oct 30 23:41:15.865346 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:15.865313 2550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-lxdsf"] Oct 30 23:41:15.865451 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:15.865430 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-f7zjp" Oct 30 23:41:15.869218 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:15.869196 2550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-l4d2m"] Oct 30 23:41:15.869346 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:15.869310 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-lxdsf" Oct 30 23:41:15.872820 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:15.872804 2550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vl8t5"] Oct 30 23:41:15.872914 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:15.872901 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-l4d2m" Oct 30 23:41:15.876420 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:15.876400 2550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-pqt2c"] Oct 30 23:41:15.876512 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:15.876498 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vl8t5" Oct 30 23:41:15.880037 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:15.880021 2550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-rxpst"] Oct 30 23:41:15.880150 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:15.880134 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-pqt2c" Oct 30 23:41:15.883528 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:15.883514 2550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-544b44f8d-5p6kz"] Oct 30 23:41:15.883626 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:15.883612 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-rxpst" Oct 30 23:41:15.886804 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:15.886787 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-544b44f8d-5p6kz" Oct 30 23:41:15.889573 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:15.889548 2550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-httg9"] Oct 30 23:41:15.892947 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:15.892928 2550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-9l5zq"] Oct 30 23:41:15.893043 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:15.893024 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-httg9" Oct 30 23:41:15.896280 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:15.896265 2550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-9qs84"] Oct 30 23:41:15.896431 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:15.896415 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-9l5zq" Oct 30 23:41:15.899088 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:15.899072 2550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-fbgth"] Oct 30 23:41:15.899211 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:15.899193 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-9qs84" Oct 30 23:41:15.902084 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:15.902069 2550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-tcxmv"] Oct 30 23:41:15.902183 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:15.902163 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-fbgth" Oct 30 23:41:15.904959 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:15.904943 2550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-66d97c9c97-qqxj5"] Oct 30 23:41:15.905097 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:15.905080 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-tcxmv" Oct 30 23:41:15.907836 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:15.907819 2550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-dd954f877-x9h9g"] Oct 30 23:41:15.907928 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:15.907916 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-66d97c9c97-qqxj5" Oct 30 23:41:15.910954 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:15.910938 2550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-46zqw"] Oct 30 23:41:15.911082 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:15.911063 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-dd954f877-x9h9g" Oct 30 23:41:15.913998 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:15.913981 2550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-xrk8k"] Oct 30 23:41:15.914082 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:15.914035 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-46zqw" Oct 30 23:41:15.920005 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:15.919987 2550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-7c4986d449-9mqqt"] Oct 30 23:41:15.920185 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:15.920165 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-xrk8k" Oct 30 23:41:15.923015 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:15.922995 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-9mqqt" Oct 30 23:41:15.926166 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:15.924652 2550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-f7zjp"] Oct 30 23:41:15.926496 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:15.926478 2550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\"" Oct 30 23:41:15.926759 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:15.926738 2550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\"" Oct 30 23:41:15.927668 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:15.927648 2550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-lxdsf"] Oct 30 23:41:15.929248 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:15.929230 2550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-rxpst"] Oct 30 23:41:15.930609 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:15.930188 2550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-pqt2c"] Oct 30 23:41:15.930967 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:15.930950 2550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Oct 30 23:41:15.931317 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:15.931298 2550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vl8t5"] Oct 30 23:41:15.934604 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:15.934579 2550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Oct 30 23:41:15.940657 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:15.940637 2550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-httg9"] Oct 30 23:41:15.949984 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:15.949952 2550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-tcxmv"] Oct 30 23:41:15.953278 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:15.953256 2550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-9l5zq"] Oct 30 23:41:15.954246 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:15.954229 2550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-xrk8k"] Oct 30 23:41:15.955497 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:15.955444 2550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-46zqw"] Oct 30 23:41:15.956635 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:15.956614 2550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-66d97c9c97-qqxj5"] Oct 30 23:41:15.960637 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:15.960616 2550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-9mqqt"] Oct 30 23:41:15.983817 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:15.983799 2550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-l4d2m"] Oct 30 23:41:15.992221 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:15.992203 2550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-fbgth"] Oct 30 23:41:15.999345 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:15.999307 2550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-9qs84"] Oct 30 23:41:16.002998 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:16.002977 2550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-544b44f8d-5p6kz"] Oct 30 23:41:16.012388 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:16.012368 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29b8bd1a-beeb-495c-9f61-e3326866a1d7-tmp-dir\") pod \"dns-default-9l5zq\" (UID: \"29b8bd1a-beeb-495c-9f61-e3326866a1d7\") " pod="openshift-dns/dns-default-9l5zq" Oct 30 23:41:16.012468 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:16.012394 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6382dfb5-a363-47a2-bc79-0b96f7bec0c3-config\") pod \"console-operator-7cd56b8dbc-pqt2c\" (UID: \"6382dfb5-a363-47a2-bc79-0b96f7bec0c3\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-pqt2c" Oct 30 23:41:16.012468 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:16.012424 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/58f7cf25-f5d0-4481-a927-4a745b692f7a-registry-tls\") pod \"image-registry-544b44f8d-5p6kz\" (UID: \"58f7cf25-f5d0-4481-a927-4a745b692f7a\") " pod="openshift-image-registry/image-registry-544b44f8d-5p6kz" Oct 30 23:41:16.012468 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:16.012458 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6382dfb5-a363-47a2-bc79-0b96f7bec0c3-serving-cert\") pod \"console-operator-7cd56b8dbc-pqt2c\" (UID: \"6382dfb5-a363-47a2-bc79-0b96f7bec0c3\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-pqt2c" Oct 30 23:41:16.012598 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:16.012503 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6382dfb5-a363-47a2-bc79-0b96f7bec0c3-trusted-ca\") pod \"console-operator-7cd56b8dbc-pqt2c\" (UID: \"6382dfb5-a363-47a2-bc79-0b96f7bec0c3\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-pqt2c" Oct 30 23:41:16.012598 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:16.012563 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/03211c6a-9761-419c-839b-21ca413d5bb2-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-httg9\" (UID: \"03211c6a-9761-419c-839b-21ca413d5bb2\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-httg9" Oct 30 23:41:16.012656 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:16.012630 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/03211c6a-9761-419c-839b-21ca413d5bb2-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-httg9\" (UID: \"03211c6a-9761-419c-839b-21ca413d5bb2\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-httg9" Oct 30 23:41:16.012727 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:16.012708 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vsfpv\" (UniqueName: \"kubernetes.io/projected/840aaae0-a215-4ccd-b420-d6f5723b65ae-kube-api-access-vsfpv\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-9qs84\" (UID: \"840aaae0-a215-4ccd-b420-d6f5723b65ae\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-9qs84" Oct 30 23:41:16.012772 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:16.012737 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/be5540b5-7b10-45b2-b40c-723efd9f1b01-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-vl8t5\" (UID: \"be5540b5-7b10-45b2-b40c-723efd9f1b01\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vl8t5" Oct 30 23:41:16.012772 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:16.012753 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/840aaae0-a215-4ccd-b420-d6f5723b65ae-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-9qs84\" (UID: \"840aaae0-a215-4ccd-b420-d6f5723b65ae\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-9qs84" Oct 30 23:41:16.012866 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:16.012795 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/29b8bd1a-beeb-495c-9f61-e3326866a1d7-config-volume\") pod \"dns-default-9l5zq\" (UID: \"29b8bd1a-beeb-495c-9f61-e3326866a1d7\") " pod="openshift-dns/dns-default-9l5zq" Oct 30 23:41:16.012866 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:16.012811 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/58f7cf25-f5d0-4481-a927-4a745b692f7a-installation-pull-secrets\") pod \"image-registry-544b44f8d-5p6kz\" (UID: \"58f7cf25-f5d0-4481-a927-4a745b692f7a\") " pod="openshift-image-registry/image-registry-544b44f8d-5p6kz" Oct 30 23:41:16.012866 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:16.012826 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9l2q\" (UniqueName: \"kubernetes.io/projected/f8eb4b4c-8225-436b-b1e8-7775414a1ca3-kube-api-access-k9l2q\") pod \"volume-data-source-validator-58c9f5f46d-tcxmv\" (UID: \"f8eb4b4c-8225-436b-b1e8-7775414a1ca3\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-tcxmv" Oct 30 23:41:16.012866 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:16.012851 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/4cf4abe6-da2b-41b6-9caa-9659d4f750fb-default-certificate\") pod \"router-default-66d97c9c97-qqxj5\" (UID: \"4cf4abe6-da2b-41b6-9caa-9659d4f750fb\") " pod="openshift-ingress/router-default-66d97c9c97-qqxj5" Oct 30 23:41:16.013032 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:16.012908 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vj8vx\" (UniqueName: \"kubernetes.io/projected/29b8bd1a-beeb-495c-9f61-e3326866a1d7-kube-api-access-vj8vx\") pod \"dns-default-9l5zq\" (UID: \"29b8bd1a-beeb-495c-9f61-e3326866a1d7\") " pod="openshift-dns/dns-default-9l5zq" Oct 30 23:41:16.013032 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:16.012961 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9700c98a-b858-45a5-9025-8d75f3c14044-cert\") pod \"ingress-canary-46zqw\" (UID: \"9700c98a-b858-45a5-9025-8d75f3c14044\") " pod="openshift-ingress-canary/ingress-canary-46zqw" Oct 30 23:41:16.013032 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:16.013004 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/b53ad6ed-90fd-4a0f-bf54-88d416ba19e9-nginx-conf\") pod \"networking-console-plugin-86c478bb88-rxpst\" (UID: \"b53ad6ed-90fd-4a0f-bf54-88d416ba19e9\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-rxpst" Oct 30 23:41:16.013177 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:16.013102 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/17461b46-7ebf-49b7-afaf-12e20e3994a3-ca-trust-extracted\") pod \"image-registry-dd954f877-x9h9g\" (UID: \"17461b46-7ebf-49b7-afaf-12e20e3994a3\") " pod="openshift-image-registry/image-registry-dd954f877-x9h9g" Oct 30 23:41:16.013177 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:16.013132 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4cf4abe6-da2b-41b6-9caa-9659d4f750fb-service-ca-bundle\") pod \"router-default-66d97c9c97-qqxj5\" (UID: \"4cf4abe6-da2b-41b6-9caa-9659d4f750fb\") " pod="openshift-ingress/router-default-66d97c9c97-qqxj5" Oct 30 23:41:16.013177 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:16.013172 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sx9pp\" (UniqueName: \"kubernetes.io/projected/58f7cf25-f5d0-4481-a927-4a745b692f7a-kube-api-access-sx9pp\") pod \"image-registry-544b44f8d-5p6kz\" (UID: \"58f7cf25-f5d0-4481-a927-4a745b692f7a\") " pod="openshift-image-registry/image-registry-544b44f8d-5p6kz" Oct 30 23:41:16.013283 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:16.013192 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5x8x6\" (UniqueName: \"kubernetes.io/projected/03211c6a-9761-419c-839b-21ca413d5bb2-kube-api-access-5x8x6\") pod \"cluster-monitoring-operator-884576c57-httg9\" (UID: \"03211c6a-9761-419c-839b-21ca413d5bb2\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-httg9" Oct 30 23:41:16.013283 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:16.013210 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vv9dt\" (UniqueName: \"kubernetes.io/projected/27ccca21-5e99-485f-8b5c-ca8ec5177668-kube-api-access-vv9dt\") pod \"network-check-source-6f78d59f99-xrk8k\" (UID: \"27ccca21-5e99-485f-8b5c-ca8ec5177668\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-xrk8k" Oct 30 23:41:16.013283 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:16.013242 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4cf4abe6-da2b-41b6-9caa-9659d4f750fb-metrics-certs\") pod \"router-default-66d97c9c97-qqxj5\" (UID: \"4cf4abe6-da2b-41b6-9caa-9659d4f750fb\") " pod="openshift-ingress/router-default-66d97c9c97-qqxj5" Oct 30 23:41:16.013283 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:16.013266 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67888\" (UniqueName: \"kubernetes.io/projected/a4f35692-894d-4d41-8b9f-3f5ba3c529b1-kube-api-access-67888\") pod \"kas-connection-checker-685895c7c9-l4d2m\" (UID: \"a4f35692-894d-4d41-8b9f-3f5ba3c529b1\") " pod="kube-system/kas-connection-checker-685895c7c9-l4d2m" Oct 30 23:41:16.013473 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:16.013284 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f464ef41-45c9-4ff4-b410-e2d9d6cdbdf5-serving-cert\") pod \"service-ca-operator-544d749c8c-fbgth\" (UID: \"f464ef41-45c9-4ff4-b410-e2d9d6cdbdf5\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-fbgth" Oct 30 23:41:16.013473 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:16.013300 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/58f7cf25-f5d0-4481-a927-4a745b692f7a-registry-certificates\") pod \"image-registry-544b44f8d-5p6kz\" (UID: \"58f7cf25-f5d0-4481-a927-4a745b692f7a\") " pod="openshift-image-registry/image-registry-544b44f8d-5p6kz" Oct 30 23:41:16.013473 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:16.013427 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/17461b46-7ebf-49b7-afaf-12e20e3994a3-image-registry-private-configuration\") pod \"image-registry-dd954f877-x9h9g\" (UID: \"17461b46-7ebf-49b7-afaf-12e20e3994a3\") " pod="openshift-image-registry/image-registry-dd954f877-x9h9g" Oct 30 23:41:16.013473 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:16.013457 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/17461b46-7ebf-49b7-afaf-12e20e3994a3-registry-certificates\") pod \"image-registry-dd954f877-x9h9g\" (UID: \"17461b46-7ebf-49b7-afaf-12e20e3994a3\") " pod="openshift-image-registry/image-registry-dd954f877-x9h9g" Oct 30 23:41:16.013602 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:16.013473 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qg8hn\" (UniqueName: \"kubernetes.io/projected/6382dfb5-a363-47a2-bc79-0b96f7bec0c3-kube-api-access-qg8hn\") pod \"console-operator-7cd56b8dbc-pqt2c\" (UID: \"6382dfb5-a363-47a2-bc79-0b96f7bec0c3\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-pqt2c" Oct 30 23:41:16.013602 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:16.013498 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtfn9\" (UniqueName: \"kubernetes.io/projected/4cf4abe6-da2b-41b6-9caa-9659d4f750fb-kube-api-access-rtfn9\") pod \"router-default-66d97c9c97-qqxj5\" (UID: \"4cf4abe6-da2b-41b6-9caa-9659d4f750fb\") " pod="openshift-ingress/router-default-66d97c9c97-qqxj5" Oct 30 23:41:16.013602 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:16.013547 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/10774606-8e0c-460b-bd5f-4cb6d121b30b-snapshots\") pod \"insights-operator-7c4986d449-9mqqt\" (UID: \"10774606-8e0c-460b-bd5f-4cb6d121b30b\") " pod="openshift-insights/insights-operator-7c4986d449-9mqqt" Oct 30 23:41:16.013602 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:16.013565 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/10774606-8e0c-460b-bd5f-4cb6d121b30b-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-9mqqt\" (UID: \"10774606-8e0c-460b-bd5f-4cb6d121b30b\") " pod="openshift-insights/insights-operator-7c4986d449-9mqqt" Oct 30 23:41:16.013602 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:16.013595 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/17461b46-7ebf-49b7-afaf-12e20e3994a3-installation-pull-secrets\") pod \"image-registry-dd954f877-x9h9g\" (UID: \"17461b46-7ebf-49b7-afaf-12e20e3994a3\") " pod="openshift-image-registry/image-registry-dd954f877-x9h9g" Oct 30 23:41:16.013806 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:16.013622 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/58f7cf25-f5d0-4481-a927-4a745b692f7a-image-registry-private-configuration\") pod \"image-registry-544b44f8d-5p6kz\" (UID: \"58f7cf25-f5d0-4481-a927-4a745b692f7a\") " pod="openshift-image-registry/image-registry-544b44f8d-5p6kz" Oct 30 23:41:16.013806 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:16.013640 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/17461b46-7ebf-49b7-afaf-12e20e3994a3-trusted-ca\") pod \"image-registry-dd954f877-x9h9g\" (UID: \"17461b46-7ebf-49b7-afaf-12e20e3994a3\") " pod="openshift-image-registry/image-registry-dd954f877-x9h9g" Oct 30 23:41:16.013806 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:16.013655 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xkthh\" (UniqueName: \"kubernetes.io/projected/17461b46-7ebf-49b7-afaf-12e20e3994a3-kube-api-access-xkthh\") pod \"image-registry-dd954f877-x9h9g\" (UID: \"17461b46-7ebf-49b7-afaf-12e20e3994a3\") " pod="openshift-image-registry/image-registry-dd954f877-x9h9g" Oct 30 23:41:16.013806 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:16.013696 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/4cf4abe6-da2b-41b6-9caa-9659d4f750fb-stats-auth\") pod \"router-default-66d97c9c97-qqxj5\" (UID: \"4cf4abe6-da2b-41b6-9caa-9659d4f750fb\") " pod="openshift-ingress/router-default-66d97c9c97-qqxj5" Oct 30 23:41:16.013806 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:16.013732 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/10774606-8e0c-460b-bd5f-4cb6d121b30b-tmp\") pod \"insights-operator-7c4986d449-9mqqt\" (UID: \"10774606-8e0c-460b-bd5f-4cb6d121b30b\") " pod="openshift-insights/insights-operator-7c4986d449-9mqqt" Oct 30 23:41:16.013806 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:16.013793 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xfx9\" (UniqueName: \"kubernetes.io/projected/3af6ccf2-97f1-4598-b8ef-12eac377133b-kube-api-access-9xfx9\") pod \"kas-connection-checker-685895c7c9-f7zjp\" (UID: \"3af6ccf2-97f1-4598-b8ef-12eac377133b\") " pod="kube-system/kas-connection-checker-685895c7c9-f7zjp" Oct 30 23:41:16.013998 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:16.013844 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/58f7cf25-f5d0-4481-a927-4a745b692f7a-ca-trust-extracted\") pod \"image-registry-544b44f8d-5p6kz\" (UID: \"58f7cf25-f5d0-4481-a927-4a745b692f7a\") " pod="openshift-image-registry/image-registry-544b44f8d-5p6kz" Oct 30 23:41:16.013998 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:16.013883 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/58f7cf25-f5d0-4481-a927-4a745b692f7a-bound-sa-token\") pod \"image-registry-544b44f8d-5p6kz\" (UID: \"58f7cf25-f5d0-4481-a927-4a745b692f7a\") " pod="openshift-image-registry/image-registry-544b44f8d-5p6kz" Oct 30 23:41:16.013998 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:16.013902 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/840aaae0-a215-4ccd-b420-d6f5723b65ae-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-9qs84\" (UID: \"840aaae0-a215-4ccd-b420-d6f5723b65ae\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-9qs84" Oct 30 23:41:16.013998 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:16.013920 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/29b8bd1a-beeb-495c-9f61-e3326866a1d7-metrics-tls\") pod \"dns-default-9l5zq\" (UID: \"29b8bd1a-beeb-495c-9f61-e3326866a1d7\") " pod="openshift-dns/dns-default-9l5zq" Oct 30 23:41:16.013998 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:16.013958 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cfwlk\" (UniqueName: \"kubernetes.io/projected/64ae3ff5-44eb-456b-948a-c1a8320e6b07-kube-api-access-cfwlk\") pod \"kas-connection-checker-685895c7c9-lxdsf\" (UID: \"64ae3ff5-44eb-456b-948a-c1a8320e6b07\") " pod="kube-system/kas-connection-checker-685895c7c9-lxdsf" Oct 30 23:41:16.013998 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:16.013985 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b53ad6ed-90fd-4a0f-bf54-88d416ba19e9-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-rxpst\" (UID: \"b53ad6ed-90fd-4a0f-bf54-88d416ba19e9\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-rxpst" Oct 30 23:41:16.014212 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:16.014035 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/10774606-8e0c-460b-bd5f-4cb6d121b30b-serving-cert\") pod \"insights-operator-7c4986d449-9mqqt\" (UID: \"10774606-8e0c-460b-bd5f-4cb6d121b30b\") " pod="openshift-insights/insights-operator-7c4986d449-9mqqt" Oct 30 23:41:16.014212 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:16.014117 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n65gn\" (UniqueName: \"kubernetes.io/projected/10774606-8e0c-460b-bd5f-4cb6d121b30b-kube-api-access-n65gn\") pod \"insights-operator-7c4986d449-9mqqt\" (UID: \"10774606-8e0c-460b-bd5f-4cb6d121b30b\") " pod="openshift-insights/insights-operator-7c4986d449-9mqqt" Oct 30 23:41:16.014212 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:16.014138 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xn7l\" (UniqueName: \"kubernetes.io/projected/9700c98a-b858-45a5-9025-8d75f3c14044-kube-api-access-5xn7l\") pod \"ingress-canary-46zqw\" (UID: \"9700c98a-b858-45a5-9025-8d75f3c14044\") " pod="openshift-ingress-canary/ingress-canary-46zqw" Oct 30 23:41:16.014212 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:16.014176 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/17461b46-7ebf-49b7-afaf-12e20e3994a3-bound-sa-token\") pod \"image-registry-dd954f877-x9h9g\" (UID: \"17461b46-7ebf-49b7-afaf-12e20e3994a3\") " pod="openshift-image-registry/image-registry-dd954f877-x9h9g" Oct 30 23:41:16.014212 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:16.014195 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/be5540b5-7b10-45b2-b40c-723efd9f1b01-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-vl8t5\" (UID: \"be5540b5-7b10-45b2-b40c-723efd9f1b01\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vl8t5" Oct 30 23:41:16.014421 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:16.014211 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45fmr\" (UniqueName: \"kubernetes.io/projected/be5540b5-7b10-45b2-b40c-723efd9f1b01-kube-api-access-45fmr\") pod \"cluster-samples-operator-6ddb6c8b94-vl8t5\" (UID: \"be5540b5-7b10-45b2-b40c-723efd9f1b01\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vl8t5" Oct 30 23:41:16.014421 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:16.014236 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f464ef41-45c9-4ff4-b410-e2d9d6cdbdf5-config\") pod \"service-ca-operator-544d749c8c-fbgth\" (UID: \"f464ef41-45c9-4ff4-b410-e2d9d6cdbdf5\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-fbgth" Oct 30 23:41:16.014421 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:16.014276 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/58f7cf25-f5d0-4481-a927-4a745b692f7a-trusted-ca\") pod \"image-registry-544b44f8d-5p6kz\" (UID: \"58f7cf25-f5d0-4481-a927-4a745b692f7a\") " pod="openshift-image-registry/image-registry-544b44f8d-5p6kz" Oct 30 23:41:16.014421 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:16.014309 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/17461b46-7ebf-49b7-afaf-12e20e3994a3-registry-tls\") pod \"image-registry-dd954f877-x9h9g\" (UID: \"17461b46-7ebf-49b7-afaf-12e20e3994a3\") " pod="openshift-image-registry/image-registry-dd954f877-x9h9g" Oct 30 23:41:16.014421 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:16.014353 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ztdxk\" (UniqueName: \"kubernetes.io/projected/f464ef41-45c9-4ff4-b410-e2d9d6cdbdf5-kube-api-access-ztdxk\") pod \"service-ca-operator-544d749c8c-fbgth\" (UID: \"f464ef41-45c9-4ff4-b410-e2d9d6cdbdf5\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-fbgth" Oct 30 23:41:16.014421 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:16.014385 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/10774606-8e0c-460b-bd5f-4cb6d121b30b-service-ca-bundle\") pod \"insights-operator-7c4986d449-9mqqt\" (UID: \"10774606-8e0c-460b-bd5f-4cb6d121b30b\") " pod="openshift-insights/insights-operator-7c4986d449-9mqqt" Oct 30 23:41:16.023829 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:16.023786 2550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-dd954f877-x9h9g"] Oct 30 23:41:16.026630 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:16.026612 2550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Oct 30 23:41:16.026630 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:16.026612 2550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Oct 30 23:41:16.026762 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:16.026700 2550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 23:41:16.026829 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:16.026806 2550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Oct 30 23:41:16.026917 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:16.026850 2550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Oct 30 23:41:16.026917 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:16.026850 2550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Oct 30 23:41:16.026917 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:16.026869 2550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Oct 30 23:41:16.027098 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:16.027078 2550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Oct 30 23:41:16.027098 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:16.027096 2550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 23:41:16.027347 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:16.027135 2550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 23:41:16.027466 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:16.027419 2550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Oct 30 23:41:16.027466 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:16.027431 2550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 23:41:16.027466 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:16.027437 2550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 23:41:16.027466 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:16.027431 2550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Oct 30 23:41:16.027466 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:16.027453 2550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\"" Oct 30 23:41:16.027466 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:16.027460 2550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Oct 30 23:41:16.027466 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:16.027453 2550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Oct 30 23:41:16.027466 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:16.027430 2550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 23:41:16.027466 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:16.027476 2550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Oct 30 23:41:16.027466 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:16.027506 2550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Oct 30 23:41:16.027466 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:16.027513 2550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Oct 30 23:41:16.028144 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:16.027517 2550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 23:41:16.028144 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:16.027630 2550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Oct 30 23:41:16.028144 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:16.027630 2550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 30 23:41:16.028144 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:16.027643 2550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Oct 30 23:41:16.028144 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:16.027664 2550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Oct 30 23:41:16.028144 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:16.027671 2550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 23:41:16.028144 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:16.027882 2550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Oct 30 23:41:16.079043 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:16.078909 2550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Oct 30 23:41:16.092287 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:16.092263 2550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Oct 30 23:41:16.093341 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:16.093306 2550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Oct 30 23:41:16.093531 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:16.093516 2550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Oct 30 23:41:16.094970 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:16.094942 2550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-6s2q2\"" Oct 30 23:41:16.095097 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:16.095077 2550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-5ndcx\"" Oct 30 23:41:16.095240 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:16.095226 2550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 23:41:16.095566 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:16.095545 2550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Oct 30 23:41:16.095647 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:16.095556 2550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Oct 30 23:41:16.095721 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:16.095706 2550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 23:41:16.095808 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:16.095711 2550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Oct 30 23:41:16.095885 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:16.095867 2550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Oct 30 23:41:16.095934 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:16.095911 2550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-jmjnd\"" Oct 30 23:41:16.096253 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:16.096240 2550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Oct 30 23:41:16.096294 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:16.096266 2550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-jh2kx\"" Oct 30 23:41:16.096294 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:16.096278 2550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Oct 30 23:41:16.100932 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:16.100916 2550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Oct 30 23:41:16.100992 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:16.100919 2550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Oct 30 23:41:16.100992 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:16.100925 2550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Oct 30 23:41:16.104362 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:16.104350 2550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Oct 30 23:41:16.104646 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:16.104623 2550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-9xzgs\"" Oct 30 23:41:16.104726 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:16.104669 2550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-pkxn2\"" Oct 30 23:41:16.104726 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:16.104674 2550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Oct 30 23:41:16.104726 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:16.104677 2550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-lhhdb\"" Oct 30 23:41:16.104726 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:16.104671 2550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-vxpd6\"" Oct 30 23:41:16.104726 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:16.104726 2550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-nl2hc\"" Oct 30 23:41:16.104950 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:16.104744 2550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-w5vpr\"" Oct 30 23:41:16.105801 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:16.105777 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:41:16.114797 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:16.114773 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/10774606-8e0c-460b-bd5f-4cb6d121b30b-serving-cert\") pod \"insights-operator-7c4986d449-9mqqt\" (UID: \"10774606-8e0c-460b-bd5f-4cb6d121b30b\") " pod="openshift-insights/insights-operator-7c4986d449-9mqqt" Oct 30 23:41:16.114873 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:16.114803 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n65gn\" (UniqueName: \"kubernetes.io/projected/10774606-8e0c-460b-bd5f-4cb6d121b30b-kube-api-access-n65gn\") pod \"insights-operator-7c4986d449-9mqqt\" (UID: \"10774606-8e0c-460b-bd5f-4cb6d121b30b\") " pod="openshift-insights/insights-operator-7c4986d449-9mqqt" Oct 30 23:41:16.114873 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:16.114818 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xn7l\" (UniqueName: \"kubernetes.io/projected/9700c98a-b858-45a5-9025-8d75f3c14044-kube-api-access-5xn7l\") pod \"ingress-canary-46zqw\" (UID: \"9700c98a-b858-45a5-9025-8d75f3c14044\") " pod="openshift-ingress-canary/ingress-canary-46zqw" Oct 30 23:41:16.114873 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:16.114833 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/17461b46-7ebf-49b7-afaf-12e20e3994a3-bound-sa-token\") pod \"image-registry-dd954f877-x9h9g\" (UID: \"17461b46-7ebf-49b7-afaf-12e20e3994a3\") " pod="openshift-image-registry/image-registry-dd954f877-x9h9g" Oct 30 23:41:16.114980 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:16.114964 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/be5540b5-7b10-45b2-b40c-723efd9f1b01-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-vl8t5\" (UID: \"be5540b5-7b10-45b2-b40c-723efd9f1b01\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vl8t5" Oct 30 23:41:16.115019 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:16.114995 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45fmr\" (UniqueName: \"kubernetes.io/projected/be5540b5-7b10-45b2-b40c-723efd9f1b01-kube-api-access-45fmr\") pod \"cluster-samples-operator-6ddb6c8b94-vl8t5\" (UID: \"be5540b5-7b10-45b2-b40c-723efd9f1b01\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vl8t5" Oct 30 23:41:16.115019 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:16.115013 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f464ef41-45c9-4ff4-b410-e2d9d6cdbdf5-config\") pod \"service-ca-operator-544d749c8c-fbgth\" (UID: \"f464ef41-45c9-4ff4-b410-e2d9d6cdbdf5\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-fbgth" Oct 30 23:41:16.115102 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:16.115036 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/58f7cf25-f5d0-4481-a927-4a745b692f7a-trusted-ca\") pod \"image-registry-544b44f8d-5p6kz\" (UID: \"58f7cf25-f5d0-4481-a927-4a745b692f7a\") " pod="openshift-image-registry/image-registry-544b44f8d-5p6kz" Oct 30 23:41:16.115102 ip-10-0-141-180 kubenswrapper[2550]: E1030 23:41:16.115046 2550 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 23:41:16.115194 ip-10-0-141-180 kubenswrapper[2550]: E1030 23:41:16.115106 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/be5540b5-7b10-45b2-b40c-723efd9f1b01-samples-operator-tls podName:be5540b5-7b10-45b2-b40c-723efd9f1b01 nodeName:}" failed. No retries permitted until 2026-10-30 23:41:16.615088705 +0000 UTC m=+32.673292614 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/be5540b5-7b10-45b2-b40c-723efd9f1b01-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-vl8t5" (UID: "be5540b5-7b10-45b2-b40c-723efd9f1b01") : secret "samples-operator-tls" not found Oct 30 23:41:16.115194 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:16.115052 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/17461b46-7ebf-49b7-afaf-12e20e3994a3-registry-tls\") pod \"image-registry-dd954f877-x9h9g\" (UID: \"17461b46-7ebf-49b7-afaf-12e20e3994a3\") " pod="openshift-image-registry/image-registry-dd954f877-x9h9g" Oct 30 23:41:16.115194 ip-10-0-141-180 kubenswrapper[2550]: E1030 23:41:16.115113 2550 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 23:41:16.115194 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:16.115144 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztdxk\" (UniqueName: \"kubernetes.io/projected/f464ef41-45c9-4ff4-b410-e2d9d6cdbdf5-kube-api-access-ztdxk\") pod \"service-ca-operator-544d749c8c-fbgth\" (UID: \"f464ef41-45c9-4ff4-b410-e2d9d6cdbdf5\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-fbgth" Oct 30 23:41:16.115194 ip-10-0-141-180 kubenswrapper[2550]: E1030 23:41:16.115153 2550 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-dd954f877-x9h9g: secret "image-registry-tls" not found Oct 30 23:41:16.115194 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:16.115174 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/10774606-8e0c-460b-bd5f-4cb6d121b30b-service-ca-bundle\") pod \"insights-operator-7c4986d449-9mqqt\" (UID: \"10774606-8e0c-460b-bd5f-4cb6d121b30b\") " pod="openshift-insights/insights-operator-7c4986d449-9mqqt" Oct 30 23:41:16.115509 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:16.115202 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29b8bd1a-beeb-495c-9f61-e3326866a1d7-tmp-dir\") pod \"dns-default-9l5zq\" (UID: \"29b8bd1a-beeb-495c-9f61-e3326866a1d7\") " pod="openshift-dns/dns-default-9l5zq" Oct 30 23:41:16.115509 ip-10-0-141-180 kubenswrapper[2550]: E1030 23:41:16.115228 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/17461b46-7ebf-49b7-afaf-12e20e3994a3-registry-tls podName:17461b46-7ebf-49b7-afaf-12e20e3994a3 nodeName:}" failed. No retries permitted until 2026-10-30 23:41:16.615210641 +0000 UTC m=+32.673414546 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/17461b46-7ebf-49b7-afaf-12e20e3994a3-registry-tls") pod "image-registry-dd954f877-x9h9g" (UID: "17461b46-7ebf-49b7-afaf-12e20e3994a3") : secret "image-registry-tls" not found Oct 30 23:41:16.115509 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:16.115260 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6382dfb5-a363-47a2-bc79-0b96f7bec0c3-config\") pod \"console-operator-7cd56b8dbc-pqt2c\" (UID: \"6382dfb5-a363-47a2-bc79-0b96f7bec0c3\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-pqt2c" Oct 30 23:41:16.115509 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:16.115302 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/58f7cf25-f5d0-4481-a927-4a745b692f7a-registry-tls\") pod \"image-registry-544b44f8d-5p6kz\" (UID: \"58f7cf25-f5d0-4481-a927-4a745b692f7a\") " pod="openshift-image-registry/image-registry-544b44f8d-5p6kz" Oct 30 23:41:16.115509 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:16.115349 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6382dfb5-a363-47a2-bc79-0b96f7bec0c3-serving-cert\") pod \"console-operator-7cd56b8dbc-pqt2c\" (UID: \"6382dfb5-a363-47a2-bc79-0b96f7bec0c3\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-pqt2c" Oct 30 23:41:16.115509 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:16.115374 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6382dfb5-a363-47a2-bc79-0b96f7bec0c3-trusted-ca\") pod \"console-operator-7cd56b8dbc-pqt2c\" (UID: \"6382dfb5-a363-47a2-bc79-0b96f7bec0c3\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-pqt2c" Oct 30 23:41:16.115509 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:16.115398 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/03211c6a-9761-419c-839b-21ca413d5bb2-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-httg9\" (UID: \"03211c6a-9761-419c-839b-21ca413d5bb2\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-httg9" Oct 30 23:41:16.115509 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:16.115426 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/03211c6a-9761-419c-839b-21ca413d5bb2-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-httg9\" (UID: \"03211c6a-9761-419c-839b-21ca413d5bb2\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-httg9" Oct 30 23:41:16.115509 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:16.115464 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vsfpv\" (UniqueName: \"kubernetes.io/projected/840aaae0-a215-4ccd-b420-d6f5723b65ae-kube-api-access-vsfpv\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-9qs84\" (UID: \"840aaae0-a215-4ccd-b420-d6f5723b65ae\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-9qs84" Oct 30 23:41:16.115509 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:16.115489 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/be5540b5-7b10-45b2-b40c-723efd9f1b01-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-vl8t5\" (UID: \"be5540b5-7b10-45b2-b40c-723efd9f1b01\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vl8t5" Oct 30 23:41:16.115509 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:16.115513 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/840aaae0-a215-4ccd-b420-d6f5723b65ae-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-9qs84\" (UID: \"840aaae0-a215-4ccd-b420-d6f5723b65ae\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-9qs84" Oct 30 23:41:16.116032 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:16.115544 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/29b8bd1a-beeb-495c-9f61-e3326866a1d7-config-volume\") pod \"dns-default-9l5zq\" (UID: \"29b8bd1a-beeb-495c-9f61-e3326866a1d7\") " pod="openshift-dns/dns-default-9l5zq" Oct 30 23:41:16.116032 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:16.115556 2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29b8bd1a-beeb-495c-9f61-e3326866a1d7-tmp-dir\") pod \"dns-default-9l5zq\" (UID: \"29b8bd1a-beeb-495c-9f61-e3326866a1d7\") " pod="openshift-dns/dns-default-9l5zq" Oct 30 23:41:16.116032 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:16.115568 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/58f7cf25-f5d0-4481-a927-4a745b692f7a-installation-pull-secrets\") pod \"image-registry-544b44f8d-5p6kz\" (UID: \"58f7cf25-f5d0-4481-a927-4a745b692f7a\") " pod="openshift-image-registry/image-registry-544b44f8d-5p6kz" Oct 30 23:41:16.116032 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:16.115595 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9l2q\" (UniqueName: \"kubernetes.io/projected/f8eb4b4c-8225-436b-b1e8-7775414a1ca3-kube-api-access-k9l2q\") pod \"volume-data-source-validator-58c9f5f46d-tcxmv\" (UID: \"f8eb4b4c-8225-436b-b1e8-7775414a1ca3\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-tcxmv" Oct 30 23:41:16.116032 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:16.115741 2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f464ef41-45c9-4ff4-b410-e2d9d6cdbdf5-config\") pod \"service-ca-operator-544d749c8c-fbgth\" (UID: \"f464ef41-45c9-4ff4-b410-e2d9d6cdbdf5\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-fbgth" Oct 30 23:41:16.116032 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:16.115846 2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/10774606-8e0c-460b-bd5f-4cb6d121b30b-service-ca-bundle\") pod \"insights-operator-7c4986d449-9mqqt\" (UID: \"10774606-8e0c-460b-bd5f-4cb6d121b30b\") " pod="openshift-insights/insights-operator-7c4986d449-9mqqt" Oct 30 23:41:16.116032 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:16.115896 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/4cf4abe6-da2b-41b6-9caa-9659d4f750fb-default-certificate\") pod \"router-default-66d97c9c97-qqxj5\" (UID: \"4cf4abe6-da2b-41b6-9caa-9659d4f750fb\") " pod="openshift-ingress/router-default-66d97c9c97-qqxj5" Oct 30 23:41:16.116032 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:16.115922 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vj8vx\" (UniqueName: \"kubernetes.io/projected/29b8bd1a-beeb-495c-9f61-e3326866a1d7-kube-api-access-vj8vx\") pod \"dns-default-9l5zq\" (UID: \"29b8bd1a-beeb-495c-9f61-e3326866a1d7\") " pod="openshift-dns/dns-default-9l5zq" Oct 30 23:41:16.116032 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:16.115945 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9700c98a-b858-45a5-9025-8d75f3c14044-cert\") pod \"ingress-canary-46zqw\" (UID: \"9700c98a-b858-45a5-9025-8d75f3c14044\") " pod="openshift-ingress-canary/ingress-canary-46zqw" Oct 30 23:41:16.116447 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:16.116056 2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6382dfb5-a363-47a2-bc79-0b96f7bec0c3-config\") pod \"console-operator-7cd56b8dbc-pqt2c\" (UID: \"6382dfb5-a363-47a2-bc79-0b96f7bec0c3\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-pqt2c" Oct 30 23:41:16.116447 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:16.116146 2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/29b8bd1a-beeb-495c-9f61-e3326866a1d7-config-volume\") pod \"dns-default-9l5zq\" (UID: \"29b8bd1a-beeb-495c-9f61-e3326866a1d7\") " pod="openshift-dns/dns-default-9l5zq" Oct 30 23:41:16.116447 ip-10-0-141-180 kubenswrapper[2550]: E1030 23:41:16.116192 2550 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 23:41:16.116447 ip-10-0-141-180 kubenswrapper[2550]: E1030 23:41:16.116294 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/03211c6a-9761-419c-839b-21ca413d5bb2-cluster-monitoring-operator-tls podName:03211c6a-9761-419c-839b-21ca413d5bb2 nodeName:}" failed. No retries permitted until 2026-10-30 23:41:16.616277675 +0000 UTC m=+32.674481584 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/03211c6a-9761-419c-839b-21ca413d5bb2-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-httg9" (UID: "03211c6a-9761-419c-839b-21ca413d5bb2") : secret "cluster-monitoring-operator-tls" not found Oct 30 23:41:16.116634 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:16.116579 2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/be5540b5-7b10-45b2-b40c-723efd9f1b01-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-vl8t5\" (UID: \"be5540b5-7b10-45b2-b40c-723efd9f1b01\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vl8t5" Oct 30 23:41:16.116634 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:16.116618 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/b53ad6ed-90fd-4a0f-bf54-88d416ba19e9-nginx-conf\") pod \"networking-console-plugin-86c478bb88-rxpst\" (UID: \"b53ad6ed-90fd-4a0f-bf54-88d416ba19e9\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-rxpst" Oct 30 23:41:16.116960 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:16.116820 2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/840aaae0-a215-4ccd-b420-d6f5723b65ae-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-9qs84\" (UID: \"840aaae0-a215-4ccd-b420-d6f5723b65ae\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-9qs84" Oct 30 23:41:16.116960 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:16.116869 2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6382dfb5-a363-47a2-bc79-0b96f7bec0c3-trusted-ca\") pod \"console-operator-7cd56b8dbc-pqt2c\" (UID: \"6382dfb5-a363-47a2-bc79-0b96f7bec0c3\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-pqt2c" Oct 30 23:41:16.116960 ip-10-0-141-180 kubenswrapper[2550]: E1030 23:41:16.116904 2550 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 23:41:16.116960 ip-10-0-141-180 kubenswrapper[2550]: E1030 23:41:16.116918 2550 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-544b44f8d-5p6kz: secret "image-registry-tls" not found Oct 30 23:41:16.117233 ip-10-0-141-180 kubenswrapper[2550]: E1030 23:41:16.116967 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/58f7cf25-f5d0-4481-a927-4a745b692f7a-registry-tls podName:58f7cf25-f5d0-4481-a927-4a745b692f7a nodeName:}" failed. No retries permitted until 2026-10-30 23:41:16.616953666 +0000 UTC m=+32.675157568 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/58f7cf25-f5d0-4481-a927-4a745b692f7a-registry-tls") pod "image-registry-544b44f8d-5p6kz" (UID: "58f7cf25-f5d0-4481-a927-4a745b692f7a") : secret "image-registry-tls" not found Oct 30 23:41:16.117233 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:16.117137 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/17461b46-7ebf-49b7-afaf-12e20e3994a3-ca-trust-extracted\") pod \"image-registry-dd954f877-x9h9g\" (UID: \"17461b46-7ebf-49b7-afaf-12e20e3994a3\") " pod="openshift-image-registry/image-registry-dd954f877-x9h9g" Oct 30 23:41:16.117233 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:16.117170 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4cf4abe6-da2b-41b6-9caa-9659d4f750fb-service-ca-bundle\") pod \"router-default-66d97c9c97-qqxj5\" (UID: \"4cf4abe6-da2b-41b6-9caa-9659d4f750fb\") " pod="openshift-ingress/router-default-66d97c9c97-qqxj5" Oct 30 23:41:16.117233 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:16.117196 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sx9pp\" (UniqueName: \"kubernetes.io/projected/58f7cf25-f5d0-4481-a927-4a745b692f7a-kube-api-access-sx9pp\") pod \"image-registry-544b44f8d-5p6kz\" (UID: \"58f7cf25-f5d0-4481-a927-4a745b692f7a\") " pod="openshift-image-registry/image-registry-544b44f8d-5p6kz" Oct 30 23:41:16.117233 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:16.117221 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5x8x6\" (UniqueName: \"kubernetes.io/projected/03211c6a-9761-419c-839b-21ca413d5bb2-kube-api-access-5x8x6\") pod \"cluster-monitoring-operator-884576c57-httg9\" (UID: \"03211c6a-9761-419c-839b-21ca413d5bb2\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-httg9" Oct 30 23:41:16.117484 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:16.117248 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vv9dt\" (UniqueName: \"kubernetes.io/projected/27ccca21-5e99-485f-8b5c-ca8ec5177668-kube-api-access-vv9dt\") pod \"network-check-source-6f78d59f99-xrk8k\" (UID: \"27ccca21-5e99-485f-8b5c-ca8ec5177668\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-xrk8k" Oct 30 23:41:16.117484 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:16.117252 2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/58f7cf25-f5d0-4481-a927-4a745b692f7a-trusted-ca\") pod \"image-registry-544b44f8d-5p6kz\" (UID: \"58f7cf25-f5d0-4481-a927-4a745b692f7a\") " pod="openshift-image-registry/image-registry-544b44f8d-5p6kz" Oct 30 23:41:16.117484 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:16.117257 2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/b53ad6ed-90fd-4a0f-bf54-88d416ba19e9-nginx-conf\") pod \"networking-console-plugin-86c478bb88-rxpst\" (UID: \"b53ad6ed-90fd-4a0f-bf54-88d416ba19e9\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-rxpst" Oct 30 23:41:16.117484 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:16.117276 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4cf4abe6-da2b-41b6-9caa-9659d4f750fb-metrics-certs\") pod \"router-default-66d97c9c97-qqxj5\" (UID: \"4cf4abe6-da2b-41b6-9caa-9659d4f750fb\") " pod="openshift-ingress/router-default-66d97c9c97-qqxj5" Oct 30 23:41:16.117484 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:16.117303 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67888\" (UniqueName: \"kubernetes.io/projected/a4f35692-894d-4d41-8b9f-3f5ba3c529b1-kube-api-access-67888\") pod \"kas-connection-checker-685895c7c9-l4d2m\" (UID: \"a4f35692-894d-4d41-8b9f-3f5ba3c529b1\") " pod="kube-system/kas-connection-checker-685895c7c9-l4d2m" Oct 30 23:41:16.117484 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:16.117346 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f464ef41-45c9-4ff4-b410-e2d9d6cdbdf5-serving-cert\") pod \"service-ca-operator-544d749c8c-fbgth\" (UID: \"f464ef41-45c9-4ff4-b410-e2d9d6cdbdf5\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-fbgth" Oct 30 23:41:16.117484 ip-10-0-141-180 kubenswrapper[2550]: E1030 23:41:16.117356 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4cf4abe6-da2b-41b6-9caa-9659d4f750fb-service-ca-bundle podName:4cf4abe6-da2b-41b6-9caa-9659d4f750fb nodeName:}" failed. No retries permitted until 2026-10-30 23:41:16.617324108 +0000 UTC m=+32.675528034 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/4cf4abe6-da2b-41b6-9caa-9659d4f750fb-service-ca-bundle") pod "router-default-66d97c9c97-qqxj5" (UID: "4cf4abe6-da2b-41b6-9caa-9659d4f750fb") : configmap references non-existent config key: service-ca.crt Oct 30 23:41:16.117484 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:16.117385 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/58f7cf25-f5d0-4481-a927-4a745b692f7a-registry-certificates\") pod \"image-registry-544b44f8d-5p6kz\" (UID: \"58f7cf25-f5d0-4481-a927-4a745b692f7a\") " pod="openshift-image-registry/image-registry-544b44f8d-5p6kz" Oct 30 23:41:16.117484 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:16.117420 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/17461b46-7ebf-49b7-afaf-12e20e3994a3-image-registry-private-configuration\") pod \"image-registry-dd954f877-x9h9g\" (UID: \"17461b46-7ebf-49b7-afaf-12e20e3994a3\") " pod="openshift-image-registry/image-registry-dd954f877-x9h9g" Oct 30 23:41:16.117484 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:16.117456 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/17461b46-7ebf-49b7-afaf-12e20e3994a3-registry-certificates\") pod \"image-registry-dd954f877-x9h9g\" (UID: \"17461b46-7ebf-49b7-afaf-12e20e3994a3\") " pod="openshift-image-registry/image-registry-dd954f877-x9h9g" Oct 30 23:41:16.117484 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:16.117480 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qg8hn\" (UniqueName: \"kubernetes.io/projected/6382dfb5-a363-47a2-bc79-0b96f7bec0c3-kube-api-access-qg8hn\") pod \"console-operator-7cd56b8dbc-pqt2c\" (UID: \"6382dfb5-a363-47a2-bc79-0b96f7bec0c3\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-pqt2c" Oct 30 23:41:16.119703 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:16.117506 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtfn9\" (UniqueName: \"kubernetes.io/projected/4cf4abe6-da2b-41b6-9caa-9659d4f750fb-kube-api-access-rtfn9\") pod \"router-default-66d97c9c97-qqxj5\" (UID: \"4cf4abe6-da2b-41b6-9caa-9659d4f750fb\") " pod="openshift-ingress/router-default-66d97c9c97-qqxj5" Oct 30 23:41:16.119703 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:16.117534 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/10774606-8e0c-460b-bd5f-4cb6d121b30b-snapshots\") pod \"insights-operator-7c4986d449-9mqqt\" (UID: \"10774606-8e0c-460b-bd5f-4cb6d121b30b\") " pod="openshift-insights/insights-operator-7c4986d449-9mqqt" Oct 30 23:41:16.119703 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:16.117560 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/10774606-8e0c-460b-bd5f-4cb6d121b30b-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-9mqqt\" (UID: \"10774606-8e0c-460b-bd5f-4cb6d121b30b\") " pod="openshift-insights/insights-operator-7c4986d449-9mqqt" Oct 30 23:41:16.119703 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:16.117590 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/17461b46-7ebf-49b7-afaf-12e20e3994a3-installation-pull-secrets\") pod \"image-registry-dd954f877-x9h9g\" (UID: \"17461b46-7ebf-49b7-afaf-12e20e3994a3\") " pod="openshift-image-registry/image-registry-dd954f877-x9h9g" Oct 30 23:41:16.119703 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:16.117619 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/58f7cf25-f5d0-4481-a927-4a745b692f7a-image-registry-private-configuration\") pod \"image-registry-544b44f8d-5p6kz\" (UID: \"58f7cf25-f5d0-4481-a927-4a745b692f7a\") " pod="openshift-image-registry/image-registry-544b44f8d-5p6kz" Oct 30 23:41:16.119703 ip-10-0-141-180 kubenswrapper[2550]: E1030 23:41:16.117630 2550 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 23:41:16.119703 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:16.117648 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/17461b46-7ebf-49b7-afaf-12e20e3994a3-trusted-ca\") pod \"image-registry-dd954f877-x9h9g\" (UID: \"17461b46-7ebf-49b7-afaf-12e20e3994a3\") " pod="openshift-image-registry/image-registry-dd954f877-x9h9g" Oct 30 23:41:16.119703 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:16.117677 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkthh\" (UniqueName: \"kubernetes.io/projected/17461b46-7ebf-49b7-afaf-12e20e3994a3-kube-api-access-xkthh\") pod \"image-registry-dd954f877-x9h9g\" (UID: \"17461b46-7ebf-49b7-afaf-12e20e3994a3\") " pod="openshift-image-registry/image-registry-dd954f877-x9h9g" Oct 30 23:41:16.119703 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:16.117701 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/4cf4abe6-da2b-41b6-9caa-9659d4f750fb-stats-auth\") pod \"router-default-66d97c9c97-qqxj5\" (UID: \"4cf4abe6-da2b-41b6-9caa-9659d4f750fb\") " pod="openshift-ingress/router-default-66d97c9c97-qqxj5" Oct 30 23:41:16.119703 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:16.117725 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/10774606-8e0c-460b-bd5f-4cb6d121b30b-tmp\") pod \"insights-operator-7c4986d449-9mqqt\" (UID: \"10774606-8e0c-460b-bd5f-4cb6d121b30b\") " pod="openshift-insights/insights-operator-7c4986d449-9mqqt" Oct 30 23:41:16.119703 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:16.117753 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xfx9\" (UniqueName: \"kubernetes.io/projected/3af6ccf2-97f1-4598-b8ef-12eac377133b-kube-api-access-9xfx9\") pod \"kas-connection-checker-685895c7c9-f7zjp\" (UID: \"3af6ccf2-97f1-4598-b8ef-12eac377133b\") " pod="kube-system/kas-connection-checker-685895c7c9-f7zjp" Oct 30 23:41:16.119703 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:16.117796 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/58f7cf25-f5d0-4481-a927-4a745b692f7a-ca-trust-extracted\") pod \"image-registry-544b44f8d-5p6kz\" (UID: \"58f7cf25-f5d0-4481-a927-4a745b692f7a\") " pod="openshift-image-registry/image-registry-544b44f8d-5p6kz" Oct 30 23:41:16.119703 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:16.117820 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/58f7cf25-f5d0-4481-a927-4a745b692f7a-bound-sa-token\") pod \"image-registry-544b44f8d-5p6kz\" (UID: \"58f7cf25-f5d0-4481-a927-4a745b692f7a\") " pod="openshift-image-registry/image-registry-544b44f8d-5p6kz" Oct 30 23:41:16.119703 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:16.117845 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/840aaae0-a215-4ccd-b420-d6f5723b65ae-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-9qs84\" (UID: \"840aaae0-a215-4ccd-b420-d6f5723b65ae\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-9qs84" Oct 30 23:41:16.119703 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:16.117979 2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/58f7cf25-f5d0-4481-a927-4a745b692f7a-registry-certificates\") pod \"image-registry-544b44f8d-5p6kz\" (UID: \"58f7cf25-f5d0-4481-a927-4a745b692f7a\") " pod="openshift-image-registry/image-registry-544b44f8d-5p6kz" Oct 30 23:41:16.119703 ip-10-0-141-180 kubenswrapper[2550]: E1030 23:41:16.118138 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4cf4abe6-da2b-41b6-9caa-9659d4f750fb-metrics-certs podName:4cf4abe6-da2b-41b6-9caa-9659d4f750fb nodeName:}" failed. No retries permitted until 2026-10-30 23:41:16.618113513 +0000 UTC m=+32.676317422 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4cf4abe6-da2b-41b6-9caa-9659d4f750fb-metrics-certs") pod "router-default-66d97c9c97-qqxj5" (UID: "4cf4abe6-da2b-41b6-9caa-9659d4f750fb") : secret "router-metrics-certs-default" not found Oct 30 23:41:16.121349 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:16.118152 2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/10774606-8e0c-460b-bd5f-4cb6d121b30b-tmp\") pod \"insights-operator-7c4986d449-9mqqt\" (UID: \"10774606-8e0c-460b-bd5f-4cb6d121b30b\") " pod="openshift-insights/insights-operator-7c4986d449-9mqqt" Oct 30 23:41:16.121349 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:16.118198 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/29b8bd1a-beeb-495c-9f61-e3326866a1d7-metrics-tls\") pod \"dns-default-9l5zq\" (UID: \"29b8bd1a-beeb-495c-9f61-e3326866a1d7\") " pod="openshift-dns/dns-default-9l5zq" Oct 30 23:41:16.121349 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:16.118235 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfwlk\" (UniqueName: \"kubernetes.io/projected/64ae3ff5-44eb-456b-948a-c1a8320e6b07-kube-api-access-cfwlk\") pod \"kas-connection-checker-685895c7c9-lxdsf\" (UID: \"64ae3ff5-44eb-456b-948a-c1a8320e6b07\") " pod="kube-system/kas-connection-checker-685895c7c9-lxdsf" Oct 30 23:41:16.121349 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:16.118263 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b53ad6ed-90fd-4a0f-bf54-88d416ba19e9-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-rxpst\" (UID: \"b53ad6ed-90fd-4a0f-bf54-88d416ba19e9\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-rxpst" Oct 30 23:41:16.121349 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:16.118310 2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/58f7cf25-f5d0-4481-a927-4a745b692f7a-ca-trust-extracted\") pod \"image-registry-544b44f8d-5p6kz\" (UID: \"58f7cf25-f5d0-4481-a927-4a745b692f7a\") " pod="openshift-image-registry/image-registry-544b44f8d-5p6kz" Oct 30 23:41:16.121349 ip-10-0-141-180 kubenswrapper[2550]: E1030 23:41:16.118411 2550 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 23:41:16.121349 ip-10-0-141-180 kubenswrapper[2550]: E1030 23:41:16.118455 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/29b8bd1a-beeb-495c-9f61-e3326866a1d7-metrics-tls podName:29b8bd1a-beeb-495c-9f61-e3326866a1d7 nodeName:}" failed. No retries permitted until 2026-10-30 23:41:16.618440548 +0000 UTC m=+32.676644463 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/29b8bd1a-beeb-495c-9f61-e3326866a1d7-metrics-tls") pod "dns-default-9l5zq" (UID: "29b8bd1a-beeb-495c-9f61-e3326866a1d7") : secret "dns-default-metrics-tls" not found Oct 30 23:41:16.121349 ip-10-0-141-180 kubenswrapper[2550]: E1030 23:41:16.118411 2550 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 23:41:16.121349 ip-10-0-141-180 kubenswrapper[2550]: E1030 23:41:16.118609 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b53ad6ed-90fd-4a0f-bf54-88d416ba19e9-networking-console-plugin-cert podName:b53ad6ed-90fd-4a0f-bf54-88d416ba19e9 nodeName:}" failed. No retries permitted until 2026-10-30 23:41:16.618595258 +0000 UTC m=+32.676799200 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/b53ad6ed-90fd-4a0f-bf54-88d416ba19e9-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-rxpst" (UID: "b53ad6ed-90fd-4a0f-bf54-88d416ba19e9") : secret "networking-console-plugin-cert" not found Oct 30 23:41:16.121349 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:16.117819 2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/17461b46-7ebf-49b7-afaf-12e20e3994a3-ca-trust-extracted\") pod \"image-registry-dd954f877-x9h9g\" (UID: \"17461b46-7ebf-49b7-afaf-12e20e3994a3\") " pod="openshift-image-registry/image-registry-dd954f877-x9h9g" Oct 30 23:41:16.121349 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:16.119026 2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/17461b46-7ebf-49b7-afaf-12e20e3994a3-trusted-ca\") pod \"image-registry-dd954f877-x9h9g\" (UID: \"17461b46-7ebf-49b7-afaf-12e20e3994a3\") " pod="openshift-image-registry/image-registry-dd954f877-x9h9g" Oct 30 23:41:16.121349 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:16.119238 2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/17461b46-7ebf-49b7-afaf-12e20e3994a3-registry-certificates\") pod \"image-registry-dd954f877-x9h9g\" (UID: \"17461b46-7ebf-49b7-afaf-12e20e3994a3\") " pod="openshift-image-registry/image-registry-dd954f877-x9h9g" Oct 30 23:41:16.121349 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:16.119392 2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/10774606-8e0c-460b-bd5f-4cb6d121b30b-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-9mqqt\" (UID: \"10774606-8e0c-460b-bd5f-4cb6d121b30b\") " pod="openshift-insights/insights-operator-7c4986d449-9mqqt" Oct 30 23:41:16.121349 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:16.119520 2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/10774606-8e0c-460b-bd5f-4cb6d121b30b-snapshots\") pod \"insights-operator-7c4986d449-9mqqt\" (UID: \"10774606-8e0c-460b-bd5f-4cb6d121b30b\") " pod="openshift-insights/insights-operator-7c4986d449-9mqqt" Oct 30 23:41:16.121349 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:16.120227 2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9700c98a-b858-45a5-9025-8d75f3c14044-cert\") pod \"ingress-canary-46zqw\" (UID: \"9700c98a-b858-45a5-9025-8d75f3c14044\") " pod="openshift-ingress-canary/ingress-canary-46zqw" Oct 30 23:41:16.121349 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:16.120278 2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/10774606-8e0c-460b-bd5f-4cb6d121b30b-serving-cert\") pod \"insights-operator-7c4986d449-9mqqt\" (UID: \"10774606-8e0c-460b-bd5f-4cb6d121b30b\") " pod="openshift-insights/insights-operator-7c4986d449-9mqqt" Oct 30 23:41:16.122044 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:16.120473 2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/4cf4abe6-da2b-41b6-9caa-9659d4f750fb-default-certificate\") pod \"router-default-66d97c9c97-qqxj5\" (UID: \"4cf4abe6-da2b-41b6-9caa-9659d4f750fb\") " pod="openshift-ingress/router-default-66d97c9c97-qqxj5" Oct 30 23:41:16.122044 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:16.120547 2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/58f7cf25-f5d0-4481-a927-4a745b692f7a-installation-pull-secrets\") pod \"image-registry-544b44f8d-5p6kz\" (UID: \"58f7cf25-f5d0-4481-a927-4a745b692f7a\") " pod="openshift-image-registry/image-registry-544b44f8d-5p6kz" Oct 30 23:41:16.122044 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:16.120663 2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/03211c6a-9761-419c-839b-21ca413d5bb2-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-httg9\" (UID: \"03211c6a-9761-419c-839b-21ca413d5bb2\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-httg9" Oct 30 23:41:16.122044 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:16.121433 2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6382dfb5-a363-47a2-bc79-0b96f7bec0c3-serving-cert\") pod \"console-operator-7cd56b8dbc-pqt2c\" (UID: \"6382dfb5-a363-47a2-bc79-0b96f7bec0c3\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-pqt2c" Oct 30 23:41:16.122044 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:16.121855 2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/58f7cf25-f5d0-4481-a927-4a745b692f7a-image-registry-private-configuration\") pod \"image-registry-544b44f8d-5p6kz\" (UID: \"58f7cf25-f5d0-4481-a927-4a745b692f7a\") " pod="openshift-image-registry/image-registry-544b44f8d-5p6kz" Oct 30 23:41:16.122253 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:16.122232 2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/4cf4abe6-da2b-41b6-9caa-9659d4f750fb-stats-auth\") pod \"router-default-66d97c9c97-qqxj5\" (UID: \"4cf4abe6-da2b-41b6-9caa-9659d4f750fb\") " pod="openshift-ingress/router-default-66d97c9c97-qqxj5" Oct 30 23:41:16.122522 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:16.122503 2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/17461b46-7ebf-49b7-afaf-12e20e3994a3-image-registry-private-configuration\") pod \"image-registry-dd954f877-x9h9g\" (UID: \"17461b46-7ebf-49b7-afaf-12e20e3994a3\") " pod="openshift-image-registry/image-registry-dd954f877-x9h9g" Oct 30 23:41:16.122614 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:16.122582 2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f464ef41-45c9-4ff4-b410-e2d9d6cdbdf5-serving-cert\") pod \"service-ca-operator-544d749c8c-fbgth\" (UID: \"f464ef41-45c9-4ff4-b410-e2d9d6cdbdf5\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-fbgth" Oct 30 23:41:16.122948 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:16.122927 2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/840aaae0-a215-4ccd-b420-d6f5723b65ae-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-9qs84\" (UID: \"840aaae0-a215-4ccd-b420-d6f5723b65ae\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-9qs84" Oct 30 23:41:16.123149 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:16.123132 2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/17461b46-7ebf-49b7-afaf-12e20e3994a3-installation-pull-secrets\") pod \"image-registry-dd954f877-x9h9g\" (UID: \"17461b46-7ebf-49b7-afaf-12e20e3994a3\") " pod="openshift-image-registry/image-registry-dd954f877-x9h9g" Oct 30 23:41:16.177866 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:16.177844 2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/17461b46-7ebf-49b7-afaf-12e20e3994a3-bound-sa-token\") pod \"image-registry-dd954f877-x9h9g\" (UID: \"17461b46-7ebf-49b7-afaf-12e20e3994a3\") " pod="openshift-image-registry/image-registry-dd954f877-x9h9g" Oct 30 23:41:16.180317 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:16.180292 2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-67888\" (UniqueName: \"kubernetes.io/projected/a4f35692-894d-4d41-8b9f-3f5ba3c529b1-kube-api-access-67888\") pod \"kas-connection-checker-685895c7c9-l4d2m\" (UID: \"a4f35692-894d-4d41-8b9f-3f5ba3c529b1\") " pod="kube-system/kas-connection-checker-685895c7c9-l4d2m" Oct 30 23:41:16.180688 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:16.180667 2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ztdxk\" (UniqueName: \"kubernetes.io/projected/f464ef41-45c9-4ff4-b410-e2d9d6cdbdf5-kube-api-access-ztdxk\") pod \"service-ca-operator-544d749c8c-fbgth\" (UID: \"f464ef41-45c9-4ff4-b410-e2d9d6cdbdf5\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-fbgth" Oct 30 23:41:16.180897 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:16.180871 2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9l2q\" (UniqueName: \"kubernetes.io/projected/f8eb4b4c-8225-436b-b1e8-7775414a1ca3-kube-api-access-k9l2q\") pod \"volume-data-source-validator-58c9f5f46d-tcxmv\" (UID: \"f8eb4b4c-8225-436b-b1e8-7775414a1ca3\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-tcxmv" Oct 30 23:41:16.180968 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:16.180904 2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-45fmr\" (UniqueName: \"kubernetes.io/projected/be5540b5-7b10-45b2-b40c-723efd9f1b01-kube-api-access-45fmr\") pod \"cluster-samples-operator-6ddb6c8b94-vl8t5\" (UID: \"be5540b5-7b10-45b2-b40c-723efd9f1b01\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vl8t5" Oct 30 23:41:16.181821 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:16.181763 2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sx9pp\" (UniqueName: \"kubernetes.io/projected/58f7cf25-f5d0-4481-a927-4a745b692f7a-kube-api-access-sx9pp\") pod \"image-registry-544b44f8d-5p6kz\" (UID: \"58f7cf25-f5d0-4481-a927-4a745b692f7a\") " pod="openshift-image-registry/image-registry-544b44f8d-5p6kz" Oct 30 23:41:16.182135 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:16.182089 2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n65gn\" (UniqueName: \"kubernetes.io/projected/10774606-8e0c-460b-bd5f-4cb6d121b30b-kube-api-access-n65gn\") pod \"insights-operator-7c4986d449-9mqqt\" (UID: \"10774606-8e0c-460b-bd5f-4cb6d121b30b\") " pod="openshift-insights/insights-operator-7c4986d449-9mqqt" Oct 30 23:41:16.184070 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:16.184001 2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfwlk\" (UniqueName: \"kubernetes.io/projected/64ae3ff5-44eb-456b-948a-c1a8320e6b07-kube-api-access-cfwlk\") pod \"kas-connection-checker-685895c7c9-lxdsf\" (UID: \"64ae3ff5-44eb-456b-948a-c1a8320e6b07\") " pod="kube-system/kas-connection-checker-685895c7c9-lxdsf" Oct 30 23:41:16.185218 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:16.185190 2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xfx9\" (UniqueName: \"kubernetes.io/projected/3af6ccf2-97f1-4598-b8ef-12eac377133b-kube-api-access-9xfx9\") pod \"kas-connection-checker-685895c7c9-f7zjp\" (UID: \"3af6ccf2-97f1-4598-b8ef-12eac377133b\") " pod="kube-system/kas-connection-checker-685895c7c9-f7zjp" Oct 30 23:41:16.185304 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:16.185275 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-lxdsf" Oct 30 23:41:16.185542 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:16.185523 2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5x8x6\" (UniqueName: \"kubernetes.io/projected/03211c6a-9761-419c-839b-21ca413d5bb2-kube-api-access-5x8x6\") pod \"cluster-monitoring-operator-884576c57-httg9\" (UID: \"03211c6a-9761-419c-839b-21ca413d5bb2\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-httg9" Oct 30 23:41:16.185884 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:16.185858 2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vsfpv\" (UniqueName: \"kubernetes.io/projected/840aaae0-a215-4ccd-b420-d6f5723b65ae-kube-api-access-vsfpv\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-9qs84\" (UID: \"840aaae0-a215-4ccd-b420-d6f5723b65ae\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-9qs84" Oct 30 23:41:16.186182 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:16.186161 2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vj8vx\" (UniqueName: \"kubernetes.io/projected/29b8bd1a-beeb-495c-9f61-e3326866a1d7-kube-api-access-vj8vx\") pod \"dns-default-9l5zq\" (UID: \"29b8bd1a-beeb-495c-9f61-e3326866a1d7\") " pod="openshift-dns/dns-default-9l5zq" Oct 30 23:41:16.186233 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:16.186174 2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xn7l\" (UniqueName: \"kubernetes.io/projected/9700c98a-b858-45a5-9025-8d75f3c14044-kube-api-access-5xn7l\") pod \"ingress-canary-46zqw\" (UID: \"9700c98a-b858-45a5-9025-8d75f3c14044\") " pod="openshift-ingress-canary/ingress-canary-46zqw" Oct 30 23:41:16.186385 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:16.186371 2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xkthh\" (UniqueName: \"kubernetes.io/projected/17461b46-7ebf-49b7-afaf-12e20e3994a3-kube-api-access-xkthh\") pod \"image-registry-dd954f877-x9h9g\" (UID: \"17461b46-7ebf-49b7-afaf-12e20e3994a3\") " pod="openshift-image-registry/image-registry-dd954f877-x9h9g" Oct 30 23:41:16.189606 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:16.189584 2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qg8hn\" (UniqueName: \"kubernetes.io/projected/6382dfb5-a363-47a2-bc79-0b96f7bec0c3-kube-api-access-qg8hn\") pod \"console-operator-7cd56b8dbc-pqt2c\" (UID: \"6382dfb5-a363-47a2-bc79-0b96f7bec0c3\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-pqt2c" Oct 30 23:41:16.190134 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:16.190110 2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vv9dt\" (UniqueName: \"kubernetes.io/projected/27ccca21-5e99-485f-8b5c-ca8ec5177668-kube-api-access-vv9dt\") pod \"network-check-source-6f78d59f99-xrk8k\" (UID: \"27ccca21-5e99-485f-8b5c-ca8ec5177668\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-xrk8k" Oct 30 23:41:16.190434 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:16.190413 2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/58f7cf25-f5d0-4481-a927-4a745b692f7a-bound-sa-token\") pod \"image-registry-544b44f8d-5p6kz\" (UID: \"58f7cf25-f5d0-4481-a927-4a745b692f7a\") " pod="openshift-image-registry/image-registry-544b44f8d-5p6kz" Oct 30 23:41:16.190512 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:16.190473 2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtfn9\" (UniqueName: \"kubernetes.io/projected/4cf4abe6-da2b-41b6-9caa-9659d4f750fb-kube-api-access-rtfn9\") pod \"router-default-66d97c9c97-qqxj5\" (UID: \"4cf4abe6-da2b-41b6-9caa-9659d4f750fb\") " pod="openshift-ingress/router-default-66d97c9c97-qqxj5" Oct 30 23:41:16.191115 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:16.191101 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-l4d2m" Oct 30 23:41:16.204157 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:16.204136 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-pqt2c" Oct 30 23:41:16.243369 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:16.243153 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-9qs84" Oct 30 23:41:16.253945 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:16.253760 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-fbgth" Oct 30 23:41:16.260782 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:16.260612 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-tcxmv" Oct 30 23:41:16.287786 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:16.278423 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-46zqw" Oct 30 23:41:16.287786 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:16.284544 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-xrk8k" Oct 30 23:41:16.296543 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:16.295679 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-9mqqt" Oct 30 23:41:16.316644 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:16.310941 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:41:16.479097 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:16.479070 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-f7zjp" Oct 30 23:41:16.585078 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:16.585016 2550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-pqt2c"] Oct 30 23:41:16.591721 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:16.588493 2550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-lxdsf"] Oct 30 23:41:16.591721 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:16.589654 2550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-l4d2m"] Oct 30 23:41:16.591721 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:41:16.591288 2550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64ae3ff5_44eb_456b_948a_c1a8320e6b07.slice/crio-722d14c37f4ca9c2f6ea9ad958e59ca48f6fb4bb8ffc975ccc53a0d713fcd4fc WatchSource:0}: Error finding container 722d14c37f4ca9c2f6ea9ad958e59ca48f6fb4bb8ffc975ccc53a0d713fcd4fc: Status 404 returned error can't find the container with id 722d14c37f4ca9c2f6ea9ad958e59ca48f6fb4bb8ffc975ccc53a0d713fcd4fc Oct 30 23:41:16.595798 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:41:16.595781 2550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4f35692_894d_4d41_8b9f_3f5ba3c529b1.slice/crio-0fd7a1669bcc37cc84500ce975c3ac8b58e4c1e03de0b5f9245530d20c9dc527 WatchSource:0}: Error finding container 0fd7a1669bcc37cc84500ce975c3ac8b58e4c1e03de0b5f9245530d20c9dc527: Status 404 returned error can't find the container with id 0fd7a1669bcc37cc84500ce975c3ac8b58e4c1e03de0b5f9245530d20c9dc527 Oct 30 23:41:16.623297 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:16.623265 2550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-pqt2c" event={"ID":"6382dfb5-a363-47a2-bc79-0b96f7bec0c3","Type":"ContainerStarted","Data":"2cd1d530b1d2e4fbf2016043d891c3f05ed64bc1e150f43ac461b47dd0c04439"} Oct 30 23:41:16.624704 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:16.624678 2550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-l4d2m" event={"ID":"a4f35692-894d-4d41-8b9f-3f5ba3c529b1","Type":"ContainerStarted","Data":"0fd7a1669bcc37cc84500ce975c3ac8b58e4c1e03de0b5f9245530d20c9dc527"} Oct 30 23:41:16.625466 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:16.625439 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4cf4abe6-da2b-41b6-9caa-9659d4f750fb-service-ca-bundle\") pod \"router-default-66d97c9c97-qqxj5\" (UID: \"4cf4abe6-da2b-41b6-9caa-9659d4f750fb\") " pod="openshift-ingress/router-default-66d97c9c97-qqxj5" Oct 30 23:41:16.625560 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:16.625484 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4cf4abe6-da2b-41b6-9caa-9659d4f750fb-metrics-certs\") pod \"router-default-66d97c9c97-qqxj5\" (UID: \"4cf4abe6-da2b-41b6-9caa-9659d4f750fb\") " pod="openshift-ingress/router-default-66d97c9c97-qqxj5" Oct 30 23:41:16.625645 ip-10-0-141-180 kubenswrapper[2550]: E1030 23:41:16.625626 2550 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 23:41:16.625701 ip-10-0-141-180 kubenswrapper[2550]: E1030 23:41:16.625655 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4cf4abe6-da2b-41b6-9caa-9659d4f750fb-service-ca-bundle podName:4cf4abe6-da2b-41b6-9caa-9659d4f750fb nodeName:}" failed. No retries permitted until 2026-10-30 23:41:17.625632842 +0000 UTC m=+33.683836759 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/4cf4abe6-da2b-41b6-9caa-9659d4f750fb-service-ca-bundle") pod "router-default-66d97c9c97-qqxj5" (UID: "4cf4abe6-da2b-41b6-9caa-9659d4f750fb") : configmap references non-existent config key: service-ca.crt Oct 30 23:41:16.625701 ip-10-0-141-180 kubenswrapper[2550]: E1030 23:41:16.625685 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4cf4abe6-da2b-41b6-9caa-9659d4f750fb-metrics-certs podName:4cf4abe6-da2b-41b6-9caa-9659d4f750fb nodeName:}" failed. No retries permitted until 2026-10-30 23:41:17.625674608 +0000 UTC m=+33.683878508 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4cf4abe6-da2b-41b6-9caa-9659d4f750fb-metrics-certs") pod "router-default-66d97c9c97-qqxj5" (UID: "4cf4abe6-da2b-41b6-9caa-9659d4f750fb") : secret "router-metrics-certs-default" not found Oct 30 23:41:16.625818 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:16.625709 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/29b8bd1a-beeb-495c-9f61-e3326866a1d7-metrics-tls\") pod \"dns-default-9l5zq\" (UID: \"29b8bd1a-beeb-495c-9f61-e3326866a1d7\") " pod="openshift-dns/dns-default-9l5zq" Oct 30 23:41:16.625818 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:16.625749 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b53ad6ed-90fd-4a0f-bf54-88d416ba19e9-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-rxpst\" (UID: \"b53ad6ed-90fd-4a0f-bf54-88d416ba19e9\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-rxpst" Oct 30 23:41:16.625818 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:16.625792 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/be5540b5-7b10-45b2-b40c-723efd9f1b01-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-vl8t5\" (UID: \"be5540b5-7b10-45b2-b40c-723efd9f1b01\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vl8t5" Oct 30 23:41:16.625970 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:16.625824 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/17461b46-7ebf-49b7-afaf-12e20e3994a3-registry-tls\") pod \"image-registry-dd954f877-x9h9g\" (UID: \"17461b46-7ebf-49b7-afaf-12e20e3994a3\") " pod="openshift-image-registry/image-registry-dd954f877-x9h9g" Oct 30 23:41:16.625970 ip-10-0-141-180 kubenswrapper[2550]: E1030 23:41:16.625846 2550 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 23:41:16.625970 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:16.625865 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/58f7cf25-f5d0-4481-a927-4a745b692f7a-registry-tls\") pod \"image-registry-544b44f8d-5p6kz\" (UID: \"58f7cf25-f5d0-4481-a927-4a745b692f7a\") " pod="openshift-image-registry/image-registry-544b44f8d-5p6kz" Oct 30 23:41:16.625970 ip-10-0-141-180 kubenswrapper[2550]: E1030 23:41:16.625887 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/29b8bd1a-beeb-495c-9f61-e3326866a1d7-metrics-tls podName:29b8bd1a-beeb-495c-9f61-e3326866a1d7 nodeName:}" failed. No retries permitted until 2026-10-30 23:41:17.625872087 +0000 UTC m=+33.684075981 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/29b8bd1a-beeb-495c-9f61-e3326866a1d7-metrics-tls") pod "dns-default-9l5zq" (UID: "29b8bd1a-beeb-495c-9f61-e3326866a1d7") : secret "dns-default-metrics-tls" not found Oct 30 23:41:16.625970 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:16.625928 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/03211c6a-9761-419c-839b-21ca413d5bb2-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-httg9\" (UID: \"03211c6a-9761-419c-839b-21ca413d5bb2\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-httg9" Oct 30 23:41:16.625970 ip-10-0-141-180 kubenswrapper[2550]: E1030 23:41:16.625938 2550 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 23:41:16.625970 ip-10-0-141-180 kubenswrapper[2550]: E1030 23:41:16.625950 2550 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-544b44f8d-5p6kz: secret "image-registry-tls" not found Oct 30 23:41:16.626301 ip-10-0-141-180 kubenswrapper[2550]: E1030 23:41:16.625992 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/58f7cf25-f5d0-4481-a927-4a745b692f7a-registry-tls podName:58f7cf25-f5d0-4481-a927-4a745b692f7a nodeName:}" failed. No retries permitted until 2026-10-30 23:41:17.625972612 +0000 UTC m=+33.684176524 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/58f7cf25-f5d0-4481-a927-4a745b692f7a-registry-tls") pod "image-registry-544b44f8d-5p6kz" (UID: "58f7cf25-f5d0-4481-a927-4a745b692f7a") : secret "image-registry-tls" not found Oct 30 23:41:16.626301 ip-10-0-141-180 kubenswrapper[2550]: E1030 23:41:16.626042 2550 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 23:41:16.626301 ip-10-0-141-180 kubenswrapper[2550]: E1030 23:41:16.626080 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/03211c6a-9761-419c-839b-21ca413d5bb2-cluster-monitoring-operator-tls podName:03211c6a-9761-419c-839b-21ca413d5bb2 nodeName:}" failed. No retries permitted until 2026-10-30 23:41:17.626068689 +0000 UTC m=+33.684272581 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/03211c6a-9761-419c-839b-21ca413d5bb2-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-httg9" (UID: "03211c6a-9761-419c-839b-21ca413d5bb2") : secret "cluster-monitoring-operator-tls" not found Oct 30 23:41:16.626301 ip-10-0-141-180 kubenswrapper[2550]: E1030 23:41:16.626046 2550 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 23:41:16.626301 ip-10-0-141-180 kubenswrapper[2550]: E1030 23:41:16.626092 2550 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 23:41:16.626301 ip-10-0-141-180 kubenswrapper[2550]: E1030 23:41:16.626112 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b53ad6ed-90fd-4a0f-bf54-88d416ba19e9-networking-console-plugin-cert podName:b53ad6ed-90fd-4a0f-bf54-88d416ba19e9 nodeName:}" failed. No retries permitted until 2026-10-30 23:41:17.626103644 +0000 UTC m=+33.684307537 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/b53ad6ed-90fd-4a0f-bf54-88d416ba19e9-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-rxpst" (UID: "b53ad6ed-90fd-4a0f-bf54-88d416ba19e9") : secret "networking-console-plugin-cert" not found Oct 30 23:41:16.626301 ip-10-0-141-180 kubenswrapper[2550]: E1030 23:41:16.626136 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/be5540b5-7b10-45b2-b40c-723efd9f1b01-samples-operator-tls podName:be5540b5-7b10-45b2-b40c-723efd9f1b01 nodeName:}" failed. No retries permitted until 2026-10-30 23:41:17.626119509 +0000 UTC m=+33.684323404 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/be5540b5-7b10-45b2-b40c-723efd9f1b01-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-vl8t5" (UID: "be5540b5-7b10-45b2-b40c-723efd9f1b01") : secret "samples-operator-tls" not found Oct 30 23:41:16.626301 ip-10-0-141-180 kubenswrapper[2550]: E1030 23:41:16.626159 2550 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 23:41:16.626301 ip-10-0-141-180 kubenswrapper[2550]: E1030 23:41:16.626167 2550 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-dd954f877-x9h9g: secret "image-registry-tls" not found Oct 30 23:41:16.626301 ip-10-0-141-180 kubenswrapper[2550]: E1030 23:41:16.626198 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/17461b46-7ebf-49b7-afaf-12e20e3994a3-registry-tls podName:17461b46-7ebf-49b7-afaf-12e20e3994a3 nodeName:}" failed. No retries permitted until 2026-10-30 23:41:17.626187259 +0000 UTC m=+33.684391172 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/17461b46-7ebf-49b7-afaf-12e20e3994a3-registry-tls") pod "image-registry-dd954f877-x9h9g" (UID: "17461b46-7ebf-49b7-afaf-12e20e3994a3") : secret "image-registry-tls" not found Oct 30 23:41:16.626301 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:16.626237 2550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-lxdsf" event={"ID":"64ae3ff5-44eb-456b-948a-c1a8320e6b07","Type":"ContainerStarted","Data":"722d14c37f4ca9c2f6ea9ad958e59ca48f6fb4bb8ffc975ccc53a0d713fcd4fc"} Oct 30 23:41:16.673107 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:16.673081 2550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-9qs84"] Oct 30 23:41:16.681257 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:41:16.681229 2550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod840aaae0_a215_4ccd_b420_d6f5723b65ae.slice/crio-01cae760460a3c29096c2785faece01841f55c4c9c9de869c85e2ef7134b502c WatchSource:0}: Error finding container 01cae760460a3c29096c2785faece01841f55c4c9c9de869c85e2ef7134b502c: Status 404 returned error can't find the container with id 01cae760460a3c29096c2785faece01841f55c4c9c9de869c85e2ef7134b502c Oct 30 23:41:16.684950 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:16.684712 2550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-tcxmv"] Oct 30 23:41:16.687727 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:41:16.687707 2550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8eb4b4c_8225_436b_b1e8_7775414a1ca3.slice/crio-6b58028156e6606c849a36e8b17c9e190dc04bfc8a69a1f24a667e08b4a26210 WatchSource:0}: Error finding container 6b58028156e6606c849a36e8b17c9e190dc04bfc8a69a1f24a667e08b4a26210: Status 404 returned error can't find the container with id 6b58028156e6606c849a36e8b17c9e190dc04bfc8a69a1f24a667e08b4a26210 Oct 30 23:41:16.726458 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:16.726434 2550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-xrk8k"] Oct 30 23:41:16.733843 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:41:16.731439 2550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27ccca21_5e99_485f_8b5c_ca8ec5177668.slice/crio-c9010557cbcc988d15e3e177b0de68e23621b842a825a094cc2557634101e5c4 WatchSource:0}: Error finding container c9010557cbcc988d15e3e177b0de68e23621b842a825a094cc2557634101e5c4: Status 404 returned error can't find the container with id c9010557cbcc988d15e3e177b0de68e23621b842a825a094cc2557634101e5c4 Oct 30 23:41:16.733843 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:16.731591 2550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-fbgth"] Oct 30 23:41:16.733843 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:16.732362 2550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-46zqw"] Oct 30 23:41:16.733843 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:16.733364 2550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-9mqqt"] Oct 30 23:41:16.733843 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:41:16.733731 2550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9700c98a_b858_45a5_9025_8d75f3c14044.slice/crio-ae3a0d475b2be1b25689edf2744177911624228a66bdbfc2ba94c3dccad12b36 WatchSource:0}: Error finding container ae3a0d475b2be1b25689edf2744177911624228a66bdbfc2ba94c3dccad12b36: Status 404 returned error can't find the container with id ae3a0d475b2be1b25689edf2744177911624228a66bdbfc2ba94c3dccad12b36 Oct 30 23:41:16.734502 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:41:16.734487 2550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10774606_8e0c_460b_bd5f_4cb6d121b30b.slice/crio-8738eb3d4e62b725466ec53a2c859cd4d77af3b229836134893039fd90cf182e WatchSource:0}: Error finding container 8738eb3d4e62b725466ec53a2c859cd4d77af3b229836134893039fd90cf182e: Status 404 returned error can't find the container with id 8738eb3d4e62b725466ec53a2c859cd4d77af3b229836134893039fd90cf182e Oct 30 23:41:16.738485 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:41:16.738466 2550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf464ef41_45c9_4ff4_b410_e2d9d6cdbdf5.slice/crio-4d24e3019382f392346762ad0cdd801460b4a56c586d5f30a283293c21f78d5c WatchSource:0}: Error finding container 4d24e3019382f392346762ad0cdd801460b4a56c586d5f30a283293c21f78d5c: Status 404 returned error can't find the container with id 4d24e3019382f392346762ad0cdd801460b4a56c586d5f30a283293c21f78d5c Oct 30 23:41:16.849663 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:16.849590 2550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-f7zjp"] Oct 30 23:41:16.853901 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:41:16.853846 2550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3af6ccf2_97f1_4598_b8ef_12eac377133b.slice/crio-719c8f74b4146c489029a3cae27c9f4cc9d6c84b8ff50c6abfc15a23ef334941 WatchSource:0}: Error finding container 719c8f74b4146c489029a3cae27c9f4cc9d6c84b8ff50c6abfc15a23ef334941: Status 404 returned error can't find the container with id 719c8f74b4146c489029a3cae27c9f4cc9d6c84b8ff50c6abfc15a23ef334941 Oct 30 23:41:17.422819 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:17.422782 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pt58s" Oct 30 23:41:17.422819 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:17.422803 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kgdzs" Oct 30 23:41:17.423051 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:17.422828 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7wrh7" Oct 30 23:41:17.432170 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:17.432147 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/145dd323-9370-4b9d-a96e-8b93d0a47a7c-metrics-certs\") pod \"network-metrics-daemon-pt58s\" (UID: \"145dd323-9370-4b9d-a96e-8b93d0a47a7c\") " pod="openshift-multus/network-metrics-daemon-pt58s" Oct 30 23:41:17.497224 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:17.497194 2550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 23:41:17.497651 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:17.497499 2550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 23:41:17.497651 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:17.497514 2550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-294w8\"" Oct 30 23:41:17.497651 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:17.497538 2550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-bmqk5\"" Oct 30 23:41:17.497651 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:17.497575 2550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-554bl\"" Oct 30 23:41:17.503209 ip-10-0-141-180 kubenswrapper[2550]: E1030 23:41:17.503193 2550 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 30 23:41:17.503266 ip-10-0-141-180 kubenswrapper[2550]: E1030 23:41:17.503255 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/145dd323-9370-4b9d-a96e-8b93d0a47a7c-metrics-certs podName:145dd323-9370-4b9d-a96e-8b93d0a47a7c nodeName:}" failed. No retries permitted until 2026-10-30 23:41:49.503235176 +0000 UTC m=+65.561439073 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/145dd323-9370-4b9d-a96e-8b93d0a47a7c-metrics-certs") pod "network-metrics-daemon-pt58s" (UID: "145dd323-9370-4b9d-a96e-8b93d0a47a7c") : secret "metrics-daemon-secret" not found Oct 30 23:41:17.533395 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:17.533371 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqmnp\" (UniqueName: \"kubernetes.io/projected/7728abfe-1747-422e-b8e2-c91c9de83d09-kube-api-access-jqmnp\") pod \"network-check-target-kgdzs\" (UID: \"7728abfe-1747-422e-b8e2-c91c9de83d09\") " pod="openshift-network-diagnostics/network-check-target-kgdzs" Oct 30 23:41:17.533488 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:17.533415 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/97da2086-b00d-4f03-b502-56de5ee3d90d-original-pull-secret\") pod \"global-pull-secret-syncer-7wrh7\" (UID: \"97da2086-b00d-4f03-b502-56de5ee3d90d\") " pod="kube-system/global-pull-secret-syncer-7wrh7" Oct 30 23:41:17.535724 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:17.535700 2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/97da2086-b00d-4f03-b502-56de5ee3d90d-original-pull-secret\") pod \"global-pull-secret-syncer-7wrh7\" (UID: \"97da2086-b00d-4f03-b502-56de5ee3d90d\") " pod="kube-system/global-pull-secret-syncer-7wrh7" Oct 30 23:41:17.535960 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:17.535943 2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqmnp\" (UniqueName: \"kubernetes.io/projected/7728abfe-1747-422e-b8e2-c91c9de83d09-kube-api-access-jqmnp\") pod \"network-check-target-kgdzs\" (UID: \"7728abfe-1747-422e-b8e2-c91c9de83d09\") " pod="openshift-network-diagnostics/network-check-target-kgdzs" Oct 30 23:41:17.630688 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:17.630655 2550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-l4d2m" event={"ID":"a4f35692-894d-4d41-8b9f-3f5ba3c529b1","Type":"ContainerStarted","Data":"2d8e3568d6329dbdef3066b2a80fc27e7f809a5e837ea723703c0aaaef03d41e"} Oct 30 23:41:17.631783 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:17.631755 2550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-9mqqt" event={"ID":"10774606-8e0c-460b-bd5f-4cb6d121b30b","Type":"ContainerStarted","Data":"8738eb3d4e62b725466ec53a2c859cd4d77af3b229836134893039fd90cf182e"} Oct 30 23:41:17.633111 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:17.633086 2550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-xrk8k" event={"ID":"27ccca21-5e99-485f-8b5c-ca8ec5177668","Type":"ContainerStarted","Data":"c9010557cbcc988d15e3e177b0de68e23621b842a825a094cc2557634101e5c4"} Oct 30 23:41:17.633982 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:17.633958 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/29b8bd1a-beeb-495c-9f61-e3326866a1d7-metrics-tls\") pod \"dns-default-9l5zq\" (UID: \"29b8bd1a-beeb-495c-9f61-e3326866a1d7\") " pod="openshift-dns/dns-default-9l5zq" Oct 30 23:41:17.634065 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:17.634004 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b53ad6ed-90fd-4a0f-bf54-88d416ba19e9-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-rxpst\" (UID: \"b53ad6ed-90fd-4a0f-bf54-88d416ba19e9\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-rxpst" Oct 30 23:41:17.634065 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:17.634043 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/be5540b5-7b10-45b2-b40c-723efd9f1b01-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-vl8t5\" (UID: \"be5540b5-7b10-45b2-b40c-723efd9f1b01\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vl8t5" Oct 30 23:41:17.634151 ip-10-0-141-180 kubenswrapper[2550]: E1030 23:41:17.634072 2550 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 23:41:17.634151 ip-10-0-141-180 kubenswrapper[2550]: E1030 23:41:17.634116 2550 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 23:41:17.634151 ip-10-0-141-180 kubenswrapper[2550]: E1030 23:41:17.634137 2550 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 23:41:17.634151 ip-10-0-141-180 kubenswrapper[2550]: E1030 23:41:17.634143 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/29b8bd1a-beeb-495c-9f61-e3326866a1d7-metrics-tls podName:29b8bd1a-beeb-495c-9f61-e3326866a1d7 nodeName:}" failed. No retries permitted until 2026-10-30 23:41:19.634123765 +0000 UTC m=+35.692327675 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/29b8bd1a-beeb-495c-9f61-e3326866a1d7-metrics-tls") pod "dns-default-9l5zq" (UID: "29b8bd1a-beeb-495c-9f61-e3326866a1d7") : secret "dns-default-metrics-tls" not found Oct 30 23:41:17.634151 ip-10-0-141-180 kubenswrapper[2550]: E1030 23:41:17.634141 2550 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 23:41:17.634292 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:17.634075 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/17461b46-7ebf-49b7-afaf-12e20e3994a3-registry-tls\") pod \"image-registry-dd954f877-x9h9g\" (UID: \"17461b46-7ebf-49b7-afaf-12e20e3994a3\") " pod="openshift-image-registry/image-registry-dd954f877-x9h9g" Oct 30 23:41:17.634292 ip-10-0-141-180 kubenswrapper[2550]: E1030 23:41:17.634163 2550 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-dd954f877-x9h9g: secret "image-registry-tls" not found Oct 30 23:41:17.634292 ip-10-0-141-180 kubenswrapper[2550]: E1030 23:41:17.634171 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b53ad6ed-90fd-4a0f-bf54-88d416ba19e9-networking-console-plugin-cert podName:b53ad6ed-90fd-4a0f-bf54-88d416ba19e9 nodeName:}" failed. No retries permitted until 2026-10-30 23:41:19.634155308 +0000 UTC m=+35.692359217 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/b53ad6ed-90fd-4a0f-bf54-88d416ba19e9-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-rxpst" (UID: "b53ad6ed-90fd-4a0f-bf54-88d416ba19e9") : secret "networking-console-plugin-cert" not found Oct 30 23:41:17.634292 ip-10-0-141-180 kubenswrapper[2550]: E1030 23:41:17.634194 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/be5540b5-7b10-45b2-b40c-723efd9f1b01-samples-operator-tls podName:be5540b5-7b10-45b2-b40c-723efd9f1b01 nodeName:}" failed. No retries permitted until 2026-10-30 23:41:19.634184964 +0000 UTC m=+35.692388864 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/be5540b5-7b10-45b2-b40c-723efd9f1b01-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-vl8t5" (UID: "be5540b5-7b10-45b2-b40c-723efd9f1b01") : secret "samples-operator-tls" not found Oct 30 23:41:17.634292 ip-10-0-141-180 kubenswrapper[2550]: E1030 23:41:17.634212 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/17461b46-7ebf-49b7-afaf-12e20e3994a3-registry-tls podName:17461b46-7ebf-49b7-afaf-12e20e3994a3 nodeName:}" failed. No retries permitted until 2026-10-30 23:41:19.63420345 +0000 UTC m=+35.692407360 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/17461b46-7ebf-49b7-afaf-12e20e3994a3-registry-tls") pod "image-registry-dd954f877-x9h9g" (UID: "17461b46-7ebf-49b7-afaf-12e20e3994a3") : secret "image-registry-tls" not found Oct 30 23:41:17.634292 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:17.634247 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/58f7cf25-f5d0-4481-a927-4a745b692f7a-registry-tls\") pod \"image-registry-544b44f8d-5p6kz\" (UID: \"58f7cf25-f5d0-4481-a927-4a745b692f7a\") " pod="openshift-image-registry/image-registry-544b44f8d-5p6kz" Oct 30 23:41:17.634292 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:17.634275 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/03211c6a-9761-419c-839b-21ca413d5bb2-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-httg9\" (UID: \"03211c6a-9761-419c-839b-21ca413d5bb2\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-httg9" Oct 30 23:41:17.634589 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:17.634306 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4cf4abe6-da2b-41b6-9caa-9659d4f750fb-service-ca-bundle\") pod \"router-default-66d97c9c97-qqxj5\" (UID: \"4cf4abe6-da2b-41b6-9caa-9659d4f750fb\") " pod="openshift-ingress/router-default-66d97c9c97-qqxj5" Oct 30 23:41:17.634589 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:17.634323 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4cf4abe6-da2b-41b6-9caa-9659d4f750fb-metrics-certs\") pod \"router-default-66d97c9c97-qqxj5\" (UID: \"4cf4abe6-da2b-41b6-9caa-9659d4f750fb\") " pod="openshift-ingress/router-default-66d97c9c97-qqxj5" Oct 30 23:41:17.634589 ip-10-0-141-180 kubenswrapper[2550]: E1030 23:41:17.634370 2550 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 23:41:17.634589 ip-10-0-141-180 kubenswrapper[2550]: E1030 23:41:17.634383 2550 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-544b44f8d-5p6kz: secret "image-registry-tls" not found Oct 30 23:41:17.634589 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:17.634408 2550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-fbgth" event={"ID":"f464ef41-45c9-4ff4-b410-e2d9d6cdbdf5","Type":"ContainerStarted","Data":"4d24e3019382f392346762ad0cdd801460b4a56c586d5f30a283293c21f78d5c"} Oct 30 23:41:17.634589 ip-10-0-141-180 kubenswrapper[2550]: E1030 23:41:17.634424 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/58f7cf25-f5d0-4481-a927-4a745b692f7a-registry-tls podName:58f7cf25-f5d0-4481-a927-4a745b692f7a nodeName:}" failed. No retries permitted until 2026-10-30 23:41:19.634410407 +0000 UTC m=+35.692614315 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/58f7cf25-f5d0-4481-a927-4a745b692f7a-registry-tls") pod "image-registry-544b44f8d-5p6kz" (UID: "58f7cf25-f5d0-4481-a927-4a745b692f7a") : secret "image-registry-tls" not found Oct 30 23:41:17.634589 ip-10-0-141-180 kubenswrapper[2550]: E1030 23:41:17.634440 2550 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 23:41:17.634589 ip-10-0-141-180 kubenswrapper[2550]: E1030 23:41:17.634441 2550 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 23:41:17.634589 ip-10-0-141-180 kubenswrapper[2550]: E1030 23:41:17.634479 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4cf4abe6-da2b-41b6-9caa-9659d4f750fb-service-ca-bundle podName:4cf4abe6-da2b-41b6-9caa-9659d4f750fb nodeName:}" failed. No retries permitted until 2026-10-30 23:41:19.634465756 +0000 UTC m=+35.692669666 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/4cf4abe6-da2b-41b6-9caa-9659d4f750fb-service-ca-bundle") pod "router-default-66d97c9c97-qqxj5" (UID: "4cf4abe6-da2b-41b6-9caa-9659d4f750fb") : configmap references non-existent config key: service-ca.crt Oct 30 23:41:17.634589 ip-10-0-141-180 kubenswrapper[2550]: E1030 23:41:17.634521 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/03211c6a-9761-419c-839b-21ca413d5bb2-cluster-monitoring-operator-tls podName:03211c6a-9761-419c-839b-21ca413d5bb2 nodeName:}" failed. No retries permitted until 2026-10-30 23:41:19.634504626 +0000 UTC m=+35.692708519 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/03211c6a-9761-419c-839b-21ca413d5bb2-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-httg9" (UID: "03211c6a-9761-419c-839b-21ca413d5bb2") : secret "cluster-monitoring-operator-tls" not found Oct 30 23:41:17.634589 ip-10-0-141-180 kubenswrapper[2550]: E1030 23:41:17.634532 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4cf4abe6-da2b-41b6-9caa-9659d4f750fb-metrics-certs podName:4cf4abe6-da2b-41b6-9caa-9659d4f750fb nodeName:}" failed. No retries permitted until 2026-10-30 23:41:19.634526136 +0000 UTC m=+35.692730028 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4cf4abe6-da2b-41b6-9caa-9659d4f750fb-metrics-certs") pod "router-default-66d97c9c97-qqxj5" (UID: "4cf4abe6-da2b-41b6-9caa-9659d4f750fb") : secret "router-metrics-certs-default" not found Oct 30 23:41:17.635909 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:17.635889 2550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-lxdsf" event={"ID":"64ae3ff5-44eb-456b-948a-c1a8320e6b07","Type":"ContainerStarted","Data":"580cea62b06793d735676975b49e58105f9a1a127335587063edcb310fb44319"} Oct 30 23:41:17.637398 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:17.637380 2550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-f7zjp" event={"ID":"3af6ccf2-97f1-4598-b8ef-12eac377133b","Type":"ContainerStarted","Data":"b0d6f3e314e16ba43a80467eea0d611b93369e52a76d24774d2a0eeb08fb6ef2"} Oct 30 23:41:17.637486 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:17.637425 2550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-f7zjp" event={"ID":"3af6ccf2-97f1-4598-b8ef-12eac377133b","Type":"ContainerStarted","Data":"719c8f74b4146c489029a3cae27c9f4cc9d6c84b8ff50c6abfc15a23ef334941"} Oct 30 23:41:17.638592 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:17.638568 2550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-46zqw" event={"ID":"9700c98a-b858-45a5-9025-8d75f3c14044","Type":"ContainerStarted","Data":"ae3a0d475b2be1b25689edf2744177911624228a66bdbfc2ba94c3dccad12b36"} Oct 30 23:41:17.639631 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:17.639607 2550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-tcxmv" event={"ID":"f8eb4b4c-8225-436b-b1e8-7775414a1ca3","Type":"ContainerStarted","Data":"6b58028156e6606c849a36e8b17c9e190dc04bfc8a69a1f24a667e08b4a26210"} Oct 30 23:41:17.640552 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:17.640532 2550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-9qs84" event={"ID":"840aaae0-a215-4ccd-b420-d6f5723b65ae","Type":"ContainerStarted","Data":"01cae760460a3c29096c2785faece01841f55c4c9c9de869c85e2ef7134b502c"} Oct 30 23:41:17.735386 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:17.734979 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kgdzs" Oct 30 23:41:17.740845 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:17.740824 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7wrh7" Oct 30 23:41:18.077709 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:18.077579 2550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-f7zjp" podStartSLOduration=408.077560706 podStartE2EDuration="6m48.077560706s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:34:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:41:18.075768129 +0000 UTC m=+34.133972038" watchObservedRunningTime="2026-10-30 23:41:18.077560706 +0000 UTC m=+34.135764621" Oct 30 23:41:18.128446 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:18.128389 2550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-kgdzs"] Oct 30 23:41:18.144966 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:18.144936 2550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-7wrh7"] Oct 30 23:41:18.312558 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:18.312471 2550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-lxdsf" podStartSLOduration=408.312451729 podStartE2EDuration="6m48.312451729s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:34:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:41:18.312433967 +0000 UTC m=+34.370637883" watchObservedRunningTime="2026-10-30 23:41:18.312451729 +0000 UTC m=+34.370655645" Oct 30 23:41:18.312871 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:18.312835 2550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-l4d2m" podStartSLOduration=408.312823361 podStartE2EDuration="6m48.312823361s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:34:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:41:18.196306941 +0000 UTC m=+34.254510856" watchObservedRunningTime="2026-10-30 23:41:18.312823361 +0000 UTC m=+34.371027277" Oct 30 23:41:18.648530 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:18.648454 2550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-kgdzs" event={"ID":"7728abfe-1747-422e-b8e2-c91c9de83d09","Type":"ContainerStarted","Data":"a755ef97a7c53aaa8735a70265d8b3996ca96b08bf812706d173efd03f99c3bc"} Oct 30 23:41:18.651289 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:18.651258 2550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-7wrh7" event={"ID":"97da2086-b00d-4f03-b502-56de5ee3d90d","Type":"ContainerStarted","Data":"b4ecc76140e7d1ce05b22f67284d328e2dc8e4efa8a8ec078a459ef9338deee9"} Oct 30 23:41:19.345007 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:19.344943 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:41:19.658991 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:19.658957 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/58f7cf25-f5d0-4481-a927-4a745b692f7a-registry-tls\") pod \"image-registry-544b44f8d-5p6kz\" (UID: \"58f7cf25-f5d0-4481-a927-4a745b692f7a\") " pod="openshift-image-registry/image-registry-544b44f8d-5p6kz" Oct 30 23:41:19.659393 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:19.659015 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/03211c6a-9761-419c-839b-21ca413d5bb2-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-httg9\" (UID: \"03211c6a-9761-419c-839b-21ca413d5bb2\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-httg9" Oct 30 23:41:19.659393 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:19.659076 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4cf4abe6-da2b-41b6-9caa-9659d4f750fb-service-ca-bundle\") pod \"router-default-66d97c9c97-qqxj5\" (UID: \"4cf4abe6-da2b-41b6-9caa-9659d4f750fb\") " pod="openshift-ingress/router-default-66d97c9c97-qqxj5" Oct 30 23:41:19.659393 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:19.659103 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4cf4abe6-da2b-41b6-9caa-9659d4f750fb-metrics-certs\") pod \"router-default-66d97c9c97-qqxj5\" (UID: \"4cf4abe6-da2b-41b6-9caa-9659d4f750fb\") " pod="openshift-ingress/router-default-66d97c9c97-qqxj5" Oct 30 23:41:19.659393 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:19.659176 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/29b8bd1a-beeb-495c-9f61-e3326866a1d7-metrics-tls\") pod \"dns-default-9l5zq\" (UID: \"29b8bd1a-beeb-495c-9f61-e3326866a1d7\") " pod="openshift-dns/dns-default-9l5zq" Oct 30 23:41:19.659393 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:19.659208 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b53ad6ed-90fd-4a0f-bf54-88d416ba19e9-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-rxpst\" (UID: \"b53ad6ed-90fd-4a0f-bf54-88d416ba19e9\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-rxpst" Oct 30 23:41:19.659393 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:19.659248 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/be5540b5-7b10-45b2-b40c-723efd9f1b01-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-vl8t5\" (UID: \"be5540b5-7b10-45b2-b40c-723efd9f1b01\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vl8t5" Oct 30 23:41:19.659393 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:19.659307 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/17461b46-7ebf-49b7-afaf-12e20e3994a3-registry-tls\") pod \"image-registry-dd954f877-x9h9g\" (UID: \"17461b46-7ebf-49b7-afaf-12e20e3994a3\") " pod="openshift-image-registry/image-registry-dd954f877-x9h9g" Oct 30 23:41:19.659722 ip-10-0-141-180 kubenswrapper[2550]: E1030 23:41:19.659454 2550 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 23:41:19.659722 ip-10-0-141-180 kubenswrapper[2550]: E1030 23:41:19.659470 2550 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-dd954f877-x9h9g: secret "image-registry-tls" not found Oct 30 23:41:19.659722 ip-10-0-141-180 kubenswrapper[2550]: E1030 23:41:19.659503 2550 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 23:41:19.659722 ip-10-0-141-180 kubenswrapper[2550]: E1030 23:41:19.659513 2550 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-544b44f8d-5p6kz: secret "image-registry-tls" not found Oct 30 23:41:19.659722 ip-10-0-141-180 kubenswrapper[2550]: E1030 23:41:19.659524 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/17461b46-7ebf-49b7-afaf-12e20e3994a3-registry-tls podName:17461b46-7ebf-49b7-afaf-12e20e3994a3 nodeName:}" failed. No retries permitted until 2026-10-30 23:41:23.659506379 +0000 UTC m=+39.717710287 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/17461b46-7ebf-49b7-afaf-12e20e3994a3-registry-tls") pod "image-registry-dd954f877-x9h9g" (UID: "17461b46-7ebf-49b7-afaf-12e20e3994a3") : secret "image-registry-tls" not found Oct 30 23:41:19.659722 ip-10-0-141-180 kubenswrapper[2550]: E1030 23:41:19.659545 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/58f7cf25-f5d0-4481-a927-4a745b692f7a-registry-tls podName:58f7cf25-f5d0-4481-a927-4a745b692f7a nodeName:}" failed. No retries permitted until 2026-10-30 23:41:23.659534756 +0000 UTC m=+39.717738653 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/58f7cf25-f5d0-4481-a927-4a745b692f7a-registry-tls") pod "image-registry-544b44f8d-5p6kz" (UID: "58f7cf25-f5d0-4481-a927-4a745b692f7a") : secret "image-registry-tls" not found Oct 30 23:41:19.659722 ip-10-0-141-180 kubenswrapper[2550]: E1030 23:41:19.659591 2550 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 23:41:19.659722 ip-10-0-141-180 kubenswrapper[2550]: E1030 23:41:19.659598 2550 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 23:41:19.659722 ip-10-0-141-180 kubenswrapper[2550]: E1030 23:41:19.659622 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/29b8bd1a-beeb-495c-9f61-e3326866a1d7-metrics-tls podName:29b8bd1a-beeb-495c-9f61-e3326866a1d7 nodeName:}" failed. No retries permitted until 2026-10-30 23:41:23.659612656 +0000 UTC m=+39.717816570 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/29b8bd1a-beeb-495c-9f61-e3326866a1d7-metrics-tls") pod "dns-default-9l5zq" (UID: "29b8bd1a-beeb-495c-9f61-e3326866a1d7") : secret "dns-default-metrics-tls" not found Oct 30 23:41:19.659722 ip-10-0-141-180 kubenswrapper[2550]: E1030 23:41:19.659639 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/03211c6a-9761-419c-839b-21ca413d5bb2-cluster-monitoring-operator-tls podName:03211c6a-9761-419c-839b-21ca413d5bb2 nodeName:}" failed. No retries permitted until 2026-10-30 23:41:23.659628818 +0000 UTC m=+39.717832712 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/03211c6a-9761-419c-839b-21ca413d5bb2-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-httg9" (UID: "03211c6a-9761-419c-839b-21ca413d5bb2") : secret "cluster-monitoring-operator-tls" not found Oct 30 23:41:19.659722 ip-10-0-141-180 kubenswrapper[2550]: E1030 23:41:19.659677 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4cf4abe6-da2b-41b6-9caa-9659d4f750fb-service-ca-bundle podName:4cf4abe6-da2b-41b6-9caa-9659d4f750fb nodeName:}" failed. No retries permitted until 2026-10-30 23:41:23.659667247 +0000 UTC m=+39.717871143 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/4cf4abe6-da2b-41b6-9caa-9659d4f750fb-service-ca-bundle") pod "router-default-66d97c9c97-qqxj5" (UID: "4cf4abe6-da2b-41b6-9caa-9659d4f750fb") : configmap references non-existent config key: service-ca.crt Oct 30 23:41:19.659722 ip-10-0-141-180 kubenswrapper[2550]: E1030 23:41:19.659456 2550 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 23:41:19.659722 ip-10-0-141-180 kubenswrapper[2550]: E1030 23:41:19.659692 2550 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 23:41:19.659722 ip-10-0-141-180 kubenswrapper[2550]: E1030 23:41:19.659710 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4cf4abe6-da2b-41b6-9caa-9659d4f750fb-metrics-certs podName:4cf4abe6-da2b-41b6-9caa-9659d4f750fb nodeName:}" failed. No retries permitted until 2026-10-30 23:41:23.659701849 +0000 UTC m=+39.717905744 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4cf4abe6-da2b-41b6-9caa-9659d4f750fb-metrics-certs") pod "router-default-66d97c9c97-qqxj5" (UID: "4cf4abe6-da2b-41b6-9caa-9659d4f750fb") : secret "router-metrics-certs-default" not found Oct 30 23:41:19.659722 ip-10-0-141-180 kubenswrapper[2550]: E1030 23:41:19.659724 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b53ad6ed-90fd-4a0f-bf54-88d416ba19e9-networking-console-plugin-cert podName:b53ad6ed-90fd-4a0f-bf54-88d416ba19e9 nodeName:}" failed. No retries permitted until 2026-10-30 23:41:23.659716048 +0000 UTC m=+39.717919946 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/b53ad6ed-90fd-4a0f-bf54-88d416ba19e9-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-rxpst" (UID: "b53ad6ed-90fd-4a0f-bf54-88d416ba19e9") : secret "networking-console-plugin-cert" not found Oct 30 23:41:19.660501 ip-10-0-141-180 kubenswrapper[2550]: E1030 23:41:19.659769 2550 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 23:41:19.660501 ip-10-0-141-180 kubenswrapper[2550]: E1030 23:41:19.659805 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/be5540b5-7b10-45b2-b40c-723efd9f1b01-samples-operator-tls podName:be5540b5-7b10-45b2-b40c-723efd9f1b01 nodeName:}" failed. No retries permitted until 2026-10-30 23:41:23.659796022 +0000 UTC m=+39.717999929 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/be5540b5-7b10-45b2-b40c-723efd9f1b01-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-vl8t5" (UID: "be5540b5-7b10-45b2-b40c-723efd9f1b01") : secret "samples-operator-tls" not found Oct 30 23:41:19.685509 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:19.685469 2550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-tcxmv" event={"ID":"f8eb4b4c-8225-436b-b1e8-7775414a1ca3","Type":"ContainerStarted","Data":"76db8ef23480f0265d77551f882e6319eab45fd0215b132a6b1d19b389a38716"} Oct 30 23:41:20.964656 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:20.964623 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:41:21.112978 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:21.112938 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:41:21.707440 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:21.706520 2550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-pqt2c" event={"ID":"6382dfb5-a363-47a2-bc79-0b96f7bec0c3","Type":"ContainerStarted","Data":"ea562cbcce5a9165a3c31006a94e569e84dea078f1a5e7d419e5484eb05982e0"} Oct 30 23:41:21.707440 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:21.707390 2550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-pqt2c" Oct 30 23:41:21.709086 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:21.709051 2550 patch_prober.go:29] interesting pod/console-operator-7cd56b8dbc-pqt2c container/console-operator namespace/openshift-console-operator: Readiness probe status=failure output="Get \"https://10.132.0.9:8443/readyz\": dial tcp 10.132.0.9:8443: connect: connection refused" start-of-body= Oct 30 23:41:21.709211 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:21.709096 2550 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-console-operator/console-operator-7cd56b8dbc-pqt2c" podUID="6382dfb5-a363-47a2-bc79-0b96f7bec0c3" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.9:8443/readyz\": dial tcp 10.132.0.9:8443: connect: connection refused" Oct 30 23:41:21.832687 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:21.831860 2550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-tcxmv" podStartSLOduration=368.000117466 podStartE2EDuration="6m10.831841909s" totalImagesPullingTime="2.831724443s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:35:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:41:20.812780004 +0000 UTC m=+36.870983932" watchObservedRunningTime="2026-10-30 23:41:21.831841909 +0000 UTC m=+37.890045826" Oct 30 23:41:22.447117 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:22.446993 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:41:22.592889 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:22.592821 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:41:22.711708 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:22.711570 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-pqt2c_6382dfb5-a363-47a2-bc79-0b96f7bec0c3/console-operator/0.log" Oct 30 23:41:22.711708 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:22.711601 2550 generic.go:356] "Generic (PLEG): container finished" podID="6382dfb5-a363-47a2-bc79-0b96f7bec0c3" containerID="ea562cbcce5a9165a3c31006a94e569e84dea078f1a5e7d419e5484eb05982e0" exitCode=255 Oct 30 23:41:22.711708 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:22.711667 2550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-pqt2c" event={"ID":"6382dfb5-a363-47a2-bc79-0b96f7bec0c3","Type":"ContainerDied","Data":"ea562cbcce5a9165a3c31006a94e569e84dea078f1a5e7d419e5484eb05982e0"} Oct 30 23:41:22.711959 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:22.711942 2550 scope.go:122] "RemoveContainer" containerID="ea562cbcce5a9165a3c31006a94e569e84dea078f1a5e7d419e5484eb05982e0" Oct 30 23:41:22.718039 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:22.716555 2550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-9mqqt" event={"ID":"10774606-8e0c-460b-bd5f-4cb6d121b30b","Type":"ContainerStarted","Data":"a437bbfb58182b8229373016d264976a03b2f98ed1529a7e12eefc31a7248f16"} Oct 30 23:41:22.953973 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:22.953909 2550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-operator-7c4986d449-9mqqt" podStartSLOduration=366.508240522 podStartE2EDuration="6m11.953890236s" totalImagesPullingTime="5.445649714s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:35:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:41:22.952662879 +0000 UTC m=+39.010866795" watchObservedRunningTime="2026-10-30 23:41:22.953890236 +0000 UTC m=+39.012094157" Oct 30 23:41:23.615942 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:23.609831 2550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-sqv69"] Oct 30 23:41:23.636224 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:23.636190 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-sqv69" Oct 30 23:41:23.668523 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:23.668497 2550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-sqv69"] Oct 30 23:41:23.690121 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:23.690097 2550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 23:41:23.690271 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:23.690175 2550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 23:41:23.715346 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:23.715302 2550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-fvkr7\"" Oct 30 23:41:23.715687 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:23.715657 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/03211c6a-9761-419c-839b-21ca413d5bb2-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-httg9\" (UID: \"03211c6a-9761-419c-839b-21ca413d5bb2\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-httg9" Oct 30 23:41:23.715761 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:23.715718 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ec330d4f-2eea-476a-a7b8-60cd295fddd2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sqv69\" (UID: \"ec330d4f-2eea-476a-a7b8-60cd295fddd2\") " pod="openshift-insights/insights-runtime-extractor-sqv69" Oct 30 23:41:23.715799 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:23.715779 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4cf4abe6-da2b-41b6-9caa-9659d4f750fb-service-ca-bundle\") pod \"router-default-66d97c9c97-qqxj5\" (UID: \"4cf4abe6-da2b-41b6-9caa-9659d4f750fb\") " pod="openshift-ingress/router-default-66d97c9c97-qqxj5" Oct 30 23:41:23.715849 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:23.715810 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4cf4abe6-da2b-41b6-9caa-9659d4f750fb-metrics-certs\") pod \"router-default-66d97c9c97-qqxj5\" (UID: \"4cf4abe6-da2b-41b6-9caa-9659d4f750fb\") " pod="openshift-ingress/router-default-66d97c9c97-qqxj5" Oct 30 23:41:23.715849 ip-10-0-141-180 kubenswrapper[2550]: E1030 23:41:23.715828 2550 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 23:41:23.715954 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:23.715849 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4n58\" (UniqueName: \"kubernetes.io/projected/ec330d4f-2eea-476a-a7b8-60cd295fddd2-kube-api-access-z4n58\") pod \"insights-runtime-extractor-sqv69\" (UID: \"ec330d4f-2eea-476a-a7b8-60cd295fddd2\") " pod="openshift-insights/insights-runtime-extractor-sqv69" Oct 30 23:41:23.715954 ip-10-0-141-180 kubenswrapper[2550]: E1030 23:41:23.715897 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/03211c6a-9761-419c-839b-21ca413d5bb2-cluster-monitoring-operator-tls podName:03211c6a-9761-419c-839b-21ca413d5bb2 nodeName:}" failed. No retries permitted until 2026-10-30 23:41:31.715876112 +0000 UTC m=+47.774080008 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/03211c6a-9761-419c-839b-21ca413d5bb2-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-httg9" (UID: "03211c6a-9761-419c-839b-21ca413d5bb2") : secret "cluster-monitoring-operator-tls" not found Oct 30 23:41:23.715954 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:23.715943 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ec330d4f-2eea-476a-a7b8-60cd295fddd2-crio-socket\") pod \"insights-runtime-extractor-sqv69\" (UID: \"ec330d4f-2eea-476a-a7b8-60cd295fddd2\") " pod="openshift-insights/insights-runtime-extractor-sqv69" Oct 30 23:41:23.716084 ip-10-0-141-180 kubenswrapper[2550]: E1030 23:41:23.715989 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4cf4abe6-da2b-41b6-9caa-9659d4f750fb-service-ca-bundle podName:4cf4abe6-da2b-41b6-9caa-9659d4f750fb nodeName:}" failed. No retries permitted until 2026-10-30 23:41:31.715973234 +0000 UTC m=+47.774177129 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/4cf4abe6-da2b-41b6-9caa-9659d4f750fb-service-ca-bundle") pod "router-default-66d97c9c97-qqxj5" (UID: "4cf4abe6-da2b-41b6-9caa-9659d4f750fb") : configmap references non-existent config key: service-ca.crt Oct 30 23:41:23.716084 ip-10-0-141-180 kubenswrapper[2550]: E1030 23:41:23.716064 2550 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 23:41:23.716188 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:23.716158 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/29b8bd1a-beeb-495c-9f61-e3326866a1d7-metrics-tls\") pod \"dns-default-9l5zq\" (UID: \"29b8bd1a-beeb-495c-9f61-e3326866a1d7\") " pod="openshift-dns/dns-default-9l5zq" Oct 30 23:41:23.716239 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:23.716189 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ec330d4f-2eea-476a-a7b8-60cd295fddd2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sqv69\" (UID: \"ec330d4f-2eea-476a-a7b8-60cd295fddd2\") " pod="openshift-insights/insights-runtime-extractor-sqv69" Oct 30 23:41:23.716239 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:23.716222 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b53ad6ed-90fd-4a0f-bf54-88d416ba19e9-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-rxpst\" (UID: \"b53ad6ed-90fd-4a0f-bf54-88d416ba19e9\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-rxpst" Oct 30 23:41:23.716367 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:23.716244 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ec330d4f-2eea-476a-a7b8-60cd295fddd2-data-volume\") pod \"insights-runtime-extractor-sqv69\" (UID: \"ec330d4f-2eea-476a-a7b8-60cd295fddd2\") " pod="openshift-insights/insights-runtime-extractor-sqv69" Oct 30 23:41:23.716367 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:23.716274 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/be5540b5-7b10-45b2-b40c-723efd9f1b01-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-vl8t5\" (UID: \"be5540b5-7b10-45b2-b40c-723efd9f1b01\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vl8t5" Oct 30 23:41:23.716367 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:23.716300 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/17461b46-7ebf-49b7-afaf-12e20e3994a3-registry-tls\") pod \"image-registry-dd954f877-x9h9g\" (UID: \"17461b46-7ebf-49b7-afaf-12e20e3994a3\") " pod="openshift-image-registry/image-registry-dd954f877-x9h9g" Oct 30 23:41:23.716367 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:23.716326 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/58f7cf25-f5d0-4481-a927-4a745b692f7a-registry-tls\") pod \"image-registry-544b44f8d-5p6kz\" (UID: \"58f7cf25-f5d0-4481-a927-4a745b692f7a\") " pod="openshift-image-registry/image-registry-544b44f8d-5p6kz" Oct 30 23:41:23.716576 ip-10-0-141-180 kubenswrapper[2550]: E1030 23:41:23.716456 2550 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 23:41:23.716576 ip-10-0-141-180 kubenswrapper[2550]: E1030 23:41:23.716526 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/be5540b5-7b10-45b2-b40c-723efd9f1b01-samples-operator-tls podName:be5540b5-7b10-45b2-b40c-723efd9f1b01 nodeName:}" failed. No retries permitted until 2026-10-30 23:41:31.7165125 +0000 UTC m=+47.774716411 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/be5540b5-7b10-45b2-b40c-723efd9f1b01-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-vl8t5" (UID: "be5540b5-7b10-45b2-b40c-723efd9f1b01") : secret "samples-operator-tls" not found Oct 30 23:41:23.716762 ip-10-0-141-180 kubenswrapper[2550]: E1030 23:41:23.716573 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4cf4abe6-da2b-41b6-9caa-9659d4f750fb-metrics-certs podName:4cf4abe6-da2b-41b6-9caa-9659d4f750fb nodeName:}" failed. No retries permitted until 2026-10-30 23:41:31.716553152 +0000 UTC m=+47.774757062 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4cf4abe6-da2b-41b6-9caa-9659d4f750fb-metrics-certs") pod "router-default-66d97c9c97-qqxj5" (UID: "4cf4abe6-da2b-41b6-9caa-9659d4f750fb") : secret "router-metrics-certs-default" not found Oct 30 23:41:23.716762 ip-10-0-141-180 kubenswrapper[2550]: E1030 23:41:23.716639 2550 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 23:41:23.716762 ip-10-0-141-180 kubenswrapper[2550]: E1030 23:41:23.716680 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b53ad6ed-90fd-4a0f-bf54-88d416ba19e9-networking-console-plugin-cert podName:b53ad6ed-90fd-4a0f-bf54-88d416ba19e9 nodeName:}" failed. No retries permitted until 2026-10-30 23:41:31.716667528 +0000 UTC m=+47.774871445 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/b53ad6ed-90fd-4a0f-bf54-88d416ba19e9-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-rxpst" (UID: "b53ad6ed-90fd-4a0f-bf54-88d416ba19e9") : secret "networking-console-plugin-cert" not found Oct 30 23:41:23.716762 ip-10-0-141-180 kubenswrapper[2550]: E1030 23:41:23.716737 2550 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 23:41:23.716762 ip-10-0-141-180 kubenswrapper[2550]: E1030 23:41:23.716749 2550 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-544b44f8d-5p6kz: secret "image-registry-tls" not found Oct 30 23:41:23.717081 ip-10-0-141-180 kubenswrapper[2550]: E1030 23:41:23.716811 2550 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 23:41:23.717081 ip-10-0-141-180 kubenswrapper[2550]: E1030 23:41:23.716879 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/58f7cf25-f5d0-4481-a927-4a745b692f7a-registry-tls podName:58f7cf25-f5d0-4481-a927-4a745b692f7a nodeName:}" failed. No retries permitted until 2026-10-30 23:41:31.716864611 +0000 UTC m=+47.775068524 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/58f7cf25-f5d0-4481-a927-4a745b692f7a-registry-tls") pod "image-registry-544b44f8d-5p6kz" (UID: "58f7cf25-f5d0-4481-a927-4a745b692f7a") : secret "image-registry-tls" not found Oct 30 23:41:23.717081 ip-10-0-141-180 kubenswrapper[2550]: E1030 23:41:23.716916 2550 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 23:41:23.717081 ip-10-0-141-180 kubenswrapper[2550]: E1030 23:41:23.716928 2550 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-dd954f877-x9h9g: secret "image-registry-tls" not found Oct 30 23:41:23.717081 ip-10-0-141-180 kubenswrapper[2550]: E1030 23:41:23.716916 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/29b8bd1a-beeb-495c-9f61-e3326866a1d7-metrics-tls podName:29b8bd1a-beeb-495c-9f61-e3326866a1d7 nodeName:}" failed. No retries permitted until 2026-10-30 23:41:31.716907075 +0000 UTC m=+47.775111003 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/29b8bd1a-beeb-495c-9f61-e3326866a1d7-metrics-tls") pod "dns-default-9l5zq" (UID: "29b8bd1a-beeb-495c-9f61-e3326866a1d7") : secret "dns-default-metrics-tls" not found Oct 30 23:41:23.717081 ip-10-0-141-180 kubenswrapper[2550]: E1030 23:41:23.716958 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/17461b46-7ebf-49b7-afaf-12e20e3994a3-registry-tls podName:17461b46-7ebf-49b7-afaf-12e20e3994a3 nodeName:}" failed. No retries permitted until 2026-10-30 23:41:31.716949026 +0000 UTC m=+47.775152948 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/17461b46-7ebf-49b7-afaf-12e20e3994a3-registry-tls") pod "image-registry-dd954f877-x9h9g" (UID: "17461b46-7ebf-49b7-afaf-12e20e3994a3") : secret "image-registry-tls" not found Oct 30 23:41:23.724627 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:23.724610 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-pqt2c_6382dfb5-a363-47a2-bc79-0b96f7bec0c3/console-operator/1.log" Oct 30 23:41:23.725257 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:23.725232 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-pqt2c_6382dfb5-a363-47a2-bc79-0b96f7bec0c3/console-operator/0.log" Oct 30 23:41:23.725375 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:23.725268 2550 generic.go:356] "Generic (PLEG): container finished" podID="6382dfb5-a363-47a2-bc79-0b96f7bec0c3" containerID="e373ab1ed0edd47d38ff51aab784dee07bf119717f872d40609595f835e4357c" exitCode=255 Oct 30 23:41:23.725375 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:23.725366 2550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-pqt2c" event={"ID":"6382dfb5-a363-47a2-bc79-0b96f7bec0c3","Type":"ContainerDied","Data":"e373ab1ed0edd47d38ff51aab784dee07bf119717f872d40609595f835e4357c"} Oct 30 23:41:23.725505 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:23.725418 2550 scope.go:122] "RemoveContainer" containerID="ea562cbcce5a9165a3c31006a94e569e84dea078f1a5e7d419e5484eb05982e0" Oct 30 23:41:23.725791 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:23.725775 2550 scope.go:122] "RemoveContainer" containerID="e373ab1ed0edd47d38ff51aab784dee07bf119717f872d40609595f835e4357c" Oct 30 23:41:23.725960 ip-10-0-141-180 kubenswrapper[2550]: E1030 23:41:23.725942 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-pqt2c_openshift-console-operator(6382dfb5-a363-47a2-bc79-0b96f7bec0c3)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-pqt2c" podUID="6382dfb5-a363-47a2-bc79-0b96f7bec0c3" Oct 30 23:41:23.728691 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:23.728592 2550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-fbgth" event={"ID":"f464ef41-45c9-4ff4-b410-e2d9d6cdbdf5","Type":"ContainerStarted","Data":"704942de4f2ddb8ae14b522030de2121153ea50e4ac74156f93579566e481cd9"} Oct 30 23:41:23.730863 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:23.730817 2550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-9qs84" event={"ID":"840aaae0-a215-4ccd-b420-d6f5723b65ae","Type":"ContainerStarted","Data":"baeb8c991d9dac5ff1f990a149a34c9e24aae4f355b3d408d8031880243578c5"} Oct 30 23:41:23.817257 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:23.817223 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ec330d4f-2eea-476a-a7b8-60cd295fddd2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sqv69\" (UID: \"ec330d4f-2eea-476a-a7b8-60cd295fddd2\") " pod="openshift-insights/insights-runtime-extractor-sqv69" Oct 30 23:41:23.817428 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:23.817407 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4n58\" (UniqueName: \"kubernetes.io/projected/ec330d4f-2eea-476a-a7b8-60cd295fddd2-kube-api-access-z4n58\") pod \"insights-runtime-extractor-sqv69\" (UID: \"ec330d4f-2eea-476a-a7b8-60cd295fddd2\") " pod="openshift-insights/insights-runtime-extractor-sqv69" Oct 30 23:41:23.817475 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:23.817455 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ec330d4f-2eea-476a-a7b8-60cd295fddd2-crio-socket\") pod \"insights-runtime-extractor-sqv69\" (UID: \"ec330d4f-2eea-476a-a7b8-60cd295fddd2\") " pod="openshift-insights/insights-runtime-extractor-sqv69" Oct 30 23:41:23.817533 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:23.817517 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ec330d4f-2eea-476a-a7b8-60cd295fddd2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sqv69\" (UID: \"ec330d4f-2eea-476a-a7b8-60cd295fddd2\") " pod="openshift-insights/insights-runtime-extractor-sqv69" Oct 30 23:41:23.817585 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:23.817564 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ec330d4f-2eea-476a-a7b8-60cd295fddd2-data-volume\") pod \"insights-runtime-extractor-sqv69\" (UID: \"ec330d4f-2eea-476a-a7b8-60cd295fddd2\") " pod="openshift-insights/insights-runtime-extractor-sqv69" Oct 30 23:41:23.818303 ip-10-0-141-180 kubenswrapper[2550]: E1030 23:41:23.818279 2550 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 23:41:23.818430 ip-10-0-141-180 kubenswrapper[2550]: E1030 23:41:23.818353 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec330d4f-2eea-476a-a7b8-60cd295fddd2-insights-runtime-extractor-tls podName:ec330d4f-2eea-476a-a7b8-60cd295fddd2 nodeName:}" failed. No retries permitted until 2026-10-30 23:41:24.318322779 +0000 UTC m=+40.376526677 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ec330d4f-2eea-476a-a7b8-60cd295fddd2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-sqv69" (UID: "ec330d4f-2eea-476a-a7b8-60cd295fddd2") : secret "insights-runtime-extractor-tls" not found Oct 30 23:41:23.818430 ip-10-0-141-180 kubenswrapper[2550]: E1030 23:41:23.818419 2550 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 23:41:23.818545 ip-10-0-141-180 kubenswrapper[2550]: E1030 23:41:23.818450 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ec330d4f-2eea-476a-a7b8-60cd295fddd2-kube-rbac-proxy-cm podName:ec330d4f-2eea-476a-a7b8-60cd295fddd2 nodeName:}" failed. No retries permitted until 2026-10-30 23:41:24.318440405 +0000 UTC m=+40.376644298 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ec330d4f-2eea-476a-a7b8-60cd295fddd2-kube-rbac-proxy-cm") pod "insights-runtime-extractor-sqv69" (UID: "ec330d4f-2eea-476a-a7b8-60cd295fddd2") : configmap "kube-rbac-proxy" not found Oct 30 23:41:23.818820 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:23.818798 2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ec330d4f-2eea-476a-a7b8-60cd295fddd2-data-volume\") pod \"insights-runtime-extractor-sqv69\" (UID: \"ec330d4f-2eea-476a-a7b8-60cd295fddd2\") " pod="openshift-insights/insights-runtime-extractor-sqv69" Oct 30 23:41:23.819371 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:23.819323 2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ec330d4f-2eea-476a-a7b8-60cd295fddd2-crio-socket\") pod \"insights-runtime-extractor-sqv69\" (UID: \"ec330d4f-2eea-476a-a7b8-60cd295fddd2\") " pod="openshift-insights/insights-runtime-extractor-sqv69" Oct 30 23:41:23.847290 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:23.847265 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:41:23.880744 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:23.880672 2550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-r9crl-vqcgk"] Oct 30 23:41:23.889081 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:23.889045 2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4n58\" (UniqueName: \"kubernetes.io/projected/ec330d4f-2eea-476a-a7b8-60cd295fddd2-kube-api-access-z4n58\") pod \"insights-runtime-extractor-sqv69\" (UID: \"ec330d4f-2eea-476a-a7b8-60cd295fddd2\") " pod="openshift-insights/insights-runtime-extractor-sqv69" Oct 30 23:41:23.908326 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:23.908301 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-r9crl-vqcgk" Oct 30 23:41:23.918769 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:23.918731 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a78879b4-333e-4f1a-88bc-e96aee0e6d6b-trusted-ca-bundle\") pod \"periodic-gathering-r9crl-vqcgk\" (UID: \"a78879b4-333e-4f1a-88bc-e96aee0e6d6b\") " pod="openshift-insights/periodic-gathering-r9crl-vqcgk" Oct 30 23:41:23.918952 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:23.918792 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdj76\" (UniqueName: \"kubernetes.io/projected/a78879b4-333e-4f1a-88bc-e96aee0e6d6b-kube-api-access-sdj76\") pod \"periodic-gathering-r9crl-vqcgk\" (UID: \"a78879b4-333e-4f1a-88bc-e96aee0e6d6b\") " pod="openshift-insights/periodic-gathering-r9crl-vqcgk" Oct 30 23:41:23.918952 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:23.918880 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a78879b4-333e-4f1a-88bc-e96aee0e6d6b-service-ca-bundle\") pod \"periodic-gathering-r9crl-vqcgk\" (UID: \"a78879b4-333e-4f1a-88bc-e96aee0e6d6b\") " pod="openshift-insights/periodic-gathering-r9crl-vqcgk" Oct 30 23:41:23.918952 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:23.918912 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/a78879b4-333e-4f1a-88bc-e96aee0e6d6b-archives-path\") pod \"periodic-gathering-r9crl-vqcgk\" (UID: \"a78879b4-333e-4f1a-88bc-e96aee0e6d6b\") " pod="openshift-insights/periodic-gathering-r9crl-vqcgk" Oct 30 23:41:23.938349 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:23.938310 2550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-r9crl-vqcgk"] Oct 30 23:41:23.995892 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:23.995828 2550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-9qs84" podStartSLOduration=366.967434771 podStartE2EDuration="6m12.995808533s" totalImagesPullingTime="6.028373762s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:35:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:41:23.995435361 +0000 UTC m=+40.053639276" watchObservedRunningTime="2026-10-30 23:41:23.995808533 +0000 UTC m=+40.054012452" Oct 30 23:41:24.005760 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:24.005697 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:41:24.019980 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:24.019944 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a78879b4-333e-4f1a-88bc-e96aee0e6d6b-trusted-ca-bundle\") pod \"periodic-gathering-r9crl-vqcgk\" (UID: \"a78879b4-333e-4f1a-88bc-e96aee0e6d6b\") " pod="openshift-insights/periodic-gathering-r9crl-vqcgk" Oct 30 23:41:24.020133 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:24.019993 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdj76\" (UniqueName: \"kubernetes.io/projected/a78879b4-333e-4f1a-88bc-e96aee0e6d6b-kube-api-access-sdj76\") pod \"periodic-gathering-r9crl-vqcgk\" (UID: \"a78879b4-333e-4f1a-88bc-e96aee0e6d6b\") " pod="openshift-insights/periodic-gathering-r9crl-vqcgk" Oct 30 23:41:24.020133 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:24.020079 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a78879b4-333e-4f1a-88bc-e96aee0e6d6b-service-ca-bundle\") pod \"periodic-gathering-r9crl-vqcgk\" (UID: \"a78879b4-333e-4f1a-88bc-e96aee0e6d6b\") " pod="openshift-insights/periodic-gathering-r9crl-vqcgk" Oct 30 23:41:24.020133 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:24.020108 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/a78879b4-333e-4f1a-88bc-e96aee0e6d6b-archives-path\") pod \"periodic-gathering-r9crl-vqcgk\" (UID: \"a78879b4-333e-4f1a-88bc-e96aee0e6d6b\") " pod="openshift-insights/periodic-gathering-r9crl-vqcgk" Oct 30 23:41:24.020590 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:24.020569 2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/a78879b4-333e-4f1a-88bc-e96aee0e6d6b-archives-path\") pod \"periodic-gathering-r9crl-vqcgk\" (UID: \"a78879b4-333e-4f1a-88bc-e96aee0e6d6b\") " pod="openshift-insights/periodic-gathering-r9crl-vqcgk" Oct 30 23:41:24.021500 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:24.021470 2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a78879b4-333e-4f1a-88bc-e96aee0e6d6b-trusted-ca-bundle\") pod \"periodic-gathering-r9crl-vqcgk\" (UID: \"a78879b4-333e-4f1a-88bc-e96aee0e6d6b\") " pod="openshift-insights/periodic-gathering-r9crl-vqcgk" Oct 30 23:41:24.022217 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:24.022170 2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a78879b4-333e-4f1a-88bc-e96aee0e6d6b-service-ca-bundle\") pod \"periodic-gathering-r9crl-vqcgk\" (UID: \"a78879b4-333e-4f1a-88bc-e96aee0e6d6b\") " pod="openshift-insights/periodic-gathering-r9crl-vqcgk" Oct 30 23:41:24.083695 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:24.083664 2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdj76\" (UniqueName: \"kubernetes.io/projected/a78879b4-333e-4f1a-88bc-e96aee0e6d6b-kube-api-access-sdj76\") pod \"periodic-gathering-r9crl-vqcgk\" (UID: \"a78879b4-333e-4f1a-88bc-e96aee0e6d6b\") " pod="openshift-insights/periodic-gathering-r9crl-vqcgk" Oct 30 23:41:24.116699 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:24.116649 2550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-fbgth" podStartSLOduration=366.49397116 podStartE2EDuration="6m13.116631821s" totalImagesPullingTime="6.622660661s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:35:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:41:24.116056796 +0000 UTC m=+40.174260713" watchObservedRunningTime="2026-10-30 23:41:24.116631821 +0000 UTC m=+40.174835737" Oct 30 23:41:24.235295 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:24.235267 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-r9crl-vqcgk" Oct 30 23:41:24.322870 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:24.322838 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ec330d4f-2eea-476a-a7b8-60cd295fddd2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sqv69\" (UID: \"ec330d4f-2eea-476a-a7b8-60cd295fddd2\") " pod="openshift-insights/insights-runtime-extractor-sqv69" Oct 30 23:41:24.323253 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:24.323000 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ec330d4f-2eea-476a-a7b8-60cd295fddd2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sqv69\" (UID: \"ec330d4f-2eea-476a-a7b8-60cd295fddd2\") " pod="openshift-insights/insights-runtime-extractor-sqv69" Oct 30 23:41:24.323253 ip-10-0-141-180 kubenswrapper[2550]: E1030 23:41:24.323089 2550 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 23:41:24.323253 ip-10-0-141-180 kubenswrapper[2550]: E1030 23:41:24.323153 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec330d4f-2eea-476a-a7b8-60cd295fddd2-insights-runtime-extractor-tls podName:ec330d4f-2eea-476a-a7b8-60cd295fddd2 nodeName:}" failed. No retries permitted until 2026-10-30 23:41:25.323133875 +0000 UTC m=+41.381337771 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ec330d4f-2eea-476a-a7b8-60cd295fddd2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-sqv69" (UID: "ec330d4f-2eea-476a-a7b8-60cd295fddd2") : secret "insights-runtime-extractor-tls" not found Oct 30 23:41:24.323253 ip-10-0-141-180 kubenswrapper[2550]: E1030 23:41:24.323170 2550 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 23:41:24.323253 ip-10-0-141-180 kubenswrapper[2550]: E1030 23:41:24.323225 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ec330d4f-2eea-476a-a7b8-60cd295fddd2-kube-rbac-proxy-cm podName:ec330d4f-2eea-476a-a7b8-60cd295fddd2 nodeName:}" failed. No retries permitted until 2026-10-30 23:41:25.323210049 +0000 UTC m=+41.381413956 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ec330d4f-2eea-476a-a7b8-60cd295fddd2-kube-rbac-proxy-cm") pod "insights-runtime-extractor-sqv69" (UID: "ec330d4f-2eea-476a-a7b8-60cd295fddd2") : configmap "kube-rbac-proxy" not found Oct 30 23:41:24.526380 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:24.526262 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:41:24.639016 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:24.638983 2550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-r9crl-vqcgk"] Oct 30 23:41:24.645603 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:41:24.645569 2550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda78879b4_333e_4f1a_88bc_e96aee0e6d6b.slice/crio-bbfd66c255357d0b1ae9f531564af260a8d8bd00c26ccfe7f763b2cdb3afeaec WatchSource:0}: Error finding container bbfd66c255357d0b1ae9f531564af260a8d8bd00c26ccfe7f763b2cdb3afeaec: Status 404 returned error can't find the container with id bbfd66c255357d0b1ae9f531564af260a8d8bd00c26ccfe7f763b2cdb3afeaec Oct 30 23:41:24.737025 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:24.736986 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-pqt2c_6382dfb5-a363-47a2-bc79-0b96f7bec0c3/console-operator/1.log" Oct 30 23:41:24.737471 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:24.737452 2550 scope.go:122] "RemoveContainer" containerID="e373ab1ed0edd47d38ff51aab784dee07bf119717f872d40609595f835e4357c" Oct 30 23:41:24.737668 ip-10-0-141-180 kubenswrapper[2550]: E1030 23:41:24.737646 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-pqt2c_openshift-console-operator(6382dfb5-a363-47a2-bc79-0b96f7bec0c3)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-pqt2c" podUID="6382dfb5-a363-47a2-bc79-0b96f7bec0c3" Oct 30 23:41:24.741095 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:24.741057 2550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-xrk8k" event={"ID":"27ccca21-5e99-485f-8b5c-ca8ec5177668","Type":"ContainerStarted","Data":"28284247b72c104b90d5965aad13c78b65cdc0b154c7ec047a7418d9588cf1bb"} Oct 30 23:41:24.743398 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:24.743365 2550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-46zqw" event={"ID":"9700c98a-b858-45a5-9025-8d75f3c14044","Type":"ContainerStarted","Data":"9dc28be039ac321fa7ebadd5f25da2c5d44b9f0032a6475553ecece200e142dd"} Oct 30 23:41:24.743511 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:24.743424 2550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-8khpq"] Oct 30 23:41:24.765590 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:24.765514 2550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-r9crl-vqcgk" event={"ID":"a78879b4-333e-4f1a-88bc-e96aee0e6d6b","Type":"ContainerStarted","Data":"bbfd66c255357d0b1ae9f531564af260a8d8bd00c26ccfe7f763b2cdb3afeaec"} Oct 30 23:41:24.765590 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:24.765577 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-8khpq" Oct 30 23:41:24.784111 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:24.784013 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:41:24.797873 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:24.797843 2550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-8khpq"] Oct 30 23:41:24.820935 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:24.820907 2550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-28w6g\"" Oct 30 23:41:24.829109 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:24.829035 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q9zsm\" (UniqueName: \"kubernetes.io/projected/aa90814d-5b4d-4039-917d-76610b1f88e4-kube-api-access-q9zsm\") pod \"migrator-66df5d496b-8khpq\" (UID: \"aa90814d-5b4d-4039-917d-76610b1f88e4\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-8khpq" Oct 30 23:41:24.887802 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:24.887677 2550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Oct 30 23:41:24.887802 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:24.887756 2550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Oct 30 23:41:24.930463 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:24.930429 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9zsm\" (UniqueName: \"kubernetes.io/projected/aa90814d-5b4d-4039-917d-76610b1f88e4-kube-api-access-q9zsm\") pod \"migrator-66df5d496b-8khpq\" (UID: \"aa90814d-5b4d-4039-917d-76610b1f88e4\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-8khpq" Oct 30 23:41:24.933778 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:24.933749 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:41:24.978033 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:24.977982 2550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-46zqw" podStartSLOduration=2.9399128660000002 podStartE2EDuration="9.977965803s" totalImagesPullingTime="7.038052937s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:41:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:41:24.977089306 +0000 UTC m=+41.035293223" watchObservedRunningTime="2026-10-30 23:41:24.977965803 +0000 UTC m=+41.036169719" Oct 30 23:41:24.995854 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:24.995822 2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9zsm\" (UniqueName: \"kubernetes.io/projected/aa90814d-5b4d-4039-917d-76610b1f88e4-kube-api-access-q9zsm\") pod \"migrator-66df5d496b-8khpq\" (UID: \"aa90814d-5b4d-4039-917d-76610b1f88e4\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-8khpq" Oct 30 23:41:25.085899 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:25.085828 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:41:25.105317 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:25.105291 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-8khpq" Oct 30 23:41:25.334569 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:25.334524 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ec330d4f-2eea-476a-a7b8-60cd295fddd2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sqv69\" (UID: \"ec330d4f-2eea-476a-a7b8-60cd295fddd2\") " pod="openshift-insights/insights-runtime-extractor-sqv69" Oct 30 23:41:25.334756 ip-10-0-141-180 kubenswrapper[2550]: E1030 23:41:25.334665 2550 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 23:41:25.334756 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:25.334670 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ec330d4f-2eea-476a-a7b8-60cd295fddd2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sqv69\" (UID: \"ec330d4f-2eea-476a-a7b8-60cd295fddd2\") " pod="openshift-insights/insights-runtime-extractor-sqv69" Oct 30 23:41:25.334756 ip-10-0-141-180 kubenswrapper[2550]: E1030 23:41:25.334744 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ec330d4f-2eea-476a-a7b8-60cd295fddd2-kube-rbac-proxy-cm podName:ec330d4f-2eea-476a-a7b8-60cd295fddd2 nodeName:}" failed. No retries permitted until 2026-10-30 23:41:27.33472287 +0000 UTC m=+43.392926783 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ec330d4f-2eea-476a-a7b8-60cd295fddd2-kube-rbac-proxy-cm") pod "insights-runtime-extractor-sqv69" (UID: "ec330d4f-2eea-476a-a7b8-60cd295fddd2") : configmap "kube-rbac-proxy" not found Oct 30 23:41:25.334941 ip-10-0-141-180 kubenswrapper[2550]: E1030 23:41:25.334759 2550 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 23:41:25.334941 ip-10-0-141-180 kubenswrapper[2550]: E1030 23:41:25.334812 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec330d4f-2eea-476a-a7b8-60cd295fddd2-insights-runtime-extractor-tls podName:ec330d4f-2eea-476a-a7b8-60cd295fddd2 nodeName:}" failed. No retries permitted until 2026-10-30 23:41:27.334796748 +0000 UTC m=+43.393000661 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ec330d4f-2eea-476a-a7b8-60cd295fddd2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-sqv69" (UID: "ec330d4f-2eea-476a-a7b8-60cd295fddd2") : secret "insights-runtime-extractor-tls" not found Oct 30 23:41:25.511165 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:25.511113 2550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-xrk8k" podStartSLOduration=366.978792267 podStartE2EDuration="6m14.511098787s" totalImagesPullingTime="7.53230652s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:35:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:41:25.212508475 +0000 UTC m=+41.270712392" watchObservedRunningTime="2026-10-30 23:41:25.511098787 +0000 UTC m=+41.569302702" Oct 30 23:41:25.511364 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:25.511250 2550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-8khpq"] Oct 30 23:41:25.515032 ip-10-0-141-180 kubenswrapper[2550]: W1030 23:41:25.515003 2550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa90814d_5b4d_4039_917d_76610b1f88e4.slice/crio-fe922c3105879e4776509bda85f73cd3fc9e6c078679a53ab53a6d6ff57f44b2 WatchSource:0}: Error finding container fe922c3105879e4776509bda85f73cd3fc9e6c078679a53ab53a6d6ff57f44b2: Status 404 returned error can't find the container with id fe922c3105879e4776509bda85f73cd3fc9e6c078679a53ab53a6d6ff57f44b2 Oct 30 23:41:25.751858 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:25.751815 2550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-kgdzs" event={"ID":"7728abfe-1747-422e-b8e2-c91c9de83d09","Type":"ContainerStarted","Data":"2a2719d608d3769e103aeecf866b6b5b680862e94c6238cce09b14783d69dc54"} Oct 30 23:41:25.752391 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:25.751937 2550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-kgdzs" Oct 30 23:41:25.753770 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:25.753741 2550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-r9crl-vqcgk" event={"ID":"a78879b4-333e-4f1a-88bc-e96aee0e6d6b","Type":"ContainerStarted","Data":"97361002a0136337989d4473e0d7b49dec6c5cdf9e9d6f07e7292fb2e86c3fd1"} Oct 30 23:41:25.755016 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:25.754989 2550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-8khpq" event={"ID":"aa90814d-5b4d-4039-917d-76610b1f88e4","Type":"ContainerStarted","Data":"fe922c3105879e4776509bda85f73cd3fc9e6c078679a53ab53a6d6ff57f44b2"} Oct 30 23:41:25.871175 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:25.871063 2550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-kgdzs" podStartSLOduration=33.925406148 podStartE2EDuration="40.871042676s" totalImagesPullingTime="6.945636528s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:40:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:41:25.869109947 +0000 UTC m=+41.927313862" watchObservedRunningTime="2026-10-30 23:41:25.871042676 +0000 UTC m=+41.929246592" Oct 30 23:41:25.993738 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:25.993670 2550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-r9crl-vqcgk" podStartSLOduration=2.993652308 podStartE2EDuration="2.993652308s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:41:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:41:25.992835774 +0000 UTC m=+42.051039690" watchObservedRunningTime="2026-10-30 23:41:25.993652308 +0000 UTC m=+42.051856225" Oct 30 23:41:26.205945 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:26.205911 2550 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-pqt2c" Oct 30 23:41:26.206355 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:26.206312 2550 scope.go:122] "RemoveContainer" containerID="e373ab1ed0edd47d38ff51aab784dee07bf119717f872d40609595f835e4357c" Oct 30 23:41:26.206561 ip-10-0-141-180 kubenswrapper[2550]: E1030 23:41:26.206539 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-pqt2c_openshift-console-operator(6382dfb5-a363-47a2-bc79-0b96f7bec0c3)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-pqt2c" podUID="6382dfb5-a363-47a2-bc79-0b96f7bec0c3" Oct 30 23:41:26.283880 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:26.283834 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:41:26.484009 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:26.483939 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:41:26.760834 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:26.760750 2550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-7wrh7" event={"ID":"97da2086-b00d-4f03-b502-56de5ee3d90d","Type":"ContainerStarted","Data":"b56251c4ca39c6cc5b0b4d091f77f7c26d746bfe437a6b92d319677451f6a035"} Oct 30 23:41:26.880886 ip-10-0-141-180 kubenswrapper[2550]: I1030 23:41:26.880825 2550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-7wrh7" podStartSLOduration=33.453936622 podStartE2EDuration="41.880805485s" totalImagesPullingTime="8.426868863s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 23:40:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:41:26.879936872 +0000 UTC m=+42.938140787" watchObservedRunningTime="2026-10-30 23:41:26.880805485 +0000 UTC m=+42.939009407" Oct 30 23:41:26.882413 ip-10-0-141-180 systemd[1]: Stopping Kubernetes Kubelet... Oct 30 23:41:26.895722 ip-10-0-141-180 systemd[1]: kubelet.service: Deactivated successfully. Oct 30 23:41:26.895939 ip-10-0-141-180 systemd[1]: Stopped Kubernetes Kubelet. Oct 30 23:41:26.895997 ip-10-0-141-180 systemd[1]: kubelet.service: Consumed 3.106s CPU time. Oct 30 23:41:26.905038 ip-10-0-141-180 systemd[1]: Starting Kubernetes Kubelet... Oct 30 23:41:26.993827 ip-10-0-141-180 kubenswrapper[4927]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 23:41:26.993827 ip-10-0-141-180 kubenswrapper[4927]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 23:41:26.993827 ip-10-0-141-180 kubenswrapper[4927]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 23:41:26.993827 ip-10-0-141-180 kubenswrapper[4927]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 23:41:26.995929 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.995855 4927 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 23:41:26.995929 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.995929 4927 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 23:41:26.996000 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.995933 4927 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 23:41:26.996000 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.995937 4927 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 23:41:26.996000 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.995941 4927 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 23:41:26.996000 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.995944 4927 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 23:41:26.996000 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.995947 4927 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 23:41:26.996000 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.995950 4927 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 23:41:26.996000 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.995952 4927 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 23:41:26.996000 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.995956 4927 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 23:41:26.996000 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.995959 4927 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 23:41:26.996000 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.995971 4927 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 23:41:26.996000 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.995974 4927 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 23:41:26.996000 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.995976 4927 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 23:41:26.996000 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.995980 4927 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 23:41:26.996000 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.995983 4927 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 23:41:26.996000 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.995985 4927 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 23:41:26.996000 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.995988 4927 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 23:41:26.996000 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.995991 4927 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 23:41:26.996000 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.995993 4927 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 23:41:26.996000 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.995996 4927 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 23:41:26.996000 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.995999 4927 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 23:41:26.996513 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.996002 4927 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 23:41:26.996513 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.996005 4927 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 23:41:26.996513 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.996008 4927 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 23:41:26.996513 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.996011 4927 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 23:41:26.996513 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.996014 4927 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 23:41:26.996513 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.996017 4927 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 23:41:26.996513 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.996020 4927 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 23:41:26.996513 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.996023 4927 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 23:41:26.996513 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.996026 4927 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 23:41:26.996513 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.996028 4927 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 23:41:26.996513 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.996031 4927 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 23:41:26.996513 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.996034 4927 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 23:41:26.996513 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.996036 4927 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 23:41:26.996513 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.996039 4927 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 23:41:26.996513 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.996042 4927 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 23:41:26.996513 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.996044 4927 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 23:41:26.996513 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.996047 4927 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 23:41:26.996513 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.996050 4927 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 23:41:26.996513 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.996052 4927 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 23:41:26.996982 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.996055 4927 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 23:41:26.996982 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.996057 4927 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 23:41:26.996982 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.996061 4927 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 23:41:26.996982 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.996063 4927 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 23:41:26.996982 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.996066 4927 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 23:41:26.996982 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.996069 4927 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 23:41:26.996982 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.996071 4927 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 23:41:26.996982 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.996074 4927 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 23:41:26.996982 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.996076 4927 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 23:41:26.996982 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.996079 4927 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 23:41:26.996982 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.996082 4927 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 23:41:26.996982 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.996088 4927 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 23:41:26.996982 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.996090 4927 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 23:41:26.996982 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.996093 4927 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 23:41:26.996982 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.996096 4927 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 23:41:26.996982 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.996099 4927 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 23:41:26.996982 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.996102 4927 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 23:41:26.996982 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.996104 4927 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 23:41:26.996982 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.996107 4927 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 23:41:26.996982 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.996110 4927 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 23:41:26.997505 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.996112 4927 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 23:41:26.997505 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.996115 4927 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 23:41:26.997505 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.996118 4927 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 23:41:26.997505 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.996120 4927 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 23:41:26.997505 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.996123 4927 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 23:41:26.997505 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.996126 4927 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 23:41:26.997505 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.996128 4927 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 23:41:26.997505 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.996131 4927 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 23:41:26.997505 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.996134 4927 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 23:41:26.997505 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.996137 4927 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 23:41:26.997505 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.996139 4927 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 23:41:26.997505 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.996142 4927 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 23:41:26.997505 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.996145 4927 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 23:41:26.997505 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.996147 4927 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 23:41:26.997505 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.996150 4927 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 23:41:26.997505 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.996153 4927 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 23:41:26.997505 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.996156 4927 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 23:41:26.997505 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.996159 4927 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 23:41:26.997505 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.996162 4927 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 23:41:26.997967 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.996165 4927 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 23:41:26.997967 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.996167 4927 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 23:41:26.997967 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.996170 4927 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 23:41:26.997967 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.996173 4927 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 23:41:26.997967 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.996178 4927 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 23:41:26.997967 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.996187 4927 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 23:41:26.997967 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.996190 4927 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 23:41:26.997967 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.996193 4927 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 23:41:26.997967 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.996196 4927 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 23:41:26.997967 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.996199 4927 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 23:41:26.997967 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.996201 4927 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 23:41:26.997967 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.996204 4927 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 23:41:26.997967 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.996207 4927 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 23:41:26.997967 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.996210 4927 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 23:41:26.997967 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.996212 4927 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 23:41:26.997967 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.996215 4927 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 23:41:26.997967 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.996218 4927 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 23:41:26.997967 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.996222 4927 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 23:41:26.997967 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.996227 4927 feature_gate.go:386] unrecognized feature gate: Example Oct 30 23:41:26.998502 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.996231 4927 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 23:41:26.998502 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.996235 4927 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 23:41:26.998502 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.996238 4927 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 23:41:26.998502 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.996241 4927 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 23:41:26.998502 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.996244 4927 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 23:41:26.998502 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.996247 4927 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 23:41:26.998502 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.996250 4927 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 23:41:26.998502 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.996982 4927 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 23:41:26.998502 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.996989 4927 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 23:41:26.998502 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.996992 4927 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 23:41:26.998502 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.996997 4927 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 23:41:26.998502 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.997000 4927 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 23:41:26.998502 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.997003 4927 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 23:41:26.998502 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.997006 4927 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 23:41:26.998502 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.997009 4927 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 23:41:26.998502 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.997012 4927 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 23:41:26.998502 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.997015 4927 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 23:41:26.998502 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.997019 4927 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 23:41:26.998502 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.997022 4927 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 23:41:26.998502 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.997024 4927 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 23:41:26.999268 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.997027 4927 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 23:41:26.999268 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.997030 4927 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 23:41:26.999268 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.997033 4927 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 23:41:26.999268 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.997037 4927 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 23:41:26.999268 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.997040 4927 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 23:41:26.999268 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.997044 4927 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 23:41:26.999268 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.997047 4927 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 23:41:26.999268 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.997049 4927 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 23:41:26.999268 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.997052 4927 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 23:41:26.999268 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.997055 4927 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 23:41:26.999268 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.997058 4927 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 23:41:26.999268 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.997061 4927 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 23:41:26.999268 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.997063 4927 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 23:41:26.999268 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.997067 4927 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 23:41:26.999268 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.997077 4927 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 23:41:26.999268 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.997081 4927 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 23:41:26.999268 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.997084 4927 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 23:41:26.999268 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.997087 4927 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 23:41:26.999268 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.997089 4927 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 23:41:26.999268 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.997092 4927 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 23:41:26.999961 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.997095 4927 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 23:41:26.999961 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.997098 4927 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 23:41:26.999961 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.997101 4927 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 23:41:26.999961 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.997104 4927 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 23:41:26.999961 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.997107 4927 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 23:41:26.999961 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.997110 4927 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 23:41:26.999961 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.997113 4927 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 23:41:26.999961 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.997116 4927 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 23:41:26.999961 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.997119 4927 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 23:41:26.999961 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.997121 4927 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 23:41:26.999961 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.997124 4927 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 23:41:26.999961 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.997127 4927 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 23:41:26.999961 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.997130 4927 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 23:41:26.999961 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.997132 4927 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 23:41:26.999961 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.997135 4927 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 23:41:26.999961 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.997137 4927 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 23:41:26.999961 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.997140 4927 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 23:41:26.999961 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.997143 4927 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 23:41:26.999961 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.997145 4927 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 23:41:27.000538 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.997148 4927 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 23:41:27.000538 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.997151 4927 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 23:41:27.000538 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.997153 4927 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 23:41:27.000538 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.997156 4927 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 23:41:27.000538 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.997159 4927 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 23:41:27.000538 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.997161 4927 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 23:41:27.000538 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.997164 4927 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 23:41:27.000538 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.997167 4927 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 23:41:27.000538 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.997169 4927 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 23:41:27.000538 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.997172 4927 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 23:41:27.000538 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.997175 4927 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 23:41:27.000538 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.997178 4927 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 23:41:27.000538 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.997180 4927 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 23:41:27.000538 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.997183 4927 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 23:41:27.000538 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.997185 4927 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 23:41:27.000538 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.997188 4927 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 23:41:27.000538 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.997191 4927 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 23:41:27.000538 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.997194 4927 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 23:41:27.000538 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.997197 4927 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 23:41:27.001007 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.997200 4927 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 23:41:27.001007 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.997203 4927 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 23:41:27.001007 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.997206 4927 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 23:41:27.001007 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.997208 4927 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 23:41:27.001007 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.997211 4927 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 23:41:27.001007 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.997213 4927 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 23:41:27.001007 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.997216 4927 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 23:41:27.001007 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.997220 4927 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 23:41:27.001007 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.997223 4927 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 23:41:27.001007 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.997225 4927 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 23:41:27.001007 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.997228 4927 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 23:41:27.001007 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.997230 4927 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 23:41:27.001007 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.997233 4927 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 23:41:27.001007 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.997236 4927 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 23:41:27.001007 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.997238 4927 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 23:41:27.001007 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.997241 4927 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 23:41:27.001007 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.997243 4927 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 23:41:27.001007 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.997246 4927 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 23:41:27.001007 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.997248 4927 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 23:41:27.001504 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.997251 4927 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 23:41:27.001504 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.997254 4927 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 23:41:27.001504 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.997259 4927 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 23:41:27.001504 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.997262 4927 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 23:41:27.001504 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.997265 4927 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 23:41:27.001504 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.997268 4927 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 23:41:27.001504 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.997271 4927 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 23:41:27.001504 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.997274 4927 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 23:41:27.001504 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.997276 4927 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 23:41:27.001504 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.997279 4927 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 23:41:27.001504 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.997281 4927 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 23:41:27.001504 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.997284 4927 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 23:41:27.001504 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.997288 4927 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 23:41:27.001504 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.997290 4927 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 23:41:27.001504 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.997294 4927 feature_gate.go:386] unrecognized feature gate: Example Oct 30 23:41:27.001504 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.997297 4927 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 23:41:27.001504 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.997792 4927 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 23:41:27.001504 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.997798 4927 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 23:41:27.001504 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.997801 4927 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 23:41:27.001993 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.997803 4927 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 23:41:27.001993 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.997806 4927 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 23:41:27.001993 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.997809 4927 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 23:41:27.001993 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.997812 4927 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 23:41:27.001993 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.997814 4927 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 23:41:27.001993 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.997817 4927 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 23:41:27.001993 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.997820 4927 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 23:41:27.001993 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.997823 4927 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 23:41:27.001993 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.997825 4927 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 23:41:27.001993 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.997828 4927 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 23:41:27.001993 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.997831 4927 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 23:41:27.001993 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.997833 4927 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 23:41:27.001993 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.997836 4927 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 23:41:27.001993 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.997839 4927 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 23:41:27.001993 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.997845 4927 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 23:41:27.001993 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.997847 4927 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 23:41:27.001993 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.997852 4927 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 23:41:27.001993 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.997856 4927 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 23:41:27.001993 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.997859 4927 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 23:41:27.002487 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.997862 4927 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 23:41:27.002487 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.997865 4927 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 23:41:27.002487 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.997868 4927 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 23:41:27.002487 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.997870 4927 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 23:41:27.002487 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.997873 4927 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 23:41:27.002487 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.997876 4927 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 23:41:27.002487 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.997887 4927 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 23:41:27.002487 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.997890 4927 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 23:41:27.002487 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.997893 4927 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 23:41:27.002487 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.997896 4927 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 23:41:27.002487 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.997899 4927 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 23:41:27.002487 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.997902 4927 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 23:41:27.002487 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.997905 4927 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 23:41:27.002487 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.997907 4927 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 23:41:27.002487 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.997910 4927 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 23:41:27.002487 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.997913 4927 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 23:41:27.002487 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.997916 4927 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 23:41:27.002487 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.997918 4927 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 23:41:27.002487 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.997921 4927 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 23:41:27.002948 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.997924 4927 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 23:41:27.002948 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.997926 4927 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 23:41:27.002948 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.997929 4927 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 23:41:27.002948 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.997932 4927 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 23:41:27.002948 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.997934 4927 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 23:41:27.002948 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.997937 4927 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 23:41:27.002948 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.997939 4927 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 23:41:27.002948 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.997942 4927 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 23:41:27.002948 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.997947 4927 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 23:41:27.002948 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.997950 4927 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 23:41:27.002948 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.997952 4927 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 23:41:27.002948 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.997955 4927 feature_gate.go:386] unrecognized feature gate: Example Oct 30 23:41:27.002948 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.997958 4927 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 23:41:27.002948 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.997961 4927 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 23:41:27.002948 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.997972 4927 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 23:41:27.002948 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.997975 4927 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 23:41:27.002948 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.997977 4927 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 23:41:27.002948 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.997980 4927 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 23:41:27.002948 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.997983 4927 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 23:41:27.003607 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.997986 4927 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 23:41:27.003607 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.997989 4927 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 23:41:27.003607 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.997993 4927 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 23:41:27.003607 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.997996 4927 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 23:41:27.003607 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.997998 4927 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 23:41:27.003607 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.998001 4927 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 23:41:27.003607 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.998004 4927 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 23:41:27.003607 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.998007 4927 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 23:41:27.003607 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.998009 4927 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 23:41:27.003607 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.998012 4927 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 23:41:27.003607 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.998015 4927 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 23:41:27.003607 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.998018 4927 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 23:41:27.003607 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.998020 4927 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 23:41:27.003607 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.998023 4927 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 23:41:27.003607 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.998025 4927 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 23:41:27.003607 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.998028 4927 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 23:41:27.003607 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.998030 4927 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 23:41:27.003607 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.998033 4927 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 23:41:27.003607 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.998036 4927 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 23:41:27.003607 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.998039 4927 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 23:41:27.004123 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.998041 4927 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 23:41:27.004123 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.998045 4927 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 23:41:27.004123 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.998048 4927 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 23:41:27.004123 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.998051 4927 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 23:41:27.004123 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.998053 4927 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 23:41:27.004123 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.998056 4927 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 23:41:27.004123 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.998059 4927 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 23:41:27.004123 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.998062 4927 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 23:41:27.004123 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.998064 4927 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 23:41:27.004123 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.998067 4927 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 23:41:27.004123 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.998070 4927 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 23:41:27.004123 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.998072 4927 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 23:41:27.004123 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.998075 4927 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 23:41:27.004123 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.998077 4927 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 23:41:27.004123 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.998081 4927 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 23:41:27.004123 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.998084 4927 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 23:41:27.004123 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.998086 4927 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 23:41:27.004123 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.998089 4927 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 23:41:27.004123 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.998092 4927 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 23:41:27.004123 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.998094 4927 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 23:41:27.004643 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.998098 4927 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 23:41:27.004643 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.998102 4927 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 23:41:27.004643 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.998105 4927 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 23:41:27.004643 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.998109 4927 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 23:41:27.004643 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.998111 4927 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 23:41:27.004643 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.998114 4927 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 23:41:27.004643 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:26.998225 4927 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 23:41:27.004643 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:26.998232 4927 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 23:41:27.004643 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:26.998241 4927 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 23:41:27.004643 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:26.998245 4927 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 23:41:27.004643 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:26.998250 4927 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 23:41:27.004643 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:26.998253 4927 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 23:41:27.004643 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:26.998257 4927 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 23:41:27.004643 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:26.998263 4927 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 23:41:27.004643 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:26.998266 4927 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 23:41:27.004643 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:26.998269 4927 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 23:41:27.004643 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:26.998273 4927 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 23:41:27.004643 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:26.998277 4927 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 23:41:27.004643 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:26.998280 4927 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 23:41:27.004643 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:26.998283 4927 flags.go:64] FLAG: --cgroup-root="" Oct 30 23:41:27.004643 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:26.998286 4927 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 23:41:27.004643 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:26.998289 4927 flags.go:64] FLAG: --client-ca-file="" Oct 30 23:41:27.005180 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:26.998293 4927 flags.go:64] FLAG: --cloud-provider="external" Oct 30 23:41:27.005180 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:26.998296 4927 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 23:41:27.005180 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:26.998300 4927 flags.go:64] FLAG: --cluster-domain="" Oct 30 23:41:27.005180 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:26.998303 4927 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 23:41:27.005180 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:26.998306 4927 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 23:41:27.005180 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:26.998310 4927 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 23:41:27.005180 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:26.998314 4927 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 23:41:27.005180 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:26.998317 4927 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 23:41:27.005180 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:26.998321 4927 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 23:41:27.005180 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:26.998324 4927 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 23:41:27.005180 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:26.998340 4927 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 23:41:27.005180 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:26.998344 4927 flags.go:64] FLAG: --contention-profiling="false" Oct 30 23:41:27.005180 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:26.998347 4927 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 23:41:27.005180 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:26.998350 4927 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 23:41:27.005180 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:26.998353 4927 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 23:41:27.005180 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:26.998357 4927 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 23:41:27.005180 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:26.998361 4927 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 23:41:27.005180 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:26.998364 4927 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 23:41:27.005180 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:26.998367 4927 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 23:41:27.005180 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:26.998377 4927 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 23:41:27.005180 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:26.998381 4927 flags.go:64] FLAG: --enable-server="true" Oct 30 23:41:27.005180 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:26.998384 4927 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 23:41:27.005180 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:26.998389 4927 flags.go:64] FLAG: --event-burst="100" Oct 30 23:41:27.005180 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:26.998395 4927 flags.go:64] FLAG: --event-qps="50" Oct 30 23:41:27.005180 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:26.998399 4927 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 23:41:27.005852 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:26.998402 4927 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 23:41:27.005852 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:26.998405 4927 flags.go:64] FLAG: --eviction-hard="" Oct 30 23:41:27.005852 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:26.998410 4927 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 23:41:27.005852 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:26.998413 4927 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 23:41:27.005852 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:26.998416 4927 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 23:41:27.005852 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:26.998419 4927 flags.go:64] FLAG: --eviction-soft="" Oct 30 23:41:27.005852 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:26.998422 4927 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 23:41:27.005852 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:26.998426 4927 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 23:41:27.005852 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:26.998430 4927 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 23:41:27.005852 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:26.998435 4927 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 23:41:27.005852 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:26.998445 4927 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 23:41:27.005852 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:26.998449 4927 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 23:41:27.005852 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:26.998453 4927 flags.go:64] FLAG: --feature-gates="" Oct 30 23:41:27.005852 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:26.998458 4927 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 23:41:27.005852 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:26.998462 4927 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 23:41:27.005852 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:26.998465 4927 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 23:41:27.005852 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:26.998469 4927 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 23:41:27.005852 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:26.998472 4927 flags.go:64] FLAG: --healthz-port="10248" Oct 30 23:41:27.005852 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:26.998476 4927 flags.go:64] FLAG: --help="false" Oct 30 23:41:27.005852 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:26.998479 4927 flags.go:64] FLAG: --hostname-override="ip-10-0-141-180.us-west-2.compute.internal" Oct 30 23:41:27.005852 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:26.998482 4927 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 23:41:27.005852 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:26.998485 4927 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 23:41:27.005852 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:26.998488 4927 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 23:41:27.006463 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:26.998492 4927 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 23:41:27.006463 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:26.998495 4927 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 23:41:27.006463 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:26.998498 4927 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 23:41:27.006463 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:26.998501 4927 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 23:41:27.006463 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:26.998505 4927 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 23:41:27.006463 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:26.998508 4927 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 23:41:27.006463 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:26.998511 4927 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 23:41:27.006463 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:26.998519 4927 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 23:41:27.006463 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:26.998522 4927 flags.go:64] FLAG: --kube-reserved="" Oct 30 23:41:27.006463 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:26.998525 4927 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 23:41:27.006463 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:26.998528 4927 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 23:41:27.006463 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:26.998532 4927 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 23:41:27.006463 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:26.998535 4927 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 23:41:27.006463 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:26.998538 4927 flags.go:64] FLAG: --lock-file="" Oct 30 23:41:27.006463 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:26.998541 4927 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 23:41:27.006463 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:26.998544 4927 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 23:41:27.006463 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:26.998547 4927 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 23:41:27.006463 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:26.998553 4927 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 23:41:27.006463 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:26.998556 4927 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 23:41:27.006463 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:26.998560 4927 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 23:41:27.006463 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:26.998563 4927 flags.go:64] FLAG: --logging-format="text" Oct 30 23:41:27.006463 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:26.998566 4927 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 23:41:27.006463 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:26.998569 4927 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 23:41:27.006463 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:26.998579 4927 flags.go:64] FLAG: --manifest-url="" Oct 30 23:41:27.007049 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:26.998582 4927 flags.go:64] FLAG: --manifest-url-header="" Oct 30 23:41:27.007049 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:26.998587 4927 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 23:41:27.007049 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:26.998590 4927 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 23:41:27.007049 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:26.998594 4927 flags.go:64] FLAG: --max-pods="110" Oct 30 23:41:27.007049 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:26.998598 4927 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 23:41:27.007049 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:26.998601 4927 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 23:41:27.007049 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:26.998604 4927 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 23:41:27.007049 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:26.998607 4927 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 23:41:27.007049 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:26.998610 4927 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 23:41:27.007049 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:26.998613 4927 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 23:41:27.007049 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:26.998616 4927 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 23:41:27.007049 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:26.998623 4927 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 23:41:27.007049 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:26.998626 4927 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 23:41:27.007049 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:26.998629 4927 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 23:41:27.007049 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:26.998632 4927 flags.go:64] FLAG: --pod-cidr="" Oct 30 23:41:27.007049 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:26.998637 4927 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 23:41:27.007049 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:26.998640 4927 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 23:41:27.007049 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:26.998643 4927 flags.go:64] FLAG: --pods-per-core="0" Oct 30 23:41:27.007049 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:26.998646 4927 flags.go:64] FLAG: --port="10250" Oct 30 23:41:27.007049 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:26.998649 4927 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 23:41:27.007049 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:26.998652 4927 flags.go:64] FLAG: --provider-id="aws:///us-west-2a/i-01256209b5ec9c35f" Oct 30 23:41:27.007049 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:26.998656 4927 flags.go:64] FLAG: --qos-reserved="" Oct 30 23:41:27.007049 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:26.998659 4927 flags.go:64] FLAG: --read-only-port="10255" Oct 30 23:41:27.007049 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:26.998662 4927 flags.go:64] FLAG: --register-node="true" Oct 30 23:41:27.007049 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:26.998665 4927 flags.go:64] FLAG: --register-with-taints="" Oct 30 23:41:27.007049 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:26.998669 4927 flags.go:64] FLAG: --registry-burst="10" Oct 30 23:41:27.007742 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:26.998672 4927 flags.go:64] FLAG: --registry-qps="5" Oct 30 23:41:27.007742 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:26.998675 4927 flags.go:64] FLAG: --reserved-cpus="" Oct 30 23:41:27.007742 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:26.998678 4927 flags.go:64] FLAG: --reserved-memory="" Oct 30 23:41:27.007742 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:26.998682 4927 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 23:41:27.007742 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:26.998685 4927 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 23:41:27.007742 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:26.998689 4927 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 23:41:27.007742 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:26.998698 4927 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 23:41:27.007742 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:26.998701 4927 flags.go:64] FLAG: --runonce="false" Oct 30 23:41:27.007742 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:26.998704 4927 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 23:41:27.007742 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:26.998707 4927 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 23:41:27.007742 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:26.998711 4927 flags.go:64] FLAG: --seccomp-default="false" Oct 30 23:41:27.007742 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:26.998714 4927 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 23:41:27.007742 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:26.998717 4927 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 23:41:27.007742 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:26.998722 4927 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 23:41:27.007742 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:26.998725 4927 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 23:41:27.007742 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:26.998728 4927 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 23:41:27.007742 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:26.998731 4927 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 23:41:27.007742 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:26.998734 4927 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 23:41:27.007742 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:26.998737 4927 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 23:41:27.007742 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:26.998740 4927 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 23:41:27.007742 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:26.998743 4927 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 23:41:27.007742 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:26.998747 4927 flags.go:64] FLAG: --system-cgroups="" Oct 30 23:41:27.007742 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:26.998750 4927 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 23:41:27.007742 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:26.998755 4927 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 23:41:27.007742 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:26.998758 4927 flags.go:64] FLAG: --tls-cert-file="" Oct 30 23:41:27.008696 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:26.998761 4927 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 23:41:27.008696 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:26.998765 4927 flags.go:64] FLAG: --tls-min-version="" Oct 30 23:41:27.008696 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:26.998768 4927 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 23:41:27.008696 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:26.998771 4927 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 23:41:27.008696 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:26.998774 4927 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 23:41:27.008696 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:26.998777 4927 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 23:41:27.008696 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:26.998780 4927 flags.go:64] FLAG: --v="2" Oct 30 23:41:27.008696 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:26.998785 4927 flags.go:64] FLAG: --version="false" Oct 30 23:41:27.008696 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:26.998789 4927 flags.go:64] FLAG: --vmodule="" Oct 30 23:41:27.008696 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:26.998793 4927 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 23:41:27.008696 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:26.998796 4927 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 23:41:27.008696 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.998963 4927 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 23:41:27.008696 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.998968 4927 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 23:41:27.008696 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.998971 4927 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 23:41:27.008696 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.998980 4927 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 23:41:27.008696 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.998984 4927 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 23:41:27.008696 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.998987 4927 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 23:41:27.008696 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.998990 4927 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 23:41:27.008696 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.998993 4927 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 23:41:27.008696 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.998998 4927 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 23:41:27.008696 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.999004 4927 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 23:41:27.008696 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.999008 4927 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 23:41:27.008696 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.999013 4927 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 23:41:27.009455 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.999017 4927 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 23:41:27.009455 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.999021 4927 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 23:41:27.009455 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.999025 4927 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 23:41:27.009455 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.999030 4927 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 23:41:27.009455 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.999035 4927 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 23:41:27.009455 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.999041 4927 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 23:41:27.009455 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.999047 4927 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 23:41:27.009455 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.999051 4927 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 23:41:27.009455 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.999055 4927 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 23:41:27.009455 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.999059 4927 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 23:41:27.009455 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.999063 4927 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 23:41:27.009455 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.999066 4927 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 23:41:27.009455 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.999070 4927 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 23:41:27.009455 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.999074 4927 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 23:41:27.009455 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.999078 4927 feature_gate.go:386] unrecognized feature gate: Example Oct 30 23:41:27.009455 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.999082 4927 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 23:41:27.009455 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.999087 4927 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 23:41:27.009455 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.999090 4927 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 23:41:27.009455 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.999095 4927 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 23:41:27.009918 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.999098 4927 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 23:41:27.009918 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.999103 4927 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 23:41:27.009918 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.999107 4927 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 23:41:27.009918 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.999111 4927 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 23:41:27.009918 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.999115 4927 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 23:41:27.009918 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.999127 4927 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 23:41:27.009918 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.999132 4927 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 23:41:27.009918 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.999137 4927 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 23:41:27.009918 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.999141 4927 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 23:41:27.009918 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.999145 4927 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 23:41:27.009918 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.999154 4927 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 23:41:27.009918 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.999158 4927 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 23:41:27.009918 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.999161 4927 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 23:41:27.009918 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.999165 4927 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 23:41:27.009918 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.999169 4927 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 23:41:27.009918 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.999173 4927 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 23:41:27.009918 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.999177 4927 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 23:41:27.009918 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.999181 4927 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 23:41:27.009918 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.999187 4927 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 23:41:27.009918 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.999191 4927 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 23:41:27.010455 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.999195 4927 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 23:41:27.010455 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.999199 4927 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 23:41:27.010455 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.999203 4927 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 23:41:27.010455 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.999206 4927 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 23:41:27.010455 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.999211 4927 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 23:41:27.010455 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.999215 4927 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 23:41:27.010455 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.999219 4927 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 23:41:27.010455 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.999223 4927 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 23:41:27.010455 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.999227 4927 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 23:41:27.010455 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.999230 4927 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 23:41:27.010455 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.999234 4927 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 23:41:27.010455 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.999238 4927 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 23:41:27.010455 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.999243 4927 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 23:41:27.010455 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.999247 4927 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 23:41:27.010455 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.999251 4927 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 23:41:27.010455 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.999255 4927 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 23:41:27.010455 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.999258 4927 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 23:41:27.010455 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.999262 4927 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 23:41:27.010455 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.999275 4927 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 23:41:27.010455 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.999280 4927 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 23:41:27.010981 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.999284 4927 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 23:41:27.010981 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.999288 4927 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 23:41:27.010981 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.999293 4927 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 23:41:27.010981 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.999298 4927 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 23:41:27.010981 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.999302 4927 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 23:41:27.010981 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.999305 4927 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 23:41:27.010981 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.999309 4927 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 23:41:27.010981 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.999314 4927 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 23:41:27.010981 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.999319 4927 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 23:41:27.010981 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.999323 4927 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 23:41:27.010981 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.999344 4927 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 23:41:27.010981 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.999349 4927 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 23:41:27.010981 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.999352 4927 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 23:41:27.010981 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.999357 4927 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 23:41:27.010981 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.999361 4927 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 23:41:27.010981 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.999365 4927 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 23:41:27.010981 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.999369 4927 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 23:41:27.010981 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.999373 4927 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 23:41:27.010981 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.999378 4927 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 23:41:27.010981 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.999382 4927 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 23:41:27.011687 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.999386 4927 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 23:41:27.011687 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.999390 4927 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 23:41:27.011687 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.999396 4927 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 23:41:27.011687 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.999403 4927 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 23:41:27.011687 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.999408 4927 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 23:41:27.011687 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.999412 4927 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 23:41:27.011687 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.999417 4927 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 23:41:27.011687 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.999421 4927 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 23:41:27.011687 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.999425 4927 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 23:41:27.011687 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.999429 4927 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 23:41:27.011687 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.999433 4927 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 23:41:27.011687 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.999445 4927 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 23:41:27.011687 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.999450 4927 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 23:41:27.011687 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.999454 4927 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 23:41:27.011687 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.999467 4927 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 23:41:27.012141 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:26.999505 4927 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 23:41:27.012141 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.999555 4927 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 23:41:27.012141 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.999559 4927 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 23:41:27.012141 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.999562 4927 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 23:41:27.012141 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.999566 4927 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 23:41:27.012141 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.999569 4927 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 23:41:27.012141 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.999572 4927 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 23:41:27.012141 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.999575 4927 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 23:41:27.012141 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.999578 4927 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 23:41:27.012141 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.999581 4927 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 23:41:27.012141 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.999584 4927 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 23:41:27.012141 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.999587 4927 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 23:41:27.012141 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.999590 4927 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 23:41:27.012141 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.999593 4927 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 23:41:27.012141 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.999595 4927 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 23:41:27.012522 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.999598 4927 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 23:41:27.012522 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.999602 4927 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 23:41:27.012522 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.999606 4927 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 23:41:27.012522 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.999610 4927 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 23:41:27.012522 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.999613 4927 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 23:41:27.012522 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.999616 4927 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 23:41:27.012522 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.999619 4927 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 23:41:27.012522 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.999622 4927 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 23:41:27.012522 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.999625 4927 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 23:41:27.012522 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.999628 4927 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 23:41:27.012522 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.999631 4927 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 23:41:27.012522 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.999634 4927 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 23:41:27.012522 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.999636 4927 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 23:41:27.012522 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.999639 4927 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 23:41:27.012522 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.999647 4927 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 23:41:27.012522 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.999650 4927 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 23:41:27.012522 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.999654 4927 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 23:41:27.012522 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.999657 4927 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 23:41:27.012522 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.999660 4927 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 23:41:27.013030 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.999662 4927 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 23:41:27.013030 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.999665 4927 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 23:41:27.013030 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.999668 4927 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 23:41:27.013030 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.999670 4927 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 23:41:27.013030 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.999673 4927 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 23:41:27.013030 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.999676 4927 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 23:41:27.013030 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.999678 4927 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 23:41:27.013030 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.999681 4927 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 23:41:27.013030 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.999684 4927 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 23:41:27.013030 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.999687 4927 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 23:41:27.013030 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.999691 4927 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 23:41:27.013030 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.999694 4927 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 23:41:27.013030 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.999697 4927 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 23:41:27.013030 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.999699 4927 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 23:41:27.013030 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.999702 4927 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 23:41:27.013030 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.999705 4927 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 23:41:27.013030 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.999707 4927 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 23:41:27.013030 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.999710 4927 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 23:41:27.013030 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.999713 4927 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 23:41:27.013679 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.999715 4927 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 23:41:27.013679 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.999718 4927 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 23:41:27.013679 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.999721 4927 feature_gate.go:386] unrecognized feature gate: Example Oct 30 23:41:27.013679 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.999724 4927 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 23:41:27.013679 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.999726 4927 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 23:41:27.013679 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.999729 4927 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 23:41:27.013679 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.999732 4927 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 23:41:27.013679 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.999734 4927 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 23:41:27.013679 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.999737 4927 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 23:41:27.013679 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.999745 4927 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 23:41:27.013679 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.999750 4927 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 23:41:27.013679 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.999752 4927 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 23:41:27.013679 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.999755 4927 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 23:41:27.013679 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.999758 4927 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 23:41:27.013679 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.999761 4927 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 23:41:27.013679 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.999764 4927 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 23:41:27.013679 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.999766 4927 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 23:41:27.013679 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.999769 4927 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 23:41:27.013679 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.999771 4927 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 23:41:27.014217 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.999774 4927 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 23:41:27.014217 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.999777 4927 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 23:41:27.014217 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.999780 4927 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 23:41:27.014217 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.999783 4927 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 23:41:27.014217 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.999786 4927 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 23:41:27.014217 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.999789 4927 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 23:41:27.014217 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.999791 4927 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 23:41:27.014217 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.999794 4927 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 23:41:27.014217 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.999796 4927 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 23:41:27.014217 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.999799 4927 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 23:41:27.014217 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.999802 4927 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 23:41:27.014217 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.999804 4927 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 23:41:27.014217 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.999807 4927 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 23:41:27.014217 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.999810 4927 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 23:41:27.014217 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.999812 4927 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 23:41:27.014217 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.999815 4927 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 23:41:27.014217 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.999819 4927 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 23:41:27.014217 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.999822 4927 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 23:41:27.014217 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.999825 4927 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 23:41:27.014684 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.999828 4927 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 23:41:27.014684 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.999830 4927 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 23:41:27.014684 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.999833 4927 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 23:41:27.014684 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.999836 4927 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 23:41:27.014684 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.999846 4927 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 23:41:27.014684 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.999849 4927 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 23:41:27.014684 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.999852 4927 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 23:41:27.014684 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.999855 4927 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 23:41:27.014684 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.999858 4927 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 23:41:27.014684 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.999861 4927 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 23:41:27.014684 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.999863 4927 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 23:41:27.014684 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.999866 4927 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 23:41:27.014684 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.999868 4927 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 23:41:27.014684 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.999871 4927 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 23:41:27.014684 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.999874 4927 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 23:41:27.014684 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:26.999877 4927 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 23:41:27.014684 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:27.004714 4927 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 23:41:27.014684 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:27.004734 4927 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 23:41:27.014684 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:27.004748 4927 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 23:41:27.014684 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:27.004753 4927 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 23:41:27.015166 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.004831 4927 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 23:41:27.015166 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.004837 4927 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 23:41:27.015166 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.004841 4927 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 23:41:27.015166 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.004844 4927 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 23:41:27.015166 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.004847 4927 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 23:41:27.015166 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.004850 4927 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 23:41:27.015166 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.004853 4927 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 23:41:27.015166 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.004856 4927 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 23:41:27.015166 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.004858 4927 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 23:41:27.015166 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.004861 4927 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 23:41:27.015166 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.004864 4927 feature_gate.go:386] unrecognized feature gate: Example Oct 30 23:41:27.015166 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.004866 4927 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 23:41:27.015166 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.004869 4927 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 23:41:27.015166 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.004872 4927 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 23:41:27.015166 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.004875 4927 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 23:41:27.015166 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.004877 4927 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 23:41:27.015166 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.004880 4927 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 23:41:27.015166 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.004883 4927 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 23:41:27.015166 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.004885 4927 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 23:41:27.015166 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.004888 4927 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 23:41:27.015657 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.004891 4927 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 23:41:27.015657 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.004894 4927 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 23:41:27.015657 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.004897 4927 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 23:41:27.015657 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.004900 4927 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 23:41:27.015657 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.004902 4927 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 23:41:27.015657 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.004905 4927 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 23:41:27.015657 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.004908 4927 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 23:41:27.015657 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.004911 4927 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 23:41:27.015657 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.004914 4927 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 23:41:27.015657 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.004922 4927 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 23:41:27.015657 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.004925 4927 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 23:41:27.015657 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.004927 4927 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 23:41:27.015657 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.004931 4927 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 23:41:27.015657 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.004933 4927 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 23:41:27.015657 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.004936 4927 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 23:41:27.015657 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.004938 4927 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 23:41:27.015657 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.004941 4927 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 23:41:27.015657 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.004944 4927 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 23:41:27.015657 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.004946 4927 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 23:41:27.015657 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.004949 4927 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 23:41:27.016138 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.004952 4927 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 23:41:27.016138 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.004955 4927 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 23:41:27.016138 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.004958 4927 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 23:41:27.016138 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.004961 4927 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 23:41:27.016138 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.004963 4927 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 23:41:27.016138 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.004966 4927 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 23:41:27.016138 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.004969 4927 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 23:41:27.016138 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.004972 4927 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 23:41:27.016138 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.004975 4927 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 23:41:27.016138 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.004978 4927 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 23:41:27.016138 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.004981 4927 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 23:41:27.016138 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.004984 4927 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 23:41:27.016138 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.004986 4927 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 23:41:27.016138 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.004989 4927 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 23:41:27.016138 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.004992 4927 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 23:41:27.016138 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.004994 4927 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 23:41:27.016138 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.004997 4927 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 23:41:27.016138 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.005000 4927 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 23:41:27.016138 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.005003 4927 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 23:41:27.016138 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.005006 4927 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 23:41:27.016956 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.005009 4927 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 23:41:27.016956 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.005011 4927 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 23:41:27.016956 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.005020 4927 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 23:41:27.016956 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.005023 4927 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 23:41:27.016956 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.005026 4927 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 23:41:27.016956 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.005029 4927 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 23:41:27.016956 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.005031 4927 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 23:41:27.016956 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.005034 4927 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 23:41:27.016956 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.005037 4927 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 23:41:27.016956 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.005040 4927 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 23:41:27.016956 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.005043 4927 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 23:41:27.016956 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.005046 4927 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 23:41:27.016956 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.005048 4927 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 23:41:27.016956 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.005053 4927 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 23:41:27.016956 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.005057 4927 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 23:41:27.016956 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.005060 4927 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 23:41:27.016956 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.005063 4927 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 23:41:27.016956 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.005066 4927 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 23:41:27.016956 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.005076 4927 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 23:41:27.017473 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.005080 4927 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 23:41:27.017473 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.005083 4927 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 23:41:27.017473 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.005085 4927 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 23:41:27.017473 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.005088 4927 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 23:41:27.017473 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.005091 4927 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 23:41:27.017473 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.005094 4927 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 23:41:27.017473 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.005097 4927 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 23:41:27.017473 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.005100 4927 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 23:41:27.017473 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.005104 4927 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 23:41:27.017473 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.005108 4927 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 23:41:27.017473 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.005111 4927 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 23:41:27.017473 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.005114 4927 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 23:41:27.017473 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.005117 4927 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 23:41:27.017473 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.005120 4927 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 23:41:27.017473 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.005122 4927 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 23:41:27.017473 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.005125 4927 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 23:41:27.017473 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.005134 4927 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 23:41:27.017473 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.005137 4927 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 23:41:27.017473 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.005139 4927 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 23:41:27.018072 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.005142 4927 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 23:41:27.018072 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.005144 4927 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 23:41:27.018072 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.005147 4927 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 23:41:27.018072 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.005150 4927 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 23:41:27.018072 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.005153 4927 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 23:41:27.018072 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.005155 4927 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 23:41:27.018072 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.005158 4927 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 23:41:27.018072 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.005161 4927 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 23:41:27.018072 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:27.005189 4927 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 23:41:27.018072 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.005391 4927 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 23:41:27.018072 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.005396 4927 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 23:41:27.018072 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.005399 4927 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 23:41:27.018072 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.005402 4927 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 23:41:27.018072 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.005405 4927 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 23:41:27.018072 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.005408 4927 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 23:41:27.018072 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.005410 4927 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 23:41:27.018482 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.005413 4927 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 23:41:27.018482 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.005416 4927 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 23:41:27.018482 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.005419 4927 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 23:41:27.018482 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.005422 4927 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 23:41:27.018482 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.005425 4927 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 23:41:27.018482 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.005427 4927 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 23:41:27.018482 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.005430 4927 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 23:41:27.018482 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.005433 4927 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 23:41:27.018482 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.005435 4927 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 23:41:27.018482 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.005438 4927 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 23:41:27.018482 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.005441 4927 feature_gate.go:386] unrecognized feature gate: Example Oct 30 23:41:27.018482 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.005444 4927 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 23:41:27.018482 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.005446 4927 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 23:41:27.018482 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.005449 4927 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 23:41:27.018482 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.005457 4927 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 23:41:27.018482 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.005460 4927 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 23:41:27.018482 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.005463 4927 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 23:41:27.018482 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.005466 4927 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 23:41:27.018482 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.005468 4927 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 23:41:27.018482 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.005471 4927 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 23:41:27.018961 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.005474 4927 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 23:41:27.018961 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.005476 4927 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 23:41:27.018961 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.005479 4927 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 23:41:27.018961 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.005482 4927 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 23:41:27.018961 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.005484 4927 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 23:41:27.018961 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.005487 4927 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 23:41:27.018961 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.005490 4927 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 23:41:27.018961 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.005493 4927 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 23:41:27.018961 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.005496 4927 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 23:41:27.018961 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.005498 4927 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 23:41:27.018961 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.005501 4927 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 23:41:27.018961 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.005504 4927 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 23:41:27.018961 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.005506 4927 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 23:41:27.018961 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.005509 4927 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 23:41:27.018961 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.005512 4927 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 23:41:27.018961 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.005515 4927 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 23:41:27.018961 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.005517 4927 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 23:41:27.018961 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.005520 4927 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 23:41:27.018961 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.005523 4927 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 23:41:27.018961 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.005525 4927 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 23:41:27.019505 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.005528 4927 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 23:41:27.019505 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.005531 4927 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 23:41:27.019505 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.005533 4927 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 23:41:27.019505 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.005536 4927 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 23:41:27.019505 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.005539 4927 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 23:41:27.019505 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.005541 4927 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 23:41:27.019505 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.005544 4927 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 23:41:27.019505 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.005552 4927 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 23:41:27.019505 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.005555 4927 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 23:41:27.019505 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.005557 4927 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 23:41:27.019505 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.005561 4927 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 23:41:27.019505 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.005563 4927 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 23:41:27.019505 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.005566 4927 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 23:41:27.019505 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.005569 4927 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 23:41:27.019505 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.005571 4927 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 23:41:27.019505 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.005574 4927 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 23:41:27.019505 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.005576 4927 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 23:41:27.019505 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.005580 4927 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 23:41:27.019505 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.005585 4927 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 23:41:27.020011 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.005588 4927 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 23:41:27.020011 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.005591 4927 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 23:41:27.020011 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.005594 4927 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 23:41:27.020011 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.005597 4927 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 23:41:27.020011 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.005599 4927 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 23:41:27.020011 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.005602 4927 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 23:41:27.020011 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.005605 4927 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 23:41:27.020011 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.005608 4927 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 23:41:27.020011 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.005610 4927 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 23:41:27.020011 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.005613 4927 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 23:41:27.020011 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.005623 4927 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 23:41:27.020011 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.005627 4927 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 23:41:27.020011 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.005629 4927 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 23:41:27.020011 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.005632 4927 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 23:41:27.020011 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.005635 4927 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 23:41:27.020011 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.005638 4927 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 23:41:27.020011 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.005640 4927 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 23:41:27.020011 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.005643 4927 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 23:41:27.020011 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.005645 4927 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 23:41:27.020011 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.005648 4927 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 23:41:27.020537 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.005651 4927 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 23:41:27.020537 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.005659 4927 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 23:41:27.020537 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.005662 4927 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 23:41:27.020537 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.005665 4927 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 23:41:27.020537 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.005667 4927 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 23:41:27.020537 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.005670 4927 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 23:41:27.020537 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.005672 4927 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 23:41:27.020537 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.005675 4927 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 23:41:27.020537 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.005678 4927 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 23:41:27.020537 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.005680 4927 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 23:41:27.020537 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.005683 4927 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 23:41:27.020537 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.005685 4927 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 23:41:27.020537 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.005688 4927 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 23:41:27.020537 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.005691 4927 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 23:41:27.020537 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.005693 4927 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 23:41:27.020537 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.005696 4927 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 23:41:27.020537 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.005699 4927 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 23:41:27.020537 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.005703 4927 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 23:41:27.020537 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.005706 4927 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 23:41:27.021002 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.005709 4927 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 23:41:27.021002 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:27.005736 4927 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 23:41:27.021002 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.005774 4927 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 23:41:27.021002 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.005778 4927 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 23:41:27.021002 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.005781 4927 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 23:41:27.021002 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.005784 4927 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 23:41:27.021002 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.005786 4927 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 23:41:27.021002 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.005789 4927 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 23:41:27.021002 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.005792 4927 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 23:41:27.021002 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.005795 4927 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 23:41:27.021002 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.005798 4927 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 23:41:27.021002 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.005801 4927 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 23:41:27.021002 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.005804 4927 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 23:41:27.021002 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.005806 4927 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 23:41:27.021002 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.005809 4927 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 23:41:27.021002 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.005817 4927 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 23:41:27.021541 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.005820 4927 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 23:41:27.021541 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.005823 4927 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 23:41:27.021541 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.005826 4927 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 23:41:27.021541 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.005828 4927 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 23:41:27.021541 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.005831 4927 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 23:41:27.021541 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.005833 4927 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 23:41:27.021541 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.005836 4927 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 23:41:27.021541 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.005839 4927 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 23:41:27.021541 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.005841 4927 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 23:41:27.021541 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.005844 4927 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 23:41:27.021541 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.005847 4927 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 23:41:27.021541 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.005850 4927 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 23:41:27.021541 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.005852 4927 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 23:41:27.021541 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.005855 4927 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 23:41:27.021541 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.005858 4927 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 23:41:27.021541 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.005861 4927 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 23:41:27.021541 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.005864 4927 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 23:41:27.021541 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.005867 4927 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 23:41:27.021541 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.005872 4927 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 23:41:27.021541 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.005875 4927 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 23:41:27.022051 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.005878 4927 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 23:41:27.022051 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.005880 4927 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 23:41:27.022051 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.005883 4927 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 23:41:27.022051 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.005885 4927 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 23:41:27.022051 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.005889 4927 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 23:41:27.022051 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.005893 4927 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 23:41:27.022051 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.005896 4927 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 23:41:27.022051 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.005899 4927 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 23:41:27.022051 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.005902 4927 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 23:41:27.022051 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.005905 4927 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 23:41:27.022051 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.005907 4927 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 23:41:27.022051 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.005910 4927 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 23:41:27.022051 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.005919 4927 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 23:41:27.022051 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.005922 4927 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 23:41:27.022051 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.005925 4927 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 23:41:27.022051 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.005927 4927 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 23:41:27.022051 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.005930 4927 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 23:41:27.022051 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.005933 4927 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 23:41:27.022051 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.005936 4927 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 23:41:27.022526 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.005938 4927 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 23:41:27.022526 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.005942 4927 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 23:41:27.022526 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.005945 4927 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 23:41:27.022526 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.005947 4927 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 23:41:27.022526 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.005950 4927 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 23:41:27.022526 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.005953 4927 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 23:41:27.022526 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.005956 4927 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 23:41:27.022526 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.005958 4927 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 23:41:27.022526 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.005961 4927 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 23:41:27.022526 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.005964 4927 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 23:41:27.022526 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.005967 4927 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 23:41:27.022526 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.005971 4927 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 23:41:27.022526 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.005973 4927 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 23:41:27.022526 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.005976 4927 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 23:41:27.022526 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.005979 4927 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 23:41:27.022526 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.005982 4927 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 23:41:27.022526 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.005984 4927 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 23:41:27.022526 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.005987 4927 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 23:41:27.022526 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.005989 4927 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 23:41:27.022982 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.005992 4927 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 23:41:27.022982 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.005995 4927 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 23:41:27.022982 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.005998 4927 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 23:41:27.022982 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.006000 4927 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 23:41:27.022982 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.006004 4927 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 23:41:27.022982 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.006007 4927 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 23:41:27.022982 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.006010 4927 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 23:41:27.022982 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.006018 4927 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 23:41:27.022982 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.006021 4927 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 23:41:27.022982 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.006024 4927 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 23:41:27.022982 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.006026 4927 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 23:41:27.022982 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.006029 4927 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 23:41:27.022982 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.006032 4927 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 23:41:27.022982 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.006035 4927 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 23:41:27.022982 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.006037 4927 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 23:41:27.022982 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.006040 4927 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 23:41:27.022982 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.006044 4927 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 23:41:27.022982 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.006047 4927 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 23:41:27.022982 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.006050 4927 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 23:41:27.023504 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.006053 4927 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 23:41:27.023504 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.006055 4927 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 23:41:27.023504 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.006058 4927 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 23:41:27.023504 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.006061 4927 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 23:41:27.023504 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.006063 4927 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 23:41:27.023504 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.006067 4927 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 23:41:27.023504 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.006070 4927 feature_gate.go:386] unrecognized feature gate: Example Oct 30 23:41:27.023504 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.006072 4927 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 23:41:27.023504 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.006076 4927 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 23:41:27.023504 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.006079 4927 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 23:41:27.023504 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.006082 4927 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 23:41:27.023504 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.006084 4927 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 23:41:27.023504 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.006087 4927 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 23:41:27.023504 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.006090 4927 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 23:41:27.023504 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:27.006093 4927 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 23:41:27.023504 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:27.006153 4927 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 23:41:27.023504 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:27.007249 4927 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Oct 30 23:41:27.023504 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:27.007301 4927 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Oct 30 23:41:27.023504 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:27.007537 4927 server.go:1014] "Starting client certificate rotation" Oct 30 23:41:27.023954 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:27.007731 4927 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 23:41:27.023954 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:27.007799 4927 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 23:35:45 +0000 UTC" deadline="2028-07-14 16:48:28.35679204 +0000 UTC" Oct 30 23:41:27.023954 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:27.007864 4927 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14945h7m1.348929554s" Oct 30 23:41:27.023954 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:27.008829 4927 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 23:41:27.023954 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:27.010292 4927 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 23:41:27.023954 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:27.011494 4927 log.go:25] "Validated CRI v1 runtime API" Oct 30 23:41:27.023954 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:27.013146 4927 log.go:25] "Validated CRI v1 image API" Oct 30 23:41:27.023954 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:27.013908 4927 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 23:41:27.023954 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:27.015827 4927 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 9dfc8826-3266-4e14-bd7f-ea674cb91201:/dev/nvme0n1p4 c78ae7d5-3ca7-44fa-b0a6-cd1ea243dc19:/dev/nvme0n1p3] Oct 30 23:41:27.024215 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:27.015849 4927 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/01cae760460a3c29096c2785faece01841f55c4c9c9de869c85e2ef7134b502c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/01cae760460a3c29096c2785faece01841f55c4c9c9de869c85e2ef7134b502c/userdata/shm major:0 minor:234 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/020dec11112038ed834b49f31a2d35496a872cbc95e5b9692e8bf311a59c5260/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/020dec11112038ed834b49f31a2d35496a872cbc95e5b9692e8bf311a59c5260/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0a3626cfde5478d63a59c0432c8cac480a568eb5bcbdf0fcbd33e342c40e75dd/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0a3626cfde5478d63a59c0432c8cac480a568eb5bcbdf0fcbd33e342c40e75dd/userdata/shm major:0 minor:86 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0fd7a1669bcc37cc84500ce975c3ac8b58e4c1e03de0b5f9245530d20c9dc527/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0fd7a1669bcc37cc84500ce975c3ac8b58e4c1e03de0b5f9245530d20c9dc527/userdata/shm major:0 minor:230 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/154e4407f570053087b1f80f38a62a9607d8146fbbadeb07629573256de48e7e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/154e4407f570053087b1f80f38a62a9607d8146fbbadeb07629573256de48e7e/userdata/shm major:0 minor:56 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1dda77b536c5cf8fe095e91e3a2da765111f2116770dcb972dc5b0b76134b542/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1dda77b536c5cf8fe095e91e3a2da765111f2116770dcb972dc5b0b76134b542/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2cd1d530b1d2e4fbf2016043d891c3f05ed64bc1e150f43ac461b47dd0c04439/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2cd1d530b1d2e4fbf2016043d891c3f05ed64bc1e150f43ac461b47dd0c04439/userdata/shm major:0 minor:232 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4357c8b1b1cd7c31d12ef70138214a17b84944e2afc3d6d78932c37e9743b036/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4357c8b1b1cd7c31d12ef70138214a17b84944e2afc3d6d78932c37e9743b036/userdata/shm major:0 minor:62 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4d24e3019382f392346762ad0cdd801460b4a56c586d5f30a283293c21f78d5c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4d24e3019382f392346762ad0cdd801460b4a56c586d5f30a283293c21f78d5c/userdata/shm major:0 minor:236 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/69619f4a1d594665a6b7373c4f0480dec83db882660369456758935c0ee3aebd/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/69619f4a1d594665a6b7373c4f0480dec83db882660369456758935c0ee3aebd/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6b58028156e6606c849a36e8b17c9e190dc04bfc8a69a1f24a667e08b4a26210/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6b58028156e6606c849a36e8b17c9e190dc04bfc8a69a1f24a667e08b4a26210/userdata/shm major:0 minor:237 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/719c8f74b4146c489029a3cae27c9f4cc9d6c84b8ff50c6abfc15a23ef334941/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/719c8f74b4146c489029a3cae27c9f4cc9d6c84b8ff50c6abfc15a23ef334941/userdata/shm major:0 minor:246 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/722d14c37f4ca9c2f6ea9ad958e59ca48f6fb4bb8ffc975ccc53a0d713fcd4fc/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/722d14c37f4ca9c2f6ea9ad958e59ca48f6fb4bb8ffc975ccc53a0d713fcd4fc/userdata/shm major:0 minor:228 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8738eb3d4e62b725466ec53a2c859cd4d77af3b229836134893039fd90cf182e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8738eb3d4e62b725466ec53a2c859cd4d77af3b229836134893039fd90cf182e/userdata/shm major:0 minor:243 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a755ef97a7c53aaa8735a70265d8b3996ca96b08bf812706d173efd03f99c3bc/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a755ef97a7c53aaa8735a70265d8b3996ca96b08bf812706d173efd03f99c3bc/userdata/shm major:0 minor:288 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a85a7532289c5eee82ff2b7f93b6834d6ceba975a039ef17c247d87a33699c05/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a85a7532289c5eee82ff2b7f93b6834d6ceba975a039ef17c247d87a33699c05/userdata/shm major:0 minor:55 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ae3a0d475b2be1b25689edf2744177911624228a66bdbfc2ba94c3dccad12b36/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ae3a0d475b2be1b25689edf2744177911624228a66bdbfc2ba94c3dccad12b36/userdata/shm major:0 minor:240 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b4ecc76140e7d1ce05b22f67284d328e2dc8e4efa8a8ec078a459ef9338deee9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b4ecc76140e7d1ce05b22f67284d328e2dc8e4efa8a8ec078a459ef9338deee9/userdata/shm major:0 minor:290 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/bbfd66c255357d0b1ae9f531564af260a8d8bd00c26ccfe7f763b2cdb3afeaec/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/bbfd66c255357d0b1ae9f531564af260a8d8bd00c26ccfe7f763b2cdb3afeaec/userdata/shm major:0 minor:328 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c374b94e0a85533ff7652b5def95a433a127c7f26a0f314e854e8962450dae40/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c374b94e0a85533ff7652b5def95a433a127c7f26a0f314e854e8962450dae40/userdata/shm major:0 minor:74 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c80c7448baf9962aeb89efcadbfe35a76110a83280e99ce6a7d38c1eb89cac18/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c80c7448baf9962aeb89efcadbfe35a76110a83280e99ce6a7d38c1eb89cac18/userdata/shm major:0 minor:82 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c9010557cbcc988d15e3e177b0de68e23621b842a825a094cc2557634101e5c4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c9010557cbcc988d15e3e177b0de68e23621b842a825a094cc2557634101e5c4/userdata/shm major:0 minor:241 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d353d4d879e678a52dad33597146b7e44151f5f2886f5c2c28a1cdddea04459b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d353d4d879e678a52dad33597146b7e44151f5f2886f5c2c28a1cdddea04459b/userdata/shm major:0 minor:78 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/fe922c3105879e4776509bda85f73cd3fc9e6c078679a53ab53a6d6ff57f44b2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/fe922c3105879e4776509bda85f73cd3fc9e6c078679a53ab53a6d6ff57f44b2/userdata/shm major:0 minor:348 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/03211c6a-9761-419c-839b-21ca413d5bb2/volumes/kubernetes.io~projected/kube-api-access-5x8x6:{mountpoint:/var/lib/kubelet/pods/03211c6a-9761-419c-839b-21ca413d5bb2/volumes/kubernetes.io~projected/kube-api-access-5x8x6 major:0 minor:219 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0796261f-298e-4cbe-9b3e-b3dcf4528e07/volumes/kubernetes.io~projected/kube-api-access-25rtq:{mountpoint:/var/lib/kubelet/pods/0796261f-298e-4cbe-9b3e-b3dcf4528e07/volumes/kubernetes.io~projected/kube-api-access-25rtq major:0 minor:50 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0d4810de-8bd2-4270-b907-341d9f840bb8/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/0d4810de-8bd2-4270-b907-341d9f840bb8/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0d4810de-8bd2-4270-b907-341d9f840bb8/volumes/kubernetes.io~projected/kube-api-access-48m7s:{mountpoint:/var/lib/kubelet/pods/0d4810de-8bd2-4270-b907-341d9f840bb8/volumes/kubernetes.io~projected/kube-api-access-48m7s major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0d4810de-8bd2-4270-b907-341d9f840bb8/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/0d4810de-8bd2-4270-b907-341d9f840bb8/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:54 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/10774606-8e0c-460b-bd5f-4cb6d121b30b/volumes/kubernetes.io~projected/kube-api-access-n65gn:{mountpoint:/var/lib/kubelet/pods/10774606-8e0c-460b-bd5f-4cb6d121b30b/volumes/kubernetes.io~projected/kube-api-access-n65gn major:0 minor:216 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/10774606-8e0c-460b-bd5f-4cb6d121b30b/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/10774606-8e0c-460b-bd5f-4cb6d121b30b/volumes/kubernetes.io~secret/serving-cert major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/145dd323-9370-4b9d-a96e-8b93d0a47a7c/volumes/kubernetes.io~projected/kube-api-access-nsggg:{mountpoint:/var/lib/kubelet/pods/145dd323-9370-4b9d-a96e-8b93d0a47a7c/volumes/kubernetes.io~projected/kube-api-access-nsggg major:0 minor:48 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/17461b46-7ebf-49b7-afaf-12e20e3994a3/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/17461b46-7ebf-49b7-afaf-12e20e3994a3/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:210 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/17461b46-7ebf-49b7-afaf-12e20e3994a3/volumes/kubernetes.io~projected/kube-api-access-xkthh:{mountpoint:/var/lib/kubelet/pods/17461b46-7ebf-49b7-afaf-12e20e3994a3/volumes/kubernetes.io~projected/kube-api-access-xkthh major:0 minor:222 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/17461b46-7ebf-49b7-afaf-12e20e3994a3/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/17461b46-7ebf-49b7-afaf-12e20e3994a3/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/17461b46-7ebf-49b7-afaf-12e20e3994a3/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/17461b46-7ebf-49b7-afaf-12e20e3994a3/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:208 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/27ccca21-5e99-485f-8b5c-ca8ec5177668/volumes/kubernetes.io~projected/kube-api-access-vv9dt:{mountpoint:/var/lib/kubelet/pods/27ccca21-5e99-485f-8b5c-ca8ec5177668/volumes/kubernetes.io~projected/kube-api-access-vv9dt major:0 minor:225 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/29b8bd1a-beeb-495c-9f61-e3326866a1d7/volumes/kubernetes.io~projected/kube-api-access-vj8vx:{mountpoint:/var/lib/kubelet/pods/29b8bd1a-beeb-495c-9f61-e3326866a1d7/volumes/kubernetes.io~projected/kube-api-access-vj8vx major:0 minor:223 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2fa7d24b-67b0-400f-94d4-903feb268782/volumes/kubernetes.io~projected/kube-api-access-ssd98:{mountpoint:/var/lib/kubelet/pods/2fa7d24b-67b0-400f-94d4-903feb268782/volumes/kubernetes.io~projected/kube-api-access-ssd98 major:0 minor:51 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3af6ccf2-97f1-4598-b8ef-12eac377133b/volumes/kubernetes.io~projected/kube-api-access-9xfx9:{mountpoint:/var/lib/kubelet/pods/3af6ccf2-97f1-4598-b8ef-12eac377133b/volumes/kubernetes.io~projected/kube-api-access-9xfx9 major:0 minor:218 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4cf4abe6-da2b-41b6-9caa-9659d4f750fb/volumes/kubernetes.io~projected/kube-api-access-rtfn9:{mountpoint:/var/lib/kubelet/pods/4cf4abe6-da2b-41b6-9caa-9659d4f750fb/volumes/kubernetes.io~projected/kube-api-access-rtfn9 major:0 minor:226 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4cf4abe6-da2b-41b6-9caa-9659d4f750fb/volumes/kubernetes.io~secret/default-certificate:{mountpoint:/var/lib/kubelet/pods/4cf4abe6-da2b-41b6-9caa-9659d4f750fb/volumes/kubernetes.io~secret/default-certificate major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4cf4abe6-da2b-41b6-9caa-9659d4f750fb/volumes/kubernetes.io~secret/stats-auth:{mountpoint:/var/lib/kubelet/pods/4cf4abe6-da2b-41b6-9caa-9659d4f750fb/volumes/kubernetes.io~secret/stats-auth major:0 minor:205 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/58f7cf25-f5d0-4481-a927-4a745b692f7a/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/58f7cf25-f5d0-4481-a927-4a745b692f7a/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:227 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/58f7cf25-f5d0-4481-a927-4a745b692f7a/volumes/kubernetes.io~projected/kube-api-access-sx9pp:{mountpoint:/var/lib/kubelet/pods/58f7cf25-f5d0-4481-a927-4a745b692f7a/volumes/kubernetes.io~projected/kube-api-access-sx9pp major:0 minor:215 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/58f7cf25-f5d0-4481-a927-4a745b692f7a/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/58f7cf25-f5d0-4481-a927-4a745b692f7a/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/58f7cf25-f5d0-4481-a927-4a745b692f7a/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/58f7cf25-f5d0-4481-a927-4a745b692f7a/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6382dfb5-a363-47a2-bc79-0b96f7bec0c3/volumes/kubernetes.io~projected/kube-api-access-qg8hn:{mountpoint:/var/lib/kubelet/pods/6382dfb5-a363-47a2-bc79-0b96f7bec0c3/volumes/kubernetes.io~projected/kube-api-access-qg8hn major:0 minor:224 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6382dfb5-a363-47a2-bc79-0b96f7bec0c3/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/6382dfb5-a363-47a2-bc79-0b96f7bec0c3/volumes/kubernetes.io~secret/serving-cert major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/64ae3ff5-44eb-456b-948a-c1a8320e6b07/volumes/kubernetes.io~projected/kube-api-access-cfwlk:{mountpoint:/var/lib/kubelet/pods/64ae3ff5-44eb-456b-948a-c1a8320e6b07/volumes/kubernetes.io~projected/kube-api-access-cfwlk major:0 minor:217 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7728abfe-1747-422e-b8e2-c91c9de83d09/volumes/kubernetes.io~projected/kube-api-access-jqmnp:{mountpoint:/var/lib/kubelet/pods/7728abfe-1747-422e-b8e2-c91c9de83d09/volumes/kubernetes.io~projected/kube-api-access-jqmnp major:0 minor:287 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/840aaae0-a215-4ccd-b420-d6f5723b65ae/volumes/kubernetes.io~projected/kube-api-access-vsfpv:{mountpoint:/var/lib/kubelet/pods/840aaae0-a215-4ccd-b420-d6f5723b65ae/volumes/kubernetes.io~projected/kube-api-access-vsfpv major:0 minor:220 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/840aaae0-a215-4ccd-b420-d6f5723b65ae/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/840aaae0-a215-4ccd-b420-d6f5723b65ae/volumes/kubernetes.io~secret/serving-cert major:0 minor:209 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/953b9446-c289-48ca-b854-a14c70e5971b/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/953b9446-c289-48ca-b854-a14c70e5971b/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:52 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/953b9446-c289-48ca-b854-a14c70e5971b/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/953b9446-c289-48ca-b854-a14c70e5971b/volumes/kubernetes.io~empty-dir/tmp major:0 minor:53 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/953b9446-c289-48ca-b854-a14c70e5971b/volumes/kubernetes.io~projected/kube-api-access-gtt94:{mountpoint:/var/lib/kubelet/pods/953b9446-c289-48ca-b854-a14c70e5971b/volumes/kubernetes.io~projected/kube-api-access-gtt94 major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9700c98a-b858-45a5-9025-8d75f3c14044/volumes/kubernetes.io~projected/kube-api-access-5xn7l:{mountpoint:/var/lib/kubelet/pods/9700c98a-b858-45a5-9025-8d75f3c14044/volumes/kubernetes.io~projected/kube-api-access-5xn7l major:0 minor:221 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9700c98a-b858-45a5-9025-8d75f3c14044/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/9700c98a-b858-45a5-9025-8d75f3c14044/volumes/kubernetes.io~secret/cert major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/97da2086-b00d-4f03-b502-56de5ee3d90d/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/97da2086-b00d-4f03-b502-56de5ee3d90d/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:282 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/99f800b6-a38c-46ca-8234-0dbd3610c8fd/volumes/kubernetes.io~projected/kube-api-access-mlhgb:{mountpoint:/var/lib/kubelet/pods/99f800b6-a38c-46ca-8234-0dbd3610c8fd/volumes/kubernetes.io~projected/kube-api-access-mlhgb major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a4f35692-894d-4d41-8b9f-3f5ba3c529b1/volumes/kubernetes.io~projected/kube-api-access-67888:{mountpoint:/var/lib/kubelet/pods/a4f35692-894d-4d41-8b9f-3f5ba3c529b1/volumes/kubernetes.io~projected/kube-api-access-67888 major:0 minor:211 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a78879b4-333e-4f1a-88bc-e96aee0e6d6b/volumes/kubernetes.io~projected/kube-api-access-sdj76:{mountpoint:/var/lib/kubelet/pods/a78879b4-333e-4f1a-88bc-e96aee0e6d6b/volumes/kubernetes.io~projected/kube-api-access-sdj76 major:0 minor:327 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a8e4c4db-197e-420c-9c60-4273cdf95377/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/a8e4c4db-197e-420c-9c60-4273cdf95377/volumes/kubernetes.io~secret/agent-certs major:0 minor:47 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/aa90814d-5b4d-4039-917d-76610b1f88e4/volumes/kubernetes.io~projected/kube-api-access-q9zsm:{mountpoint:/var/lib/kubelet/pods/aa90814d-5b4d-4039-917d-76610b1f88e4/volumes/kubernetes.io~projected/kube-api-access-q9zsm major:0 minor:340 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b22fe41c-cf52-4e97-81f1-2f64bcc43640/volumes/kubernetes.io~projected/kube-api-access-btgs8:{mountpoint:/var/lib/kubelet/pods/b22fe41c-cf52-4e97-81f1-2f64bcc43640/volumes/kubernetes.io~projected/kube-api-access-btgs8 major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/be5540b5-7b10-45b2-b40c-723efd9f1b01/volumes/kubernetes.io~projected/kube-api-access-45fmr:{mountpoint:/var/lib/kubelet/pods/be5540b5-7b10-45b2-b40c-723efd9f1b01/volumes/kubernetes.io~projected/kube-api-access-45fmr major:0 minor:214 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ec330d4f-2eea-476a-a7b8-60cd295fddd2/volumes/kubernetes.io~projected/kube-api-access-z4n58:{mountpoint:/var/lib/kubelet/pods/ec330d4f-2eea-476a-a7b8-60cd295fddd2/volumes/kubernetes.io~projected/kube-api-access-z4n58 major:0 minor:322 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ece07686-4d8e-4068-9fb2-5889046cb73b/volumes/kubernetes.io~projected/kube-api-access-2cxtv:{mountpoint:/var/lib/kubelet/pods/ece07686-4d8e-4068-9fb2-5889046cb73b/volumes/kubernetes.io~projected/kube-api-access-2cxtv major:0 minor:49 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f464ef41-45c9-4ff4-b410-e2d9d6cdbdf5/volumes/kubernetes.io~projected/kube-api-access-ztdxk:{mountpoint:/var/lib/kubelet/pods/f464ef41-45c9-4ff4-b410-e2d9d6cdbdf5/volumes/kubernetes.io~projected/kube-api-access-ztdxk major:0 minor:213 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f464ef41-45c9-4ff4-b410-e2d9d6cdbdf5/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/f464ef41-45c9-4ff4-b410-e2d9d6cdbdf5/volumes/kubernetes.io~secret/serving-cert major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f8b33ecc-59a4-4680-9521-299648ea5055/volumes/kubernetes.io~projected/kube-api-access-7p9rj:{mountpoint:/var/lib/kubelet/pods/f8b33ecc-59a4-4680-9521-299648ea5055/volumes/kubernetes.io~projected/kube-api-access-7p9rj major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f8eb4b4c-8225-436b-b1e8-7775414a1ca3/volumes/kubernetes.io~projected/kube-api-access-k9l2q:{mountpoint:/var/lib/kubelet/pods/f8eb4b4c-8225-436b-b1e8-7775414a1ca3/volumes/kubernetes.io~projected/kube-api-access-k9l2q major:0 minor:212 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/c5a59c1843d5801336bb0fc2495b874458402ace607904d3078b702f312c03fd/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/27a6ffba02a3934f9cb309253dbfce3e9861aadb0c2a2ec1a0478fc8143ed0ae/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/6d974eed1b1429964b4d408c7949659cc9e00a7658159f0ea06562f407598ded/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-109:{mountpoint:/var/lib/containers/storage/overlay/a59b5f95bcbf157842a5afb53bacf947dbe0e592d6647a03f816400ca726064e/merged major:0 minor:109 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/4e0940b1b77ecee7c2888209fe7ce85526b6f4a492d380b51afd6e44787b6e5c/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/a52f5b83333cb6ccfaea845633a978c8b6b8fced2d489cbbf94019fc6edfb862/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/6bf5fc454fa4ed25ec1c2892fe0a769873fb5ea45efeb61f58e3dfc87308dd85/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/739d358caf7eeb6e9c67926e6ce39b789d13581ab05679f98a6a3dd1c7360023/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/6c1a1c5913fd915cd17564faf5b3918dc61fc947a3a43bcac89412ce210eeb60/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/c5dc08a9eec46b0f76cf5b9f5c5cf70511b69bd25343a3e670e4984044415336/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-139:{mountpoint:/var/lib/containers/storage/overlay/cfce2d5d51b77e8b1c4e43b2111c439f49e358058323b01fd851d38ff89ac798/merged major:0 minor:139 fsType:overlay blockSize:0} overlay_0-147:{mountpoint:/var/lib/containers/storage/overlay/51f96a627aa55338140c9c78039aa4e295635f028d447a94c0c0359855e87a7a/merged major:0 minor:147 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/ca2b7e5e9b7726300606823a6327b84f2aa6961c648ed621a24f81b8d0cd2bb5/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/9606d5afd5e0a8ff2eafbffa7654e026f187ec31c8436a453bba5d026d8affe4/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/ccd2d019aae3072e1354c99c18a91487d52d34cc0986098c91ea9aa3b176c2ff/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/a1b94b39d822ee4053105ec353faaa73ba8c79e9849cb7421eeaa62c95ddf8cd/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/221283a5df77c062307bd512de0637f0c74eccda1291c51396e413ef20ac13f4/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/b67bb36286fb83d3ac56abb9f43e7db68d139e87b8a21e317f4298df16d61f11/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/97df80e86cd59247e3a1412a38e302da59419c91370a957077c06137289bd475/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/a301049b6b4cd24a668bfd1c3b4d8f1fb0baf2fa3c8af24ae3bb8354e4acda6d/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/2ec51804cf61b1204d694a39e17f9130fcf8147affcfd9b716cbeca37cfb4907/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-248:{mountpoint:/var/lib/containers/storage/overlay/f63b7af5946137b1655b999ecf2ef638339adfba66ec7a00bff79eaf28fea386/merged major:0 minor:248 fsType:overlay blockSize:0} overlay_0-250:{mountpoint:/var/lib/containers/storage/overlay/8862d958d819c14117559e784fc8bf67a90d73475a88cd0b6650b9bfb20ddf0d/merged major:0 minor:250 fsType:overlay blockSize:0} overlay_0-252:{mountpoint:/var/lib/containers/storage/overlay/6374bd9911546195b5fa3185b2bb504357d06caec8be467157eee329dc8e0b66/merged major:0 minor:252 fsType:overlay blockSize:0} overlay_0-254:{mountpoint:/var/lib/containers/storage/overlay/4b86df0013ef9a498513f38465ca8bd948b3a65ce70092a25797c9439b7e98e0/merged major:0 minor:254 fsType:overlay blockSize:0} overlay_0-256:{mountpoint:/var/lib/containers/storage/overlay/3d8a37517c5d46fee9724068d6969f9a1b8502ddd82b5ddc5eb7e6211a0fb7ed/merged major:0 minor:256 fsType:overlay blockSize:0} overlay_0-258:{mountpoint:/var/lib/containers/storage/overlay/2f154824f0ae5513bd1574388ef1fa20ce07fb63ae001384b9f5689cdc40e7c6/merged major:0 minor:258 fsType:overlay blockSize:0} overlay_0-260:{mountpoint:/var/lib/containers/storage/overlay/b30748b7ec7d95b435feaa1a15af09e49abc363ebf845a5743894d7cf4a3858f/merged major:0 minor:260 fsType:overlay blockSize:0} overlay_0-261:{mountpoint:/var/lib/containers/storage/overlay/034e8c19d07dc96d8b102f1bad85587afd4656ea37a65b35e4baff1e27f7198d/merged major:0 minor:261 fsType:overlay blockSize:0} overlay_0-272:{mountpoint:/var/lib/containers/storage/overlay/dbb53566ca64ece76e6c242dacf04b302aeb353cba371aba978146fee8065a79/merged major:0 minor:272 fsType:overlay blockSize:0} overlay_0-274:{mountpoint:/var/lib/containers/storage/overlay/4e5076419a6469c2d07da8ec7570a5ca3a06341ef4272006575b140ba99ca501/merged major:0 minor:274 fsType:overlay blockSize:0} overlay_0-276:{mountpoint:/var/lib/containers/storage/overlay/cf40c9f09dae840b1fd18beaf48767bc8b768a3bb980f0235c3992c54a3d7845/merged major:0 minor:276 fsType:overlay blockSize:0} overlay_0-278:{mountpoint:/var/lib/containers/storage/overlay/fc53004f5f9cf024baf2b483b85219c3371f7b1d924f07b09c6c1cf9ab4a40ba/merged major:0 minor:278 fsType:overlay blockSize:0} overlay_0-280:{mountpoint:/var/lib/containers/storage/overlay/aa7b964c275b5f27463857389ebeb3934c91722b1c9f06933232f0dab174b47a/merged major:0 minor:280 fsType:overlay blockSize:0} overlay_0-292:{mountpoint:/var/lib/containers/storage/overlay/e7a47c6d1886f52770d6a460e99faa088e4843677ebed5f959dedb6f12c7153f/merged major:0 minor:292 fsType:overlay blockSize:0} overlay_0-294:{mountpoint:/var/lib/containers/storage/overlay/28146a8f463cf5caf74361ad1082a52a6109c7719e85e0933640d74cd5f6b2c3/merged major:0 minor:294 fsType:overlay blockSize:0} overlay_0-296:{mountpoint:/var/lib/containers/storage/overlay/470870daf2920f507d5209343e49635ab08c70f0a49a028dd53a739330ba7ed1/merged major:0 minor:296 fsType:overlay blockSize:0} overlay_0-298:{mountpoint:/var/lib/containers/storage/overlay/2f7aaada216196a1e8ce7ac1b5e5db7f7db86548188976f0f9aa363dc82dff2b/merged major:0 minor:298 fsType:overlay blockSize:0} overlay_0-304:{mountpoint:/var/lib/containers/storage/overlay/f822ba921348766b43ec77baeaec80294859f6244494b89415679646fdc52f1d/merged major:0 minor:304 fsType:overlay blockSize:0} overlay_0-310:{mountpoint:/var/lib/containers/storage/overlay/c2356e67e8430c51cd71c9fba53a3ac71d441a09776df8b41c79ede3516e58f6/merged major:0 minor:310 fsType:overlay blockSize:0} overlay_0-312:{mountpoint:/var/lib/containers/storage/overlay/bd83ec5f33a9a7971c817709c3cfb41c5c3efb8a9b822b47201735ac9ea1b405/merged major:0 minor:312 fsType:overlay blockSize:0} overlay_0-330:{mountpoint:/var/lib/containers/storage/overlay/9d5660bc59838f64518ac5a590de8fa016ffcc711dfb57a6bd5ca0535504d13d/merged major:0 minor:330 fsType:overlay blockSize:0} overlay_0-332:{mountpoint:/var/lib/containers/storage/overlay/e436e3cf38a1d73233c22e57dbd522ba3fa9d27ead9c9e11edcdfe0eac7ad994/merged major:0 minor:332 fsType:overlay blockSize:0} overlay_0-338:{mountpoint:/var/lib/containers/storage/overlay/2f9fcf2c269a77947c64c61e2c15e69e76bd6c9122fb6e0b926b37051d7dcc56/merged major:0 minor:338 fsType:overlay blockSize:0} overlay_0-345:{mountpoint:/var/lib/containers/storage/overlay/d8f958ee526f27c59b113f8227bd26f0c8dc80c3618cd41dd2aec2287a904b48/merged major:0 minor:345 fsType:overlay blockSize:0} overlay_0-347:{mountpoint:/var/lib/containers/storage/overlay/641cd83884e5c2e3af93cedc313c2644df3bd7f81c1a7f1bfa1c9c3171d439de/merged major:0 minor:347 fsType:overlay blockSize:0} overlay_0-355:{mountpoint:/var/lib/containers/storage/overlay/abe7404316815682c737f45f2a20d39427d9165a0f3230e9ea7c9eaba487b805/merged major:0 minor:355 fsType:overlay blockSize:0} overlay_0-66:{mountpoint:/var/lib/containers/storage/overlay/db811de94386c3000f58627e826a11a229f4e21d05bc100fe83f674aadd39344/merged major:0 minor:66 fsType:overlay blockSize:0} overlay_0-68:{mountpoint:/var/lib/containers/storage/overlay/1e82c93c54770c80e717bea1c294ed753287104feb99ca475623e7fd246c21d0/merged major:0 minor:68 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/e16032db80583e2541554923b69e08eec938b162129376c0b3a31044a70b0cf8/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-72:{mountpoint:/var/lib/containers/storage/overlay/2e01e0fc2b62bc45ae7531099dcfd2137fb46f8ce8063be45b83d7b86b8e70df/merged major:0 minor:72 fsType:overlay blockSize:0} overlay_0-76:{mountpoint:/var/lib/containers/storage/overlay/210ceac2daf9274820c6b32d3f2455f310bf942cfc7d078c1951412dd570743c/merged major:0 minor:76 fsType:overlay blockSize:0} overlay_0-80:{mountpoint:/var/lib/containers/storage/overlay/92cc99813dd5b0a0cd58976336aa1b034ab441d275761b084ebeff5bb1126f9c/merged major:0 minor:80 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/3f1001be5e5a55d02f2df209d49aed5506905efedc99ed33e8d4fccd7c5ab92c/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/a8e764192ab41b2061a875a31938816fc8a4d6c6898bb1cbf78f1107691c065c/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/2a7b2326d9385771bc5bbd8e8ef385a1d8dc2d1aed7a5c3f2379d8dd4d850bf9/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/c5f40da9f2b2b3e0a5cc394a121b76ea2c6f86bba1680741de870682621d17ed/merged major:0 minor:96 fsType:overlay blockSize:0}] Oct 30 23:41:27.024453 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:27.023757 4927 manager.go:217] Machine: {Timestamp:2026-10-30 23:41:27.02327484 +0000 UTC m=+0.059605652 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3059135 MemoryCapacity:33164337152 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2acb4b4090c41d8b0bad096078018e SystemUUID:ec2acb4b-4090-c41d-8b0b-ad096078018e BootID:70c52820-3dd5-4ff4-ab01-ce728104e9a7 Filesystems:[{Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0d4810de-8bd2-4270-b907-341d9f840bb8/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/f464ef41-45c9-4ff4-b410-e2d9d6cdbdf5/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:207 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-274 DeviceMajor:0 DeviceMinor:274 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b22fe41c-cf52-4e97-81f1-2f64bcc43640/volumes/kubernetes.io~projected/kube-api-access-btgs8 DeviceMajor:0 DeviceMinor:58 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/953b9446-c289-48ca-b854-a14c70e5971b/volumes/kubernetes.io~projected/kube-api-access-gtt94 DeviceMajor:0 DeviceMinor:61 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-139 DeviceMajor:0 DeviceMinor:139 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0fd7a1669bcc37cc84500ce975c3ac8b58e4c1e03de0b5f9245530d20c9dc527/userdata/shm DeviceMajor:0 DeviceMinor:230 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/64ae3ff5-44eb-456b-948a-c1a8320e6b07/volumes/kubernetes.io~projected/kube-api-access-cfwlk DeviceMajor:0 DeviceMinor:217 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/9700c98a-b858-45a5-9025-8d75f3c14044/volumes/kubernetes.io~projected/kube-api-access-5xn7l DeviceMajor:0 DeviceMinor:221 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/97da2086-b00d-4f03-b502-56de5ee3d90d/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:282 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b4ecc76140e7d1ce05b22f67284d328e2dc8e4efa8a8ec078a459ef9338deee9/userdata/shm DeviceMajor:0 DeviceMinor:290 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/a8e4c4db-197e-420c-9c60-4273cdf95377/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:47 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c80c7448baf9962aeb89efcadbfe35a76110a83280e99ce6a7d38c1eb89cac18/userdata/shm DeviceMajor:0 DeviceMinor:82 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-276 DeviceMajor:0 DeviceMinor:276 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0d4810de-8bd2-4270-b907-341d9f840bb8/volumes/kubernetes.io~projected/kube-api-access-48m7s DeviceMajor:0 DeviceMinor:60 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/953b9446-c289-48ca-b854-a14c70e5971b/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:53 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-76 DeviceMajor:0 DeviceMinor:76 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-345 DeviceMajor:0 DeviceMinor:345 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/020dec11112038ed834b49f31a2d35496a872cbc95e5b9692e8bf311a59c5260/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/17461b46-7ebf-49b7-afaf-12e20e3994a3/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:208 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/17461b46-7ebf-49b7-afaf-12e20e3994a3/volumes/kubernetes.io~projected/kube-api-access-xkthh DeviceMajor:0 DeviceMinor:222 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/ec330d4f-2eea-476a-a7b8-60cd295fddd2/volumes/kubernetes.io~projected/kube-api-access-z4n58 DeviceMajor:0 DeviceMinor:322 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-66 DeviceMajor:0 DeviceMinor:66 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-338 DeviceMajor:0 DeviceMinor:338 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/154e4407f570053087b1f80f38a62a9607d8146fbbadeb07629573256de48e7e/userdata/shm DeviceMajor:0 DeviceMinor:56 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-72 DeviceMajor:0 DeviceMinor:72 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f464ef41-45c9-4ff4-b410-e2d9d6cdbdf5/volumes/kubernetes.io~projected/kube-api-access-ztdxk DeviceMajor:0 DeviceMinor:213 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/6382dfb5-a363-47a2-bc79-0b96f7bec0c3/volumes/kubernetes.io~projected/kube-api-access-qg8hn DeviceMajor:0 DeviceMinor:224 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-248 DeviceMajor:0 DeviceMinor:248 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/bbfd66c255357d0b1ae9f531564af260a8d8bd00c26ccfe7f763b2cdb3afeaec/userdata/shm DeviceMajor:0 DeviceMinor:328 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c374b94e0a85533ff7652b5def95a433a127c7f26a0f314e854e8962450dae40/userdata/shm DeviceMajor:0 DeviceMinor:74 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-330 DeviceMajor:0 DeviceMinor:330 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4cf4abe6-da2b-41b6-9caa-9659d4f750fb/volumes/kubernetes.io~projected/kube-api-access-rtfn9 DeviceMajor:0 DeviceMinor:226 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-258 DeviceMajor:0 DeviceMinor:258 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4357c8b1b1cd7c31d12ef70138214a17b84944e2afc3d6d78932c37e9743b036/userdata/shm DeviceMajor:0 DeviceMinor:62 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/69619f4a1d594665a6b7373c4f0480dec83db882660369456758935c0ee3aebd/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/be5540b5-7b10-45b2-b40c-723efd9f1b01/volumes/kubernetes.io~projected/kube-api-access-45fmr DeviceMajor:0 DeviceMinor:214 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-296 DeviceMajor:0 DeviceMinor:296 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f8b33ecc-59a4-4680-9521-299648ea5055/volumes/kubernetes.io~projected/kube-api-access-7p9rj DeviceMajor:0 DeviceMinor:59 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/01cae760460a3c29096c2785faece01841f55c4c9c9de869c85e2ef7134b502c/userdata/shm DeviceMajor:0 DeviceMinor:234 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/ece07686-4d8e-4068-9fb2-5889046cb73b/volumes/kubernetes.io~projected/kube-api-access-2cxtv DeviceMajor:0 DeviceMinor:49 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/145dd323-9370-4b9d-a96e-8b93d0a47a7c/volumes/kubernetes.io~projected/kube-api-access-nsggg DeviceMajor:0 DeviceMinor:48 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/953b9446-c289-48ca-b854-a14c70e5971b/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:52 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/27ccca21-5e99-485f-8b5c-ca8ec5177668/volumes/kubernetes.io~projected/kube-api-access-vv9dt DeviceMajor:0 DeviceMinor:225 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-250 DeviceMajor:0 DeviceMinor:250 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0a3626cfde5478d63a59c0432c8cac480a568eb5bcbdf0fcbd33e342c40e75dd/userdata/shm DeviceMajor:0 DeviceMinor:86 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/840aaae0-a215-4ccd-b420-d6f5723b65ae/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:209 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/3af6ccf2-97f1-4598-b8ef-12eac377133b/volumes/kubernetes.io~projected/kube-api-access-9xfx9 DeviceMajor:0 DeviceMinor:218 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/719c8f74b4146c489029a3cae27c9f4cc9d6c84b8ff50c6abfc15a23ef334941/userdata/shm DeviceMajor:0 DeviceMinor:246 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-280 DeviceMajor:0 DeviceMinor:280 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-294 DeviceMajor:0 DeviceMinor:294 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ae3a0d475b2be1b25689edf2744177911624228a66bdbfc2ba94c3dccad12b36/userdata/shm DeviceMajor:0 DeviceMinor:240 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-256 DeviceMajor:0 DeviceMinor:256 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-260 DeviceMajor:0 DeviceMinor:260 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/58f7cf25-f5d0-4481-a927-4a745b692f7a/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:204 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/840aaae0-a215-4ccd-b420-d6f5723b65ae/volumes/kubernetes.io~projected/kube-api-access-vsfpv DeviceMajor:0 DeviceMinor:220 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a85a7532289c5eee82ff2b7f93b6834d6ceba975a039ef17c247d87a33699c05/userdata/shm DeviceMajor:0 DeviceMinor:55 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d353d4d879e678a52dad33597146b7e44151f5f2886f5c2c28a1cdddea04459b/userdata/shm DeviceMajor:0 DeviceMinor:78 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4cf4abe6-da2b-41b6-9caa-9659d4f750fb/volumes/kubernetes.io~secret/default-certificate DeviceMajor:0 DeviceMinor:197 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6b58028156e6606c849a36e8b17c9e190dc04bfc8a69a1f24a667e08b4a26210/userdata/shm DeviceMajor:0 DeviceMinor:237 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/17461b46-7ebf-49b7-afaf-12e20e3994a3/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:210 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-310 DeviceMajor:0 DeviceMinor:310 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/fe922c3105879e4776509bda85f73cd3fc9e6c078679a53ab53a6d6ff57f44b2/userdata/shm DeviceMajor:0 DeviceMinor:348 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/58f7cf25-f5d0-4481-a927-4a745b692f7a/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:227 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2cd1d530b1d2e4fbf2016043d891c3f05ed64bc1e150f43ac461b47dd0c04439/userdata/shm DeviceMajor:0 DeviceMinor:232 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-261 DeviceMajor:0 DeviceMinor:261 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-332 DeviceMajor:0 DeviceMinor:332 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8738eb3d4e62b725466ec53a2c859cd4d77af3b229836134893039fd90cf182e/userdata/shm DeviceMajor:0 DeviceMinor:243 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-304 DeviceMajor:0 DeviceMinor:304 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-68 DeviceMajor:0 DeviceMinor:68 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f8eb4b4c-8225-436b-b1e8-7775414a1ca3/volumes/kubernetes.io~projected/kube-api-access-k9l2q DeviceMajor:0 DeviceMinor:212 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/03211c6a-9761-419c-839b-21ca413d5bb2/volumes/kubernetes.io~projected/kube-api-access-5x8x6 DeviceMajor:0 DeviceMinor:219 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-298 DeviceMajor:0 DeviceMinor:298 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/29b8bd1a-beeb-495c-9f61-e3326866a1d7/volumes/kubernetes.io~projected/kube-api-access-vj8vx DeviceMajor:0 DeviceMinor:223 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/722d14c37f4ca9c2f6ea9ad958e59ca48f6fb4bb8ffc975ccc53a0d713fcd4fc/userdata/shm DeviceMajor:0 DeviceMinor:228 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4d24e3019382f392346762ad0cdd801460b4a56c586d5f30a283293c21f78d5c/userdata/shm DeviceMajor:0 DeviceMinor:236 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/7728abfe-1747-422e-b8e2-c91c9de83d09/volumes/kubernetes.io~projected/kube-api-access-jqmnp DeviceMajor:0 DeviceMinor:287 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/2fa7d24b-67b0-400f-94d4-903feb268782/volumes/kubernetes.io~projected/kube-api-access-ssd98 DeviceMajor:0 DeviceMinor:51 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/4cf4abe6-da2b-41b6-9caa-9659d4f750fb/volumes/kubernetes.io~secret/stats-auth DeviceMajor:0 DeviceMinor:205 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c9010557cbcc988d15e3e177b0de68e23621b842a825a094cc2557634101e5c4/userdata/shm DeviceMajor:0 DeviceMinor:241 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/a78879b4-333e-4f1a-88bc-e96aee0e6d6b/volumes/kubernetes.io~projected/kube-api-access-sdj76 DeviceMajor:0 DeviceMinor:327 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-347 DeviceMajor:0 DeviceMinor:347 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-147 DeviceMajor:0 DeviceMinor:147 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a4f35692-894d-4d41-8b9f-3f5ba3c529b1/volumes/kubernetes.io~projected/kube-api-access-67888 DeviceMajor:0 DeviceMinor:211 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/10774606-8e0c-460b-bd5f-4cb6d121b30b/volumes/kubernetes.io~projected/kube-api-access-n65gn DeviceMajor:0 DeviceMinor:216 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-278 DeviceMajor:0 DeviceMinor:278 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-292 DeviceMajor:0 DeviceMinor:292 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/10774606-8e0c-460b-bd5f-4cb6d121b30b/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:201 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-254 DeviceMajor:0 DeviceMinor:254 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a755ef97a7c53aaa8735a70265d8b3996ca96b08bf812706d173efd03f99c3bc/userdata/shm DeviceMajor:0 DeviceMinor:288 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/0796261f-298e-4cbe-9b3e-b3dcf4528e07/volumes/kubernetes.io~projected/kube-api-access-25rtq DeviceMajor:0 DeviceMinor:50 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/58f7cf25-f5d0-4481-a927-4a745b692f7a/volumes/kubernetes.io~projected/kube-api-access-sx9pp DeviceMajor:0 DeviceMinor:215 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-272 DeviceMajor:0 DeviceMinor:272 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-355 DeviceMajor:0 DeviceMinor:355 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1dda77b536c5cf8fe095e91e3a2da765111f2116770dcb972dc5b0b76134b542/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-109 DeviceMajor:0 DeviceMinor:109 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9700c98a-b858-45a5-9025-8d75f3c14044/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:45 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/0d4810de-8bd2-4270-b907-341d9f840bb8/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:54 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/var/lib/kubelet/pods/17461b46-7ebf-49b7-afaf-12e20e3994a3/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:206 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/99f800b6-a38c-46ca-8234-0dbd3610c8fd/volumes/kubernetes.io~projected/kube-api-access-mlhgb DeviceMajor:0 DeviceMinor:57 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/58f7cf25-f5d0-4481-a927-4a745b692f7a/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:202 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/aa90814d-5b4d-4039-917d-76610b1f88e4/volumes/kubernetes.io~projected/kube-api-access-q9zsm DeviceMajor:0 DeviceMinor:340 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-252 DeviceMajor:0 DeviceMinor:252 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-80 DeviceMajor:0 DeviceMinor:80 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6382dfb5-a363-47a2-bc79-0b96f7bec0c3/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:203 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-312 DeviceMajor:0 DeviceMinor:312 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:01cae760460a3c2 MacAddress:52:f7:69:ee:c5:06 Speed:10000 Mtu:8901} {Name:0fd7a1669bcc37c MacAddress:7e:69:d0:91:e5:a3 Speed:10000 Mtu:8901} {Name:2cd1d530b1d2e4f MacAddress:1e:bc:cd:fb:b7:4a Speed:10000 Mtu:8901} {Name:4d24e3019382f39 MacAddress:a2:a2:98:56:34:41 Speed:10000 Mtu:8901} {Name:6b58028156e6606 MacAddress:be:2d:58:6e:74:dd Speed:10000 Mtu:8901} {Name:719c8f74b4146c4 MacAddress:6e:34:42:4b:3a:ba Speed:10000 Mtu:8901} {Name:722d14c37f4ca9c MacAddress:4a:54:04:8e:7d:9d Speed:10000 Mtu:8901} {Name:8738eb3d4e62b72 MacAddress:c2:cb:09:a7:0f:24 Speed:10000 Mtu:8901} {Name:a755ef97a7c53aa MacAddress:2a:3f:ce:4b:28:ec Speed:10000 Mtu:8901} {Name:ae3a0d475b2be1b MacAddress:1e:4e:fa:d3:5c:a0 Speed:10000 Mtu:8901} {Name:b4ecc76140e7d1c MacAddress:82:6d:79:49:ba:22 Speed:10000 Mtu:8901} {Name:bbfd66c255357d0 MacAddress:d2:14:16:f5:9b:a5 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:06:ff:d9:eb:dd:69 Speed:0 Mtu:9001} {Name:br-int MacAddress:fa:3b:b3:6b:5e:8e Speed:0 Mtu:8901} {Name:c9010557cbcc988 MacAddress:12:67:ac:0e:8a:82 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:06:ff:d9:eb:dd:69 Speed:0 Mtu:9001} {Name:fe922c3105879e4 MacAddress:d2:b8:29:f1:4b:4b Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:a2:a6:28:d4:c8:a4 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:84:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:56:a4:ef:fa:f3:68 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164337152 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 23:41:27.024453 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:27.024107 4927 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 23:41:27.024453 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:27.024149 4927 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 23:41:27.024453 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:27.024429 4927 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 23:41:27.024698 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:27.024451 4927 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-141-180.us-west-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 23:41:27.024698 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:27.024593 4927 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 23:41:27.024698 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:27.024601 4927 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 23:41:27.024698 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:27.024613 4927 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 23:41:27.024698 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:27.024626 4927 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 23:41:27.024698 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:27.024643 4927 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 23:41:27.024860 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:27.024799 4927 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 23:41:27.024918 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:27.024907 4927 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 23:41:27.024950 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:27.024946 4927 kubelet.go:493] "Attempting to sync node with API server" Oct 30 23:41:27.024978 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:27.024959 4927 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 23:41:27.024978 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:27.024972 4927 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 23:41:27.025027 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:27.024980 4927 kubelet.go:405] "Adding apiserver pod source" Oct 30 23:41:27.025027 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:27.024988 4927 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 23:41:27.026870 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:27.026557 4927 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 23:41:27.026870 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:27.026578 4927 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 23:41:27.027171 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:27.027155 4927 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 23:41:27.038798 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:27.038780 4927 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 23:41:27.038798 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:27.038795 4927 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 23:41:27.038918 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:27.038890 4927 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Oct 30 23:41:27.039078 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:27.039067 4927 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 23:41:27.039114 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:27.039096 4927 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 23:41:27.039203 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:27.039194 4927 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 23:41:27.039235 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:27.039208 4927 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 23:41:27.039235 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:27.039215 4927 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 23:41:27.039235 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:27.039221 4927 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 23:41:27.039235 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:27.039226 4927 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 23:41:27.039235 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:27.039232 4927 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 23:41:27.039235 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:27.039238 4927 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 23:41:27.039478 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:27.039249 4927 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 23:41:27.039478 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:27.039256 4927 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 23:41:27.039478 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:27.039263 4927 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 23:41:27.039478 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:27.039275 4927 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 23:41:27.039478 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:27.039286 4927 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 23:41:27.039478 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:27.039311 4927 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 23:41:27.039478 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:27.039317 4927 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 23:41:27.042794 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:27.042779 4927 server.go:1263] "Started kubelet" Oct 30 23:41:27.043273 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:27.042939 4927 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 23:41:27.043273 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:27.042982 4927 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 23:41:27.043273 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:27.043071 4927 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 23:41:27.043517 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:27.043395 4927 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 23:41:27.043326 ip-10-0-141-180 systemd[1]: Started Kubernetes Kubelet. Oct 30 23:41:27.044234 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:27.044187 4927 server.go:317] "Adding debug handlers to kubelet server" Oct 30 23:41:27.049803 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:27.046993 4927 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 23:41:27.049803 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:27.047356 4927 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 23:41:27.049803 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:27.047459 4927 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 23:41:27.049803 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:27.047473 4927 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 23:41:27.049803 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:27.047595 4927 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 23:41:27.049803 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:27.049266 4927 factory.go:55] Registering systemd factory Oct 30 23:41:27.049803 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:27.049282 4927 factory.go:223] Registration of the systemd container factory successfully Oct 30 23:41:27.049803 ip-10-0-141-180 kubenswrapper[4927]: E1030 23:41:27.049260 4927 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-180.us-west-2.compute.internal\" not found" Oct 30 23:41:27.049803 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:27.049317 4927 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 23:35:46 +0000 UTC" deadline="2028-06-06 20:47:28.418211267 +0000 UTC" Oct 30 23:41:27.049803 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:27.049388 4927 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14037h6m1.368827495s" Oct 30 23:41:27.051097 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:27.049974 4927 factory.go:153] Registering CRI-O factory Oct 30 23:41:27.051097 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:27.050000 4927 factory.go:223] Registration of the crio container factory successfully Oct 30 23:41:27.051097 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:27.050056 4927 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 23:41:27.051097 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:27.050076 4927 factory.go:103] Registering Raw factory Oct 30 23:41:27.051097 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:27.050094 4927 manager.go:1199] Started watching for new ooms in manager Oct 30 23:41:27.051423 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:27.051371 4927 manager.go:319] Starting recovery of all containers Oct 30 23:41:27.059733 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:27.053211 4927 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0d4810de-8bd2-4270-b907-341d9f840bb8" volumeName="kubernetes.io/projected/0d4810de-8bd2-4270-b907-341d9f840bb8-kube-api-access-48m7s" seLinuxMountContext="" Oct 30 23:41:27.059733 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:27.053249 4927 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6382dfb5-a363-47a2-bc79-0b96f7bec0c3" volumeName="kubernetes.io/secret/6382dfb5-a363-47a2-bc79-0b96f7bec0c3-serving-cert" seLinuxMountContext="" Oct 30 23:41:27.059733 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:27.053264 4927 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="840aaae0-a215-4ccd-b420-d6f5723b65ae" volumeName="kubernetes.io/secret/840aaae0-a215-4ccd-b420-d6f5723b65ae-serving-cert" seLinuxMountContext="" Oct 30 23:41:27.059733 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:27.053278 4927 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b22fe41c-cf52-4e97-81f1-2f64bcc43640" volumeName="kubernetes.io/configmap/b22fe41c-cf52-4e97-81f1-2f64bcc43640-cni-binary-copy" seLinuxMountContext="" Oct 30 23:41:27.059733 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:27.053294 4927 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="17461b46-7ebf-49b7-afaf-12e20e3994a3" volumeName="kubernetes.io/configmap/17461b46-7ebf-49b7-afaf-12e20e3994a3-registry-certificates" seLinuxMountContext="" Oct 30 23:41:27.059733 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:27.053307 4927 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4cf4abe6-da2b-41b6-9caa-9659d4f750fb" volumeName="kubernetes.io/secret/4cf4abe6-da2b-41b6-9caa-9659d4f750fb-stats-auth" seLinuxMountContext="" Oct 30 23:41:27.059733 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:27.053318 4927 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="953b9446-c289-48ca-b854-a14c70e5971b" volumeName="kubernetes.io/projected/953b9446-c289-48ca-b854-a14c70e5971b-kube-api-access-gtt94" seLinuxMountContext="" Oct 30 23:41:27.059733 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:27.053344 4927 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9700c98a-b858-45a5-9025-8d75f3c14044" volumeName="kubernetes.io/secret/9700c98a-b858-45a5-9025-8d75f3c14044-cert" seLinuxMountContext="" Oct 30 23:41:27.059733 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:27.053359 4927 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b22fe41c-cf52-4e97-81f1-2f64bcc43640" volumeName="kubernetes.io/configmap/b22fe41c-cf52-4e97-81f1-2f64bcc43640-whereabouts-flatfile-configmap" seLinuxMountContext="" Oct 30 23:41:27.059733 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:27.053371 4927 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ece07686-4d8e-4068-9fb2-5889046cb73b" volumeName="kubernetes.io/projected/ece07686-4d8e-4068-9fb2-5889046cb73b-kube-api-access-2cxtv" seLinuxMountContext="" Oct 30 23:41:27.059733 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:27.053384 4927 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="10774606-8e0c-460b-bd5f-4cb6d121b30b" volumeName="kubernetes.io/secret/10774606-8e0c-460b-bd5f-4cb6d121b30b-serving-cert" seLinuxMountContext="" Oct 30 23:41:27.059733 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:27.053396 4927 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="10774606-8e0c-460b-bd5f-4cb6d121b30b" volumeName="kubernetes.io/empty-dir/10774606-8e0c-460b-bd5f-4cb6d121b30b-snapshots" seLinuxMountContext="" Oct 30 23:41:27.059733 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:27.053409 4927 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="10774606-8e0c-460b-bd5f-4cb6d121b30b" volumeName="kubernetes.io/projected/10774606-8e0c-460b-bd5f-4cb6d121b30b-kube-api-access-n65gn" seLinuxMountContext="" Oct 30 23:41:27.059733 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:27.053422 4927 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="17461b46-7ebf-49b7-afaf-12e20e3994a3" volumeName="kubernetes.io/configmap/17461b46-7ebf-49b7-afaf-12e20e3994a3-trusted-ca" seLinuxMountContext="" Oct 30 23:41:27.059733 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:27.053437 4927 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="17461b46-7ebf-49b7-afaf-12e20e3994a3" volumeName="kubernetes.io/projected/17461b46-7ebf-49b7-afaf-12e20e3994a3-bound-sa-token" seLinuxMountContext="" Oct 30 23:41:27.059733 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:27.053450 4927 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="17461b46-7ebf-49b7-afaf-12e20e3994a3" volumeName="kubernetes.io/secret/17461b46-7ebf-49b7-afaf-12e20e3994a3-image-registry-private-configuration" seLinuxMountContext="" Oct 30 23:41:27.059733 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:27.053463 4927 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="58f7cf25-f5d0-4481-a927-4a745b692f7a" volumeName="kubernetes.io/secret/58f7cf25-f5d0-4481-a927-4a745b692f7a-image-registry-private-configuration" seLinuxMountContext="" Oct 30 23:41:27.059733 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:27.053474 4927 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="aa90814d-5b4d-4039-917d-76610b1f88e4" volumeName="kubernetes.io/projected/aa90814d-5b4d-4039-917d-76610b1f88e4-kube-api-access-q9zsm" seLinuxMountContext="" Oct 30 23:41:27.059733 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:27.053486 4927 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0d4810de-8bd2-4270-b907-341d9f840bb8" volumeName="kubernetes.io/configmap/0d4810de-8bd2-4270-b907-341d9f840bb8-ovnkube-config" seLinuxMountContext="" Oct 30 23:41:27.059733 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:27.053497 4927 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="27ccca21-5e99-485f-8b5c-ca8ec5177668" volumeName="kubernetes.io/projected/27ccca21-5e99-485f-8b5c-ca8ec5177668-kube-api-access-vv9dt" seLinuxMountContext="" Oct 30 23:41:27.059733 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:27.053509 4927 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="58f7cf25-f5d0-4481-a927-4a745b692f7a" volumeName="kubernetes.io/secret/58f7cf25-f5d0-4481-a927-4a745b692f7a-installation-pull-secrets" seLinuxMountContext="" Oct 30 23:41:27.059733 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:27.053520 4927 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a78879b4-333e-4f1a-88bc-e96aee0e6d6b" volumeName="kubernetes.io/empty-dir/a78879b4-333e-4f1a-88bc-e96aee0e6d6b-archives-path" seLinuxMountContext="" Oct 30 23:41:27.059733 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:27.053532 4927 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b22fe41c-cf52-4e97-81f1-2f64bcc43640" volumeName="kubernetes.io/configmap/b22fe41c-cf52-4e97-81f1-2f64bcc43640-cni-sysctl-allowlist" seLinuxMountContext="" Oct 30 23:41:27.059733 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:27.053560 4927 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="be5540b5-7b10-45b2-b40c-723efd9f1b01" volumeName="kubernetes.io/projected/be5540b5-7b10-45b2-b40c-723efd9f1b01-kube-api-access-45fmr" seLinuxMountContext="" Oct 30 23:41:27.059733 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:27.053590 4927 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ece07686-4d8e-4068-9fb2-5889046cb73b" volumeName="kubernetes.io/configmap/ece07686-4d8e-4068-9fb2-5889046cb73b-iptables-alerter-script" seLinuxMountContext="" Oct 30 23:41:27.059733 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:27.053602 4927 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="17461b46-7ebf-49b7-afaf-12e20e3994a3" volumeName="kubernetes.io/empty-dir/17461b46-7ebf-49b7-afaf-12e20e3994a3-ca-trust-extracted" seLinuxMountContext="" Oct 30 23:41:27.059733 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:27.053642 4927 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="17461b46-7ebf-49b7-afaf-12e20e3994a3" volumeName="kubernetes.io/secret/17461b46-7ebf-49b7-afaf-12e20e3994a3-installation-pull-secrets" seLinuxMountContext="" Oct 30 23:41:27.059733 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:27.053659 4927 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f464ef41-45c9-4ff4-b410-e2d9d6cdbdf5" volumeName="kubernetes.io/configmap/f464ef41-45c9-4ff4-b410-e2d9d6cdbdf5-config" seLinuxMountContext="" Oct 30 23:41:27.059733 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:27.053693 4927 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="10774606-8e0c-460b-bd5f-4cb6d121b30b" volumeName="kubernetes.io/empty-dir/10774606-8e0c-460b-bd5f-4cb6d121b30b-tmp" seLinuxMountContext="" Oct 30 23:41:27.059733 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:27.053708 4927 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="58f7cf25-f5d0-4481-a927-4a745b692f7a" volumeName="kubernetes.io/configmap/58f7cf25-f5d0-4481-a927-4a745b692f7a-registry-certificates" seLinuxMountContext="" Oct 30 23:41:27.059733 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:27.053716 4927 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="58f7cf25-f5d0-4481-a927-4a745b692f7a" volumeName="kubernetes.io/empty-dir/58f7cf25-f5d0-4481-a927-4a745b692f7a-ca-trust-extracted" seLinuxMountContext="" Oct 30 23:41:27.059733 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:27.053724 4927 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="58f7cf25-f5d0-4481-a927-4a745b692f7a" volumeName="kubernetes.io/projected/58f7cf25-f5d0-4481-a927-4a745b692f7a-bound-sa-token" seLinuxMountContext="" Oct 30 23:41:27.059733 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:27.053733 4927 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7728abfe-1747-422e-b8e2-c91c9de83d09" volumeName="kubernetes.io/projected/7728abfe-1747-422e-b8e2-c91c9de83d09-kube-api-access-jqmnp" seLinuxMountContext="" Oct 30 23:41:27.059733 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:27.053742 4927 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="840aaae0-a215-4ccd-b420-d6f5723b65ae" volumeName="kubernetes.io/projected/840aaae0-a215-4ccd-b420-d6f5723b65ae-kube-api-access-vsfpv" seLinuxMountContext="" Oct 30 23:41:27.059733 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:27.053756 4927 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a78879b4-333e-4f1a-88bc-e96aee0e6d6b" volumeName="kubernetes.io/configmap/a78879b4-333e-4f1a-88bc-e96aee0e6d6b-service-ca-bundle" seLinuxMountContext="" Oct 30 23:41:27.059733 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:27.053769 4927 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b53ad6ed-90fd-4a0f-bf54-88d416ba19e9" volumeName="kubernetes.io/configmap/b53ad6ed-90fd-4a0f-bf54-88d416ba19e9-nginx-conf" seLinuxMountContext="" Oct 30 23:41:27.059733 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:27.053781 4927 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="10774606-8e0c-460b-bd5f-4cb6d121b30b" volumeName="kubernetes.io/configmap/10774606-8e0c-460b-bd5f-4cb6d121b30b-trusted-ca-bundle" seLinuxMountContext="" Oct 30 23:41:27.059733 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:27.053789 4927 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3af6ccf2-97f1-4598-b8ef-12eac377133b" volumeName="kubernetes.io/projected/3af6ccf2-97f1-4598-b8ef-12eac377133b-kube-api-access-9xfx9" seLinuxMountContext="" Oct 30 23:41:27.059733 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:27.053797 4927 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="97da2086-b00d-4f03-b502-56de5ee3d90d" volumeName="kubernetes.io/secret/97da2086-b00d-4f03-b502-56de5ee3d90d-original-pull-secret" seLinuxMountContext="" Oct 30 23:41:27.059733 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:27.053822 4927 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0d4810de-8bd2-4270-b907-341d9f840bb8" volumeName="kubernetes.io/secret/0d4810de-8bd2-4270-b907-341d9f840bb8-ovn-node-metrics-cert" seLinuxMountContext="" Oct 30 23:41:27.059733 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:27.053844 4927 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2fa7d24b-67b0-400f-94d4-903feb268782" volumeName="kubernetes.io/configmap/2fa7d24b-67b0-400f-94d4-903feb268782-multus-daemon-config" seLinuxMountContext="" Oct 30 23:41:27.059733 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:27.053856 4927 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="64ae3ff5-44eb-456b-948a-c1a8320e6b07" volumeName="kubernetes.io/projected/64ae3ff5-44eb-456b-948a-c1a8320e6b07-kube-api-access-cfwlk" seLinuxMountContext="" Oct 30 23:41:27.059733 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:27.053868 4927 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="953b9446-c289-48ca-b854-a14c70e5971b" volumeName="kubernetes.io/empty-dir/953b9446-c289-48ca-b854-a14c70e5971b-etc-tuned" seLinuxMountContext="" Oct 30 23:41:27.059733 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:27.053881 4927 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="99f800b6-a38c-46ca-8234-0dbd3610c8fd" volumeName="kubernetes.io/configmap/99f800b6-a38c-46ca-8234-0dbd3610c8fd-serviceca" seLinuxMountContext="" Oct 30 23:41:27.059733 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:27.053894 4927 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a78879b4-333e-4f1a-88bc-e96aee0e6d6b" volumeName="kubernetes.io/configmap/a78879b4-333e-4f1a-88bc-e96aee0e6d6b-trusted-ca-bundle" seLinuxMountContext="" Oct 30 23:41:27.059733 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:27.053907 4927 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f464ef41-45c9-4ff4-b410-e2d9d6cdbdf5" volumeName="kubernetes.io/projected/f464ef41-45c9-4ff4-b410-e2d9d6cdbdf5-kube-api-access-ztdxk" seLinuxMountContext="" Oct 30 23:41:27.059733 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:27.053924 4927 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f464ef41-45c9-4ff4-b410-e2d9d6cdbdf5" volumeName="kubernetes.io/secret/f464ef41-45c9-4ff4-b410-e2d9d6cdbdf5-serving-cert" seLinuxMountContext="" Oct 30 23:41:27.059733 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:27.053938 4927 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0d4810de-8bd2-4270-b907-341d9f840bb8" volumeName="kubernetes.io/configmap/0d4810de-8bd2-4270-b907-341d9f840bb8-env-overrides" seLinuxMountContext="" Oct 30 23:41:27.059733 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:27.053951 4927 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2fa7d24b-67b0-400f-94d4-903feb268782" volumeName="kubernetes.io/configmap/2fa7d24b-67b0-400f-94d4-903feb268782-cni-binary-copy" seLinuxMountContext="" Oct 30 23:41:27.059733 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:27.053964 4927 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="58f7cf25-f5d0-4481-a927-4a745b692f7a" volumeName="kubernetes.io/configmap/58f7cf25-f5d0-4481-a927-4a745b692f7a-trusted-ca" seLinuxMountContext="" Oct 30 23:41:27.059733 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:27.053979 4927 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="99f800b6-a38c-46ca-8234-0dbd3610c8fd" volumeName="kubernetes.io/projected/99f800b6-a38c-46ca-8234-0dbd3610c8fd-kube-api-access-mlhgb" seLinuxMountContext="" Oct 30 23:41:27.059733 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:27.053993 4927 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ec330d4f-2eea-476a-a7b8-60cd295fddd2" volumeName="kubernetes.io/empty-dir/ec330d4f-2eea-476a-a7b8-60cd295fddd2-data-volume" seLinuxMountContext="" Oct 30 23:41:27.059733 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:27.054008 4927 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f8b33ecc-59a4-4680-9521-299648ea5055" volumeName="kubernetes.io/empty-dir/f8b33ecc-59a4-4680-9521-299648ea5055-tmp-dir" seLinuxMountContext="" Oct 30 23:41:27.059733 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:27.054022 4927 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f8b33ecc-59a4-4680-9521-299648ea5055" volumeName="kubernetes.io/projected/f8b33ecc-59a4-4680-9521-299648ea5055-kube-api-access-7p9rj" seLinuxMountContext="" Oct 30 23:41:27.059733 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:27.054036 4927 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f8eb4b4c-8225-436b-b1e8-7775414a1ca3" volumeName="kubernetes.io/projected/f8eb4b4c-8225-436b-b1e8-7775414a1ca3-kube-api-access-k9l2q" seLinuxMountContext="" Oct 30 23:41:27.059733 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:27.054048 4927 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="03211c6a-9761-419c-839b-21ca413d5bb2" volumeName="kubernetes.io/projected/03211c6a-9761-419c-839b-21ca413d5bb2-kube-api-access-5x8x6" seLinuxMountContext="" Oct 30 23:41:27.059733 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:27.054072 4927 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="29b8bd1a-beeb-495c-9f61-e3326866a1d7" volumeName="kubernetes.io/empty-dir/29b8bd1a-beeb-495c-9f61-e3326866a1d7-tmp-dir" seLinuxMountContext="" Oct 30 23:41:27.059733 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:27.054086 4927 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="29b8bd1a-beeb-495c-9f61-e3326866a1d7" volumeName="kubernetes.io/projected/29b8bd1a-beeb-495c-9f61-e3326866a1d7-kube-api-access-vj8vx" seLinuxMountContext="" Oct 30 23:41:27.059733 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:27.054100 4927 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4cf4abe6-da2b-41b6-9caa-9659d4f750fb" volumeName="kubernetes.io/secret/4cf4abe6-da2b-41b6-9caa-9659d4f750fb-default-certificate" seLinuxMountContext="" Oct 30 23:41:27.059733 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:27.054113 4927 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="be5540b5-7b10-45b2-b40c-723efd9f1b01" volumeName="kubernetes.io/configmap/be5540b5-7b10-45b2-b40c-723efd9f1b01-samples-operator-config" seLinuxMountContext="" Oct 30 23:41:27.059733 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:27.054127 4927 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="03211c6a-9761-419c-839b-21ca413d5bb2" volumeName="kubernetes.io/configmap/03211c6a-9761-419c-839b-21ca413d5bb2-telemetry-config" seLinuxMountContext="" Oct 30 23:41:27.059733 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:27.054140 4927 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4cf4abe6-da2b-41b6-9caa-9659d4f750fb" volumeName="kubernetes.io/projected/4cf4abe6-da2b-41b6-9caa-9659d4f750fb-kube-api-access-rtfn9" seLinuxMountContext="" Oct 30 23:41:27.059733 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:27.054154 4927 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="840aaae0-a215-4ccd-b420-d6f5723b65ae" volumeName="kubernetes.io/configmap/840aaae0-a215-4ccd-b420-d6f5723b65ae-config" seLinuxMountContext="" Oct 30 23:41:27.059733 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:27.054167 4927 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9700c98a-b858-45a5-9025-8d75f3c14044" volumeName="kubernetes.io/projected/9700c98a-b858-45a5-9025-8d75f3c14044-kube-api-access-5xn7l" seLinuxMountContext="" Oct 30 23:41:27.059733 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:27.054182 4927 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0796261f-298e-4cbe-9b3e-b3dcf4528e07" volumeName="kubernetes.io/projected/0796261f-298e-4cbe-9b3e-b3dcf4528e07-kube-api-access-25rtq" seLinuxMountContext="" Oct 30 23:41:27.059733 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:27.054195 4927 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="145dd323-9370-4b9d-a96e-8b93d0a47a7c" volumeName="kubernetes.io/projected/145dd323-9370-4b9d-a96e-8b93d0a47a7c-kube-api-access-nsggg" seLinuxMountContext="" Oct 30 23:41:27.059733 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:27.054223 4927 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="17461b46-7ebf-49b7-afaf-12e20e3994a3" volumeName="kubernetes.io/projected/17461b46-7ebf-49b7-afaf-12e20e3994a3-kube-api-access-xkthh" seLinuxMountContext="" Oct 30 23:41:27.059733 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:27.054236 4927 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a4f35692-894d-4d41-8b9f-3f5ba3c529b1" volumeName="kubernetes.io/projected/a4f35692-894d-4d41-8b9f-3f5ba3c529b1-kube-api-access-67888" seLinuxMountContext="" Oct 30 23:41:27.059733 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:27.054251 4927 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="10774606-8e0c-460b-bd5f-4cb6d121b30b" volumeName="kubernetes.io/configmap/10774606-8e0c-460b-bd5f-4cb6d121b30b-service-ca-bundle" seLinuxMountContext="" Oct 30 23:41:27.059733 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:27.054265 4927 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6382dfb5-a363-47a2-bc79-0b96f7bec0c3" volumeName="kubernetes.io/configmap/6382dfb5-a363-47a2-bc79-0b96f7bec0c3-config" seLinuxMountContext="" Oct 30 23:41:27.059733 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:27.054282 4927 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="953b9446-c289-48ca-b854-a14c70e5971b" volumeName="kubernetes.io/empty-dir/953b9446-c289-48ca-b854-a14c70e5971b-tmp" seLinuxMountContext="" Oct 30 23:41:27.059733 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:27.054296 4927 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a8e4c4db-197e-420c-9c60-4273cdf95377" volumeName="kubernetes.io/secret/a8e4c4db-197e-420c-9c60-4273cdf95377-agent-certs" seLinuxMountContext="" Oct 30 23:41:27.059733 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:27.054312 4927 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="58f7cf25-f5d0-4481-a927-4a745b692f7a" volumeName="kubernetes.io/projected/58f7cf25-f5d0-4481-a927-4a745b692f7a-kube-api-access-sx9pp" seLinuxMountContext="" Oct 30 23:41:27.059733 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:27.054342 4927 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6382dfb5-a363-47a2-bc79-0b96f7bec0c3" volumeName="kubernetes.io/projected/6382dfb5-a363-47a2-bc79-0b96f7bec0c3-kube-api-access-qg8hn" seLinuxMountContext="" Oct 30 23:41:27.059733 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:27.054356 4927 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0d4810de-8bd2-4270-b907-341d9f840bb8" volumeName="kubernetes.io/configmap/0d4810de-8bd2-4270-b907-341d9f840bb8-ovnkube-script-lib" seLinuxMountContext="" Oct 30 23:41:27.059733 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:27.054368 4927 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="29b8bd1a-beeb-495c-9f61-e3326866a1d7" volumeName="kubernetes.io/configmap/29b8bd1a-beeb-495c-9f61-e3326866a1d7-config-volume" seLinuxMountContext="" Oct 30 23:41:27.059733 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:27.054383 4927 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2fa7d24b-67b0-400f-94d4-903feb268782" volumeName="kubernetes.io/projected/2fa7d24b-67b0-400f-94d4-903feb268782-kube-api-access-ssd98" seLinuxMountContext="" Oct 30 23:41:27.059733 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:27.054395 4927 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a78879b4-333e-4f1a-88bc-e96aee0e6d6b" volumeName="kubernetes.io/projected/a78879b4-333e-4f1a-88bc-e96aee0e6d6b-kube-api-access-sdj76" seLinuxMountContext="" Oct 30 23:41:27.059733 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:27.054408 4927 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a8e4c4db-197e-420c-9c60-4273cdf95377" volumeName="kubernetes.io/configmap/a8e4c4db-197e-420c-9c60-4273cdf95377-konnectivity-ca" seLinuxMountContext="" Oct 30 23:41:27.059733 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:27.054420 4927 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b22fe41c-cf52-4e97-81f1-2f64bcc43640" volumeName="kubernetes.io/projected/b22fe41c-cf52-4e97-81f1-2f64bcc43640-kube-api-access-btgs8" seLinuxMountContext="" Oct 30 23:41:27.059733 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:27.054433 4927 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6382dfb5-a363-47a2-bc79-0b96f7bec0c3" volumeName="kubernetes.io/configmap/6382dfb5-a363-47a2-bc79-0b96f7bec0c3-trusted-ca" seLinuxMountContext="" Oct 30 23:41:27.059733 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:27.054445 4927 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ec330d4f-2eea-476a-a7b8-60cd295fddd2" volumeName="kubernetes.io/projected/ec330d4f-2eea-476a-a7b8-60cd295fddd2-kube-api-access-z4n58" seLinuxMountContext="" Oct 30 23:41:27.059733 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:27.054459 4927 reconstruct.go:97] "Volume reconstruction finished" Oct 30 23:41:27.059733 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:27.054467 4927 reconciler.go:29] "Reconciler: start to sync state" Oct 30 23:41:27.059733 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:27.057591 4927 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-pqt2c_6382dfb5-a363-47a2-bc79-0b96f7bec0c3/console-operator/1.log" Oct 30 23:41:27.059733 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:27.059512 4927 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-pqt2c_6382dfb5-a363-47a2-bc79-0b96f7bec0c3/console-operator/1.log" Oct 30 23:41:27.076516 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:27.076469 4927 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 23:41:27.077904 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:27.077881 4927 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 23:41:27.077904 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:27.077901 4927 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 23:41:27.078425 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:27.077920 4927 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 23:41:27.078425 ip-10-0-141-180 kubenswrapper[4927]: E1030 23:41:27.077963 4927 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Oct 30 23:41:27.100312 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:27.100282 4927 generic.go:356] "Generic (PLEG): container finished" podID="b22fe41c-cf52-4e97-81f1-2f64bcc43640" containerID="b8c54e9dd60ac3d6b707158cdbb1f65969985423d3907afdd4ac51290047332f" exitCode=0 Oct 30 23:41:27.100420 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:27.100312 4927 generic.go:356] "Generic (PLEG): container finished" podID="b22fe41c-cf52-4e97-81f1-2f64bcc43640" containerID="5f51926d68e2a0e635eb974f550ce17ad576034eef7ad09e8250de50c547cf3a" exitCode=0 Oct 30 23:41:27.100420 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:27.100322 4927 generic.go:356] "Generic (PLEG): container finished" podID="b22fe41c-cf52-4e97-81f1-2f64bcc43640" containerID="e158c325258c29c4ef0643a4493be3843d84484b3dc8daf0da769ecc29722de9" exitCode=0 Oct 30 23:41:27.100420 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:27.100353 4927 generic.go:356] "Generic (PLEG): container finished" podID="b22fe41c-cf52-4e97-81f1-2f64bcc43640" containerID="bfdcd4ef9e891b5bd8f68623867a8a250524ed0d5cb742b332ed7f5916830147" exitCode=0 Oct 30 23:41:27.100420 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:27.100363 4927 generic.go:356] "Generic (PLEG): container finished" podID="b22fe41c-cf52-4e97-81f1-2f64bcc43640" containerID="b5ae94dd9747f2578c6cfa7d12200dece2c3865629b1717ad2c2e93e7a9595c7" exitCode=0 Oct 30 23:41:27.100420 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:27.100371 4927 generic.go:356] "Generic (PLEG): container finished" podID="b22fe41c-cf52-4e97-81f1-2f64bcc43640" containerID="ce628f435799ce877d768d21adb4750878b8e18381849400b2afe380bb635067" exitCode=0 Oct 30 23:41:27.106949 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:27.106929 4927 generic.go:356] "Generic (PLEG): container finished" podID="fd028af9af4f41b9772b02bdb5343dc5" containerID="0cbd49cfc73086c0ebe144eeead70ae2fe4039fe472d088d347917b95fa867ea" exitCode=0 Oct 30 23:41:27.119116 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:27.119081 4927 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-pqt2c_6382dfb5-a363-47a2-bc79-0b96f7bec0c3/console-operator/1.log" Oct 30 23:41:27.119199 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:27.119114 4927 generic.go:356] "Generic (PLEG): container finished" podID="6382dfb5-a363-47a2-bc79-0b96f7bec0c3" containerID="e373ab1ed0edd47d38ff51aab784dee07bf119717f872d40609595f835e4357c" exitCode=255 Oct 30 23:41:27.142148 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:27.142126 4927 manager.go:324] Recovery completed Oct 30 23:41:27.149437 ip-10-0-141-180 kubenswrapper[4927]: E1030 23:41:27.149414 4927 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-180.us-west-2.compute.internal\" not found" Oct 30 23:41:27.158148 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:27.158132 4927 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 23:41:27.159301 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:27.159286 4927 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-180.us-west-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 23:41:27.159390 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:27.159318 4927 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-180.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 23:41:27.159390 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:27.159356 4927 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-180.us-west-2.compute.internal" event="NodeHasSufficientPID" Oct 30 23:41:27.161782 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:27.161765 4927 cpu_manager.go:226] "Starting" policy="none" Oct 30 23:41:27.161782 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:27.161779 4927 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 23:41:27.161919 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:27.161798 4927 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 23:41:27.161919 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:27.161837 4927 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 23:41:27.162036 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:27.161945 4927 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Oct 30 23:41:27.162036 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:27.161961 4927 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Oct 30 23:41:27.162036 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:27.161985 4927 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Oct 30 23:41:27.162036 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:27.161994 4927 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Oct 30 23:41:27.162036 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:27.162002 4927 policy_none.go:50] "Start" Oct 30 23:41:27.162036 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:27.162024 4927 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 23:41:27.162036 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:27.162037 4927 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 23:41:27.162730 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:27.162717 4927 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-180.us-west-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 23:41:27.162774 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:27.162743 4927 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-180.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 23:41:27.162774 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:27.162753 4927 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-180.us-west-2.compute.internal" event="NodeHasSufficientPID" Oct 30 23:41:27.175238 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:27.175220 4927 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Oct 30 23:41:27.175327 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:27.175240 4927 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Oct 30 23:41:27.175327 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:27.175251 4927 policy_none.go:44] "Start" Oct 30 23:41:27.178398 ip-10-0-141-180 kubenswrapper[4927]: E1030 23:41:27.178376 4927 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet" Oct 30 23:41:27.180866 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:27.180839 4927 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 23:41:27.183966 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:27.183952 4927 manager.go:341] "Starting Device Plugin manager" Oct 30 23:41:27.184038 ip-10-0-141-180 kubenswrapper[4927]: E1030 23:41:27.183977 4927 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 23:41:27.184038 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:27.183989 4927 server.go:87] "Starting device plugin registration server" Oct 30 23:41:27.184252 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:27.184238 4927 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 23:41:27.184310 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:27.184251 4927 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 23:41:27.184388 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:27.184371 4927 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 23:41:27.184501 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:27.184465 4927 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 23:41:27.184678 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:27.184652 4927 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 23:41:27.184763 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:27.184718 4927 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 23:41:27.184833 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:27.184755 4927 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T23:41:27.184495702Z","UUID":"f6463a52-80e5-4489-a0b3-ea8d238462b8","Handler":null,"Name":"","Endpoint":""} Oct 30 23:41:27.185988 ip-10-0-141-180 kubenswrapper[4927]: E1030 23:41:27.185965 4927 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 23:41:27.186102 ip-10-0-141-180 kubenswrapper[4927]: E1030 23:41:27.186015 4927 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-141-180.us-west-2.compute.internal\" not found" Oct 30 23:41:27.186172 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:27.186114 4927 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 23:41:27.186223 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:27.186182 4927 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 23:41:27.242622 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:27.242589 4927 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 23:41:27.242622 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:27.242596 4927 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 23:41:27.242773 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:27.242717 4927 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Oct 30 23:41:27.243579 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:27.243564 4927 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 23:41:27.285094 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:27.285027 4927 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 23:41:27.286059 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:27.286044 4927 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-180.us-west-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 23:41:27.286125 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:27.286074 4927 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-180.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 23:41:27.286125 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:27.286084 4927 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-180.us-west-2.compute.internal" event="NodeHasSufficientPID" Oct 30 23:41:27.286125 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:27.286125 4927 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-141-180.us-west-2.compute.internal" Oct 30 23:41:27.294916 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:27.294883 4927 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 23:41:27.379146 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:27.379099 4927 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-180.us-west-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-141-180.us-west-2.compute.internal"] Oct 30 23:41:27.379816 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:27.379777 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-180.us-west-2.compute.internal" event={"ID":"fd028af9af4f41b9772b02bdb5343dc5","Type":"ContainerStarted","Data":"45a922e83b9d3d59575a0f5f53ff8c6543356a7dd0694833ed3ce55429beb9fe"} Oct 30 23:41:27.379889 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:27.379814 4927 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-180.us-west-2.compute.internal" Oct 30 23:41:27.379889 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:27.379822 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-180.us-west-2.compute.internal" event={"ID":"fd028af9af4f41b9772b02bdb5343dc5","Type":"ContainerDied","Data":"0cbd49cfc73086c0ebe144eeead70ae2fe4039fe472d088d347917b95fa867ea"} Oct 30 23:41:27.379982 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:27.379906 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-180.us-west-2.compute.internal" event={"ID":"fd028af9af4f41b9772b02bdb5343dc5","Type":"ContainerStarted","Data":"4357c8b1b1cd7c31d12ef70138214a17b84944e2afc3d6d78932c37e9743b036"} Oct 30 23:41:27.379982 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:27.379927 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-180.us-west-2.compute.internal" event={"ID":"2c1c7fe71dc5ea64e6fbc8dd666819fa","Type":"ContainerStarted","Data":"2043c7d133b2def891e736545ab2908e1466262942abb4176193031621dd73b7"} Oct 30 23:41:27.379982 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:27.379943 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-180.us-west-2.compute.internal" event={"ID":"2c1c7fe71dc5ea64e6fbc8dd666819fa","Type":"ContainerStarted","Data":"154e4407f570053087b1f80f38a62a9607d8146fbbadeb07629573256de48e7e"} Oct 30 23:41:27.380100 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:27.380088 4927 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-180.us-west-2.compute.internal" Oct 30 23:41:27.400259 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:27.400242 4927 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-141-180.us-west-2.compute.internal" Oct 30 23:41:27.400319 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:27.400298 4927 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-141-180.us-west-2.compute.internal" Oct 30 23:41:27.400872 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:27.400849 4927 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-180.us-west-2.compute.internal" event="NodeNotReady" Oct 30 23:41:27.400949 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:27.400876 4927 setters.go:546] "Node became not ready" node="ip-10-0-141-180.us-west-2.compute.internal" condition={"type":"Ready","status":"False","lastHeartbeatTime":"2026-10-30T23:41:27Z","lastTransitionTime":"2026-10-30T23:41:27Z","reason":"KubeletNotReady","message":"CSINode is not yet initialized"} Oct 30 23:41:27.411256 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:27.411236 4927 nodeinfomanager.go:471] existing CSINode "ip-10-0-141-180.us-west-2.compute.internal" is owned by different node (oldNodeID="95cc2d32-770f-4596-8388-aa371e00c592", newNodeID=""), cleaning up... Oct 30 23:41:27.442326 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:27.442308 4927 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 23:41:27.442415 ip-10-0-141-180 kubenswrapper[4927]: E1030 23:41:27.442365 4927 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-141-180.us-west-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-180.us-west-2.compute.internal" Oct 30 23:41:27.442967 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:27.442953 4927 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 23:41:27.443000 ip-10-0-141-180 kubenswrapper[4927]: E1030 23:41:27.442987 4927 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-141-180.us-west-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-180.us-west-2.compute.internal" Oct 30 23:41:27.544758 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:27.544688 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/fd028af9af4f41b9772b02bdb5343dc5-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-180.us-west-2.compute.internal\" (UID: \"fd028af9af4f41b9772b02bdb5343dc5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-180.us-west-2.compute.internal" Oct 30 23:41:27.544758 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:27.544737 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fd028af9af4f41b9772b02bdb5343dc5-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-180.us-west-2.compute.internal\" (UID: \"fd028af9af4f41b9772b02bdb5343dc5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-180.us-west-2.compute.internal" Oct 30 23:41:27.544919 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:27.544758 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2c1c7fe71dc5ea64e6fbc8dd666819fa-config\") pod \"kube-apiserver-proxy-ip-10-0-141-180.us-west-2.compute.internal\" (UID: \"2c1c7fe71dc5ea64e6fbc8dd666819fa\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-180.us-west-2.compute.internal" Oct 30 23:41:27.645377 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:27.645347 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/fd028af9af4f41b9772b02bdb5343dc5-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-180.us-west-2.compute.internal\" (UID: \"fd028af9af4f41b9772b02bdb5343dc5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-180.us-west-2.compute.internal" Oct 30 23:41:27.645377 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:27.645377 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fd028af9af4f41b9772b02bdb5343dc5-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-180.us-west-2.compute.internal\" (UID: \"fd028af9af4f41b9772b02bdb5343dc5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-180.us-west-2.compute.internal" Oct 30 23:41:27.645535 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:27.645394 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2c1c7fe71dc5ea64e6fbc8dd666819fa-config\") pod \"kube-apiserver-proxy-ip-10-0-141-180.us-west-2.compute.internal\" (UID: \"2c1c7fe71dc5ea64e6fbc8dd666819fa\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-180.us-west-2.compute.internal" Oct 30 23:41:27.645535 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:27.645434 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2c1c7fe71dc5ea64e6fbc8dd666819fa-config\") pod \"kube-apiserver-proxy-ip-10-0-141-180.us-west-2.compute.internal\" (UID: \"2c1c7fe71dc5ea64e6fbc8dd666819fa\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-180.us-west-2.compute.internal" Oct 30 23:41:27.645535 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:27.645455 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/fd028af9af4f41b9772b02bdb5343dc5-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-180.us-west-2.compute.internal\" (UID: \"fd028af9af4f41b9772b02bdb5343dc5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-180.us-west-2.compute.internal" Oct 30 23:41:27.645535 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:27.645457 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fd028af9af4f41b9772b02bdb5343dc5-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-180.us-west-2.compute.internal\" (UID: \"fd028af9af4f41b9772b02bdb5343dc5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-180.us-west-2.compute.internal" Oct 30 23:41:27.675440 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:27.675419 4927 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-180.us-west-2.compute.internal" event="NodeReady" Oct 30 23:41:27.675525 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:27.675493 4927 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Oct 30 23:41:28.025370 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.025341 4927 apiserver.go:52] "Watching apiserver" Oct 30 23:41:28.133421 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.133396 4927 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-180.us-west-2.compute.internal" Oct 30 23:41:28.133610 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.133505 4927 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-180.us-west-2.compute.internal" Oct 30 23:41:28.182752 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.182725 4927 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 23:41:28.183434 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.183410 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-66d97c9c97-qqxj5","openshift-multus/multus-cb9vj","openshift-network-diagnostics/network-check-source-6f78d59f99-xrk8k","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-180.us-west-2.compute.internal","openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vl8t5","openshift-network-diagnostics/network-check-target-kgdzs","openshift-network-operator/iptables-alerter-57xrk","openshift-monitoring/cluster-monitoring-operator-884576c57-httg9","openshift-multus/multus-additional-cni-plugins-7mp5r","openshift-service-ca-operator/service-ca-operator-544d749c8c-fbgth","kube-system/kube-apiserver-proxy-ip-10-0-141-180.us-west-2.compute.internal","openshift-image-registry/node-ca-7sbzf","openshift-console-operator/console-operator-7cd56b8dbc-pqt2c","openshift-network-console/networking-console-plugin-86c478bb88-rxpst","kube-system/konnectivity-agent-9s2fx","openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-9qs84","openshift-ovn-kubernetes/ovnkube-node-26v6x","kube-system/kas-connection-checker-685895c7c9-f7zjp","kube-system/global-pull-secret-syncer-7wrh7","openshift-multus/network-metrics-daemon-pt58s","openshift-insights/insights-runtime-extractor-sqv69","openshift-image-registry/image-registry-dd954f877-x9h9g","kube-system/kas-connection-checker-685895c7c9-l4d2m","openshift-cluster-node-tuning-operator/tuned-rsv7z","openshift-insights/periodic-gathering-r9crl-vqcgk","openshift-image-registry/image-registry-544b44f8d-5p6kz","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j7n2l","openshift-insights/insights-operator-7c4986d449-9mqqt","openshift-dns/node-resolver-tfnr7","openshift-kube-storage-version-migrator/migrator-66df5d496b-8khpq","kube-system/kas-connection-checker-685895c7c9-lxdsf","openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-tcxmv","openshift-dns/dns-default-9l5zq","openshift-ingress-canary/ingress-canary-46zqw"] Oct 30 23:41:28.184645 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.184619 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-66d97c9c97-qqxj5" Oct 30 23:41:28.184983 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.184966 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vl8t5" Oct 30 23:41:28.185796 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.185776 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-dd954f877-x9h9g" Oct 30 23:41:28.187405 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.187387 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-httg9" Oct 30 23:41:28.187772 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.187752 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-rxpst" Oct 30 23:41:28.190050 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.190029 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pt58s" Oct 30 23:41:28.195962 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.195937 4927 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 23:41:28.196066 ip-10-0-141-180 kubenswrapper[4927]: E1030 23:41:28.196030 4927 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-141-180.us-west-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-180.us-west-2.compute.internal" Oct 30 23:41:28.196132 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.196094 4927 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 23:41:28.196188 ip-10-0-141-180 kubenswrapper[4927]: E1030 23:41:28.196136 4927 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-141-180.us-west-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-180.us-west-2.compute.internal" Oct 30 23:41:28.197554 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.197537 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-544b44f8d-5p6kz" Oct 30 23:41:28.198440 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.198423 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-9l5zq" Oct 30 23:41:28.199968 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.199954 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-sqv69" Oct 30 23:41:28.237902 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.237884 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\"" Oct 30 23:41:28.238167 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.238149 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\"" Oct 30 23:41:28.238240 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.238228 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Oct 30 23:41:28.238405 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.238395 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Oct 30 23:41:28.239103 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.239091 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Oct 30 23:41:28.239199 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.239187 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Oct 30 23:41:28.239420 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.239408 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Oct 30 23:41:28.240417 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.240395 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Oct 30 23:41:28.240417 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.240415 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Oct 30 23:41:28.240557 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.240451 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Oct 30 23:41:28.240557 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.240450 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Oct 30 23:41:28.240557 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.240487 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-vxpd6\"" Oct 30 23:41:28.240557 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.240506 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-pkxn2\"" Oct 30 23:41:28.240557 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.240514 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Oct 30 23:41:28.240557 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.240535 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Oct 30 23:41:28.240813 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.240633 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Oct 30 23:41:28.240813 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.240727 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 23:41:28.240813 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.240759 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\"" Oct 30 23:41:28.240813 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.240769 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Oct 30 23:41:28.241313 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.241297 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Oct 30 23:41:28.241388 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.241301 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Oct 30 23:41:28.241692 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.241676 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-9xzgs\"" Oct 30 23:41:28.241788 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.241775 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Oct 30 23:41:28.241853 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.241840 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-lhhdb\"" Oct 30 23:41:28.241979 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.241959 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Oct 30 23:41:28.242079 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.241979 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-nl2hc\"" Oct 30 23:41:28.242262 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.242242 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-jmjnd\"" Oct 30 23:41:28.242366 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.242244 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 23:41:28.242366 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.242249 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 23:41:28.242366 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.242252 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 23:41:28.242366 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.242266 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Oct 30 23:41:28.242366 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.242359 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Oct 30 23:41:28.242549 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.242382 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Oct 30 23:41:28.242745 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.242728 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Oct 30 23:41:28.242817 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.242768 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 23:41:28.242817 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.242774 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 23:41:28.242817 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.242786 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Oct 30 23:41:28.242817 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.242797 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 23:41:28.242817 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.242798 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Oct 30 23:41:28.242971 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.242827 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Oct 30 23:41:28.242971 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.242830 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Oct 30 23:41:28.242971 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.242912 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Oct 30 23:41:28.242971 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.242942 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Oct 30 23:41:28.243130 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.243113 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Oct 30 23:41:28.249921 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.249901 4927 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 23:41:28.294513 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.294450 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Oct 30 23:41:28.294589 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.294551 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Oct 30 23:41:28.294772 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.294757 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Oct 30 23:41:28.337930 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.337904 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 23:41:28.338027 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.337933 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 23:41:28.338027 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.337964 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 23:41:28.338027 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.337973 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Oct 30 23:41:28.338212 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.338073 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 23:41:28.338212 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.338091 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 30 23:41:28.338212 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.338103 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 23:41:28.338212 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.338106 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 23:41:28.338212 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.338139 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 23:41:28.338435 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.338399 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 23:41:28.338532 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.338511 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 23:41:28.338864 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.338511 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 23:41:28.338985 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.338542 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 23:41:28.338985 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.338554 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 23:41:28.338985 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.338579 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 23:41:28.338985 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.338927 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 23:41:28.338985 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.338584 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 23:41:28.338985 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.338614 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 23:41:28.338985 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.338605 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 23:41:28.339274 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.338642 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 23:41:28.339274 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.338624 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 23:41:28.339274 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.338675 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Oct 30 23:41:28.339274 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.338673 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 23:41:28.339274 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.338693 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Oct 30 23:41:28.339274 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.338723 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 23:41:28.339274 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.338811 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 23:41:28.339274 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.338819 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 23:41:28.339274 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.338661 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 23:41:28.339274 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.339276 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 23:41:28.339695 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.339583 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 23:41:28.339695 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.339631 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 23:41:28.339695 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.339651 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Oct 30 23:41:28.339695 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.339672 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 23:41:28.339944 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.339928 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-bmqk5\"" Oct 30 23:41:28.340005 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.339960 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-jh2kx\"" Oct 30 23:41:28.340005 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.339978 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-zg485\"" Oct 30 23:41:28.340005 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.340000 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 23:41:28.340132 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.339967 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Oct 30 23:41:28.340132 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.340058 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 23:41:28.340132 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.340106 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-hcpbw\"" Oct 30 23:41:28.340132 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.340124 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-5ndcx\"" Oct 30 23:41:28.340132 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.340010 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Oct 30 23:41:28.340360 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.340206 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-554bl\"" Oct 30 23:41:28.340360 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.340226 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-ksfp2\"" Oct 30 23:41:28.340360 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.340229 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-72946\"" Oct 30 23:41:28.340360 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.340299 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-v7rnf\"" Oct 30 23:41:28.340545 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.340365 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-zwz57\"" Oct 30 23:41:28.340545 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.340390 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 23:41:28.340545 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.340411 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-hnph4\"" Oct 30 23:41:28.340545 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.340437 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-xnb7d\"" Oct 30 23:41:28.340545 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.340517 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 23:41:28.340545 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.340528 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 23:41:28.340545 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.340537 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-294w8\"" Oct 30 23:41:28.341175 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.341154 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-w5vpr\"" Oct 30 23:41:28.341175 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.341166 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-28w6g\"" Oct 30 23:41:28.341296 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.341219 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 23:41:28.341388 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.341369 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-tnlbs\"" Oct 30 23:41:28.341522 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.341505 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-fvkr7\"" Oct 30 23:41:28.341638 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.341623 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-6s2q2\"" Oct 30 23:41:28.348629 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.348601 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29b8bd1a-beeb-495c-9f61-e3326866a1d7-tmp-dir\") pod \"dns-default-9l5zq\" (UID: \"29b8bd1a-beeb-495c-9f61-e3326866a1d7\") " pod="openshift-dns/dns-default-9l5zq" Oct 30 23:41:28.348723 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.348631 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f8b33ecc-59a4-4680-9521-299648ea5055-hosts-file\") pod \"node-resolver-tfnr7\" (UID: \"f8b33ecc-59a4-4680-9521-299648ea5055\") " pod="openshift-dns/node-resolver-tfnr7" Oct 30 23:41:28.348723 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.348647 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0d4810de-8bd2-4270-b907-341d9f840bb8-node-log\") pod \"ovnkube-node-26v6x\" (UID: \"0d4810de-8bd2-4270-b907-341d9f840bb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-26v6x" Oct 30 23:41:28.348723 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.348663 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0d4810de-8bd2-4270-b907-341d9f840bb8-host-run-ovn-kubernetes\") pod \"ovnkube-node-26v6x\" (UID: \"0d4810de-8bd2-4270-b907-341d9f840bb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-26v6x" Oct 30 23:41:28.348723 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.348678 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0d4810de-8bd2-4270-b907-341d9f840bb8-host-cni-bin\") pod \"ovnkube-node-26v6x\" (UID: \"0d4810de-8bd2-4270-b907-341d9f840bb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-26v6x" Oct 30 23:41:28.348723 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.348693 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2fa7d24b-67b0-400f-94d4-903feb268782-multus-cni-dir\") pod \"multus-cb9vj\" (UID: \"2fa7d24b-67b0-400f-94d4-903feb268782\") " pod="openshift-multus/multus-cb9vj" Oct 30 23:41:28.348913 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.348768 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/17461b46-7ebf-49b7-afaf-12e20e3994a3-registry-certificates\") pod \"image-registry-dd954f877-x9h9g\" (UID: \"17461b46-7ebf-49b7-afaf-12e20e3994a3\") " pod="openshift-image-registry/image-registry-dd954f877-x9h9g" Oct 30 23:41:28.348913 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.348827 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0d4810de-8bd2-4270-b907-341d9f840bb8-host-slash\") pod \"ovnkube-node-26v6x\" (UID: \"0d4810de-8bd2-4270-b907-341d9f840bb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-26v6x" Oct 30 23:41:28.348913 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.348863 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29b8bd1a-beeb-495c-9f61-e3326866a1d7-tmp-dir\") pod \"dns-default-9l5zq\" (UID: \"29b8bd1a-beeb-495c-9f61-e3326866a1d7\") " pod="openshift-dns/dns-default-9l5zq" Oct 30 23:41:28.349045 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.348915 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2fa7d24b-67b0-400f-94d4-903feb268782-cni-binary-copy\") pod \"multus-cb9vj\" (UID: \"2fa7d24b-67b0-400f-94d4-903feb268782\") " pod="openshift-multus/multus-cb9vj" Oct 30 23:41:28.349045 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.348952 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2fa7d24b-67b0-400f-94d4-903feb268782-host-var-lib-cni-bin\") pod \"multus-cb9vj\" (UID: \"2fa7d24b-67b0-400f-94d4-903feb268782\") " pod="openshift-multus/multus-cb9vj" Oct 30 23:41:28.349045 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.348982 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b53ad6ed-90fd-4a0f-bf54-88d416ba19e9-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-rxpst\" (UID: \"b53ad6ed-90fd-4a0f-bf54-88d416ba19e9\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-rxpst" Oct 30 23:41:28.349045 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.348988 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/17461b46-7ebf-49b7-afaf-12e20e3994a3-registry-certificates\") pod \"image-registry-dd954f877-x9h9g\" (UID: \"17461b46-7ebf-49b7-afaf-12e20e3994a3\") " pod="openshift-image-registry/image-registry-dd954f877-x9h9g" Oct 30 23:41:28.349045 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.349009 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/10774606-8e0c-460b-bd5f-4cb6d121b30b-serving-cert\") pod \"insights-operator-7c4986d449-9mqqt\" (UID: \"10774606-8e0c-460b-bd5f-4cb6d121b30b\") " pod="openshift-insights/insights-operator-7c4986d449-9mqqt" Oct 30 23:41:28.349045 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.349039 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/a78879b4-333e-4f1a-88bc-e96aee0e6d6b-archives-path\") pod \"periodic-gathering-r9crl-vqcgk\" (UID: \"a78879b4-333e-4f1a-88bc-e96aee0e6d6b\") " pod="openshift-insights/periodic-gathering-r9crl-vqcgk" Oct 30 23:41:28.349238 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.349062 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdj76\" (UniqueName: \"kubernetes.io/projected/a78879b4-333e-4f1a-88bc-e96aee0e6d6b-kube-api-access-sdj76\") pod \"periodic-gathering-r9crl-vqcgk\" (UID: \"a78879b4-333e-4f1a-88bc-e96aee0e6d6b\") " pod="openshift-insights/periodic-gathering-r9crl-vqcgk" Oct 30 23:41:28.349238 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.349189 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ec330d4f-2eea-476a-a7b8-60cd295fddd2-crio-socket\") pod \"insights-runtime-extractor-sqv69\" (UID: \"ec330d4f-2eea-476a-a7b8-60cd295fddd2\") " pod="openshift-insights/insights-runtime-extractor-sqv69" Oct 30 23:41:28.349238 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.349194 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/10774606-8e0c-460b-bd5f-4cb6d121b30b-serving-cert\") pod \"insights-operator-7c4986d449-9mqqt\" (UID: \"10774606-8e0c-460b-bd5f-4cb6d121b30b\") " pod="openshift-insights/insights-operator-7c4986d449-9mqqt" Oct 30 23:41:28.349238 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.349214 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0d4810de-8bd2-4270-b907-341d9f840bb8-run-systemd\") pod \"ovnkube-node-26v6x\" (UID: \"0d4810de-8bd2-4270-b907-341d9f840bb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-26v6x" Oct 30 23:41:28.349238 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.349221 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/a78879b4-333e-4f1a-88bc-e96aee0e6d6b-archives-path\") pod \"periodic-gathering-r9crl-vqcgk\" (UID: \"a78879b4-333e-4f1a-88bc-e96aee0e6d6b\") " pod="openshift-insights/periodic-gathering-r9crl-vqcgk" Oct 30 23:41:28.349436 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.349239 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/953b9446-c289-48ca-b854-a14c70e5971b-lib-modules\") pod \"tuned-rsv7z\" (UID: \"953b9446-c289-48ca-b854-a14c70e5971b\") " pod="openshift-cluster-node-tuning-operator/tuned-rsv7z" Oct 30 23:41:28.349436 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.349261 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2fa7d24b-67b0-400f-94d4-903feb268782-cni-binary-copy\") pod \"multus-cb9vj\" (UID: \"2fa7d24b-67b0-400f-94d4-903feb268782\") " pod="openshift-multus/multus-cb9vj" Oct 30 23:41:28.349436 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.349269 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2fa7d24b-67b0-400f-94d4-903feb268782-etc-kubernetes\") pod \"multus-cb9vj\" (UID: \"2fa7d24b-67b0-400f-94d4-903feb268782\") " pod="openshift-multus/multus-cb9vj" Oct 30 23:41:28.349436 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.349296 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a78879b4-333e-4f1a-88bc-e96aee0e6d6b-trusted-ca-bundle\") pod \"periodic-gathering-r9crl-vqcgk\" (UID: \"a78879b4-333e-4f1a-88bc-e96aee0e6d6b\") " pod="openshift-insights/periodic-gathering-r9crl-vqcgk" Oct 30 23:41:28.349436 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.349327 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/03211c6a-9761-419c-839b-21ca413d5bb2-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-httg9\" (UID: \"03211c6a-9761-419c-839b-21ca413d5bb2\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-httg9" Oct 30 23:41:28.349668 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.349458 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/953b9446-c289-48ca-b854-a14c70e5971b-etc-modprobe-d\") pod \"tuned-rsv7z\" (UID: \"953b9446-c289-48ca-b854-a14c70e5971b\") " pod="openshift-cluster-node-tuning-operator/tuned-rsv7z" Oct 30 23:41:28.349668 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.349508 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/953b9446-c289-48ca-b854-a14c70e5971b-etc-sysctl-conf\") pod \"tuned-rsv7z\" (UID: \"953b9446-c289-48ca-b854-a14c70e5971b\") " pod="openshift-cluster-node-tuning-operator/tuned-rsv7z" Oct 30 23:41:28.349668 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.349546 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/953b9446-c289-48ca-b854-a14c70e5971b-var-lib-kubelet\") pod \"tuned-rsv7z\" (UID: \"953b9446-c289-48ca-b854-a14c70e5971b\") " pod="openshift-cluster-node-tuning-operator/tuned-rsv7z" Oct 30 23:41:28.349668 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.349594 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/58f7cf25-f5d0-4481-a927-4a745b692f7a-ca-trust-extracted\") pod \"image-registry-544b44f8d-5p6kz\" (UID: \"58f7cf25-f5d0-4481-a927-4a745b692f7a\") " pod="openshift-image-registry/image-registry-544b44f8d-5p6kz" Oct 30 23:41:28.349668 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.349624 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a78879b4-333e-4f1a-88bc-e96aee0e6d6b-trusted-ca-bundle\") pod \"periodic-gathering-r9crl-vqcgk\" (UID: \"a78879b4-333e-4f1a-88bc-e96aee0e6d6b\") " pod="openshift-insights/periodic-gathering-r9crl-vqcgk" Oct 30 23:41:28.349668 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.349624 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f464ef41-45c9-4ff4-b410-e2d9d6cdbdf5-config\") pod \"service-ca-operator-544d749c8c-fbgth\" (UID: \"f464ef41-45c9-4ff4-b410-e2d9d6cdbdf5\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-fbgth" Oct 30 23:41:28.349668 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.349663 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/4cf4abe6-da2b-41b6-9caa-9659d4f750fb-stats-auth\") pod \"router-default-66d97c9c97-qqxj5\" (UID: \"4cf4abe6-da2b-41b6-9caa-9659d4f750fb\") " pod="openshift-ingress/router-default-66d97c9c97-qqxj5" Oct 30 23:41:28.349926 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.349685 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/be5540b5-7b10-45b2-b40c-723efd9f1b01-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-vl8t5\" (UID: \"be5540b5-7b10-45b2-b40c-723efd9f1b01\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vl8t5" Oct 30 23:41:28.349926 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.349684 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/03211c6a-9761-419c-839b-21ca413d5bb2-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-httg9\" (UID: \"03211c6a-9761-419c-839b-21ca413d5bb2\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-httg9" Oct 30 23:41:28.349926 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.349701 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/953b9446-c289-48ca-b854-a14c70e5971b-run\") pod \"tuned-rsv7z\" (UID: \"953b9446-c289-48ca-b854-a14c70e5971b\") " pod="openshift-cluster-node-tuning-operator/tuned-rsv7z" Oct 30 23:41:28.349926 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.349706 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/58f7cf25-f5d0-4481-a927-4a745b692f7a-ca-trust-extracted\") pod \"image-registry-544b44f8d-5p6kz\" (UID: \"58f7cf25-f5d0-4481-a927-4a745b692f7a\") " pod="openshift-image-registry/image-registry-544b44f8d-5p6kz" Oct 30 23:41:28.349926 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.349719 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a78879b4-333e-4f1a-88bc-e96aee0e6d6b-service-ca-bundle\") pod \"periodic-gathering-r9crl-vqcgk\" (UID: \"a78879b4-333e-4f1a-88bc-e96aee0e6d6b\") " pod="openshift-insights/periodic-gathering-r9crl-vqcgk" Oct 30 23:41:28.349926 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.349769 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f464ef41-45c9-4ff4-b410-e2d9d6cdbdf5-config\") pod \"service-ca-operator-544d749c8c-fbgth\" (UID: \"f464ef41-45c9-4ff4-b410-e2d9d6cdbdf5\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-fbgth" Oct 30 23:41:28.349926 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.349831 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9700c98a-b858-45a5-9025-8d75f3c14044-cert\") pod \"ingress-canary-46zqw\" (UID: \"9700c98a-b858-45a5-9025-8d75f3c14044\") " pod="openshift-ingress-canary/ingress-canary-46zqw" Oct 30 23:41:28.349926 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.349861 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/953b9446-c289-48ca-b854-a14c70e5971b-host\") pod \"tuned-rsv7z\" (UID: \"953b9446-c289-48ca-b854-a14c70e5971b\") " pod="openshift-cluster-node-tuning-operator/tuned-rsv7z" Oct 30 23:41:28.349926 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.349892 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a78879b4-333e-4f1a-88bc-e96aee0e6d6b-service-ca-bundle\") pod \"periodic-gathering-r9crl-vqcgk\" (UID: \"a78879b4-333e-4f1a-88bc-e96aee0e6d6b\") " pod="openshift-insights/periodic-gathering-r9crl-vqcgk" Oct 30 23:41:28.349926 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.349897 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/4cf4abe6-da2b-41b6-9caa-9659d4f750fb-stats-auth\") pod \"router-default-66d97c9c97-qqxj5\" (UID: \"4cf4abe6-da2b-41b6-9caa-9659d4f750fb\") " pod="openshift-ingress/router-default-66d97c9c97-qqxj5" Oct 30 23:41:28.350281 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.349927 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/953b9446-c289-48ca-b854-a14c70e5971b-etc-sysctl-d\") pod \"tuned-rsv7z\" (UID: \"953b9446-c289-48ca-b854-a14c70e5971b\") " pod="openshift-cluster-node-tuning-operator/tuned-rsv7z" Oct 30 23:41:28.350281 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.349961 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b22fe41c-cf52-4e97-81f1-2f64bcc43640-cnibin\") pod \"multus-additional-cni-plugins-7mp5r\" (UID: \"b22fe41c-cf52-4e97-81f1-2f64bcc43640\") " pod="openshift-multus/multus-additional-cni-plugins-7mp5r" Oct 30 23:41:28.350281 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.349977 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0796261f-298e-4cbe-9b3e-b3dcf4528e07-device-dir\") pod \"aws-ebs-csi-driver-node-j7n2l\" (UID: \"0796261f-298e-4cbe-9b3e-b3dcf4528e07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j7n2l" Oct 30 23:41:28.350281 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.350004 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2fa7d24b-67b0-400f-94d4-903feb268782-hostroot\") pod \"multus-cb9vj\" (UID: \"2fa7d24b-67b0-400f-94d4-903feb268782\") " pod="openshift-multus/multus-cb9vj" Oct 30 23:41:28.350281 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.350025 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ec330d4f-2eea-476a-a7b8-60cd295fddd2-data-volume\") pod \"insights-runtime-extractor-sqv69\" (UID: \"ec330d4f-2eea-476a-a7b8-60cd295fddd2\") " pod="openshift-insights/insights-runtime-extractor-sqv69" Oct 30 23:41:28.350281 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.350047 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/29b8bd1a-beeb-495c-9f61-e3326866a1d7-metrics-tls\") pod \"dns-default-9l5zq\" (UID: \"29b8bd1a-beeb-495c-9f61-e3326866a1d7\") " pod="openshift-dns/dns-default-9l5zq" Oct 30 23:41:28.350281 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.350085 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ec330d4f-2eea-476a-a7b8-60cd295fddd2-data-volume\") pod \"insights-runtime-extractor-sqv69\" (UID: \"ec330d4f-2eea-476a-a7b8-60cd295fddd2\") " pod="openshift-insights/insights-runtime-extractor-sqv69" Oct 30 23:41:28.350281 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.350093 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9700c98a-b858-45a5-9025-8d75f3c14044-cert\") pod \"ingress-canary-46zqw\" (UID: \"9700c98a-b858-45a5-9025-8d75f3c14044\") " pod="openshift-ingress-canary/ingress-canary-46zqw" Oct 30 23:41:28.350281 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.350120 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xfx9\" (UniqueName: \"kubernetes.io/projected/3af6ccf2-97f1-4598-b8ef-12eac377133b-kube-api-access-9xfx9\") pod \"kas-connection-checker-685895c7c9-f7zjp\" (UID: \"3af6ccf2-97f1-4598-b8ef-12eac377133b\") " pod="kube-system/kas-connection-checker-685895c7c9-f7zjp" Oct 30 23:41:28.350281 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.350221 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b22fe41c-cf52-4e97-81f1-2f64bcc43640-tuning-conf-dir\") pod \"multus-additional-cni-plugins-7mp5r\" (UID: \"b22fe41c-cf52-4e97-81f1-2f64bcc43640\") " pod="openshift-multus/multus-additional-cni-plugins-7mp5r" Oct 30 23:41:28.350281 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.350259 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ece07686-4d8e-4068-9fb2-5889046cb73b-iptables-alerter-script\") pod \"iptables-alerter-57xrk\" (UID: \"ece07686-4d8e-4068-9fb2-5889046cb73b\") " pod="openshift-network-operator/iptables-alerter-57xrk" Oct 30 23:41:28.350641 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.350318 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0d4810de-8bd2-4270-b907-341d9f840bb8-env-overrides\") pod \"ovnkube-node-26v6x\" (UID: \"0d4810de-8bd2-4270-b907-341d9f840bb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-26v6x" Oct 30 23:41:28.350641 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.350390 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2fa7d24b-67b0-400f-94d4-903feb268782-host-var-lib-kubelet\") pod \"multus-cb9vj\" (UID: \"2fa7d24b-67b0-400f-94d4-903feb268782\") " pod="openshift-multus/multus-cb9vj" Oct 30 23:41:28.350641 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.350441 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/99f800b6-a38c-46ca-8234-0dbd3610c8fd-serviceca\") pod \"node-ca-7sbzf\" (UID: \"99f800b6-a38c-46ca-8234-0dbd3610c8fd\") " pod="openshift-image-registry/node-ca-7sbzf" Oct 30 23:41:28.350641 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.350471 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b22fe41c-cf52-4e97-81f1-2f64bcc43640-system-cni-dir\") pod \"multus-additional-cni-plugins-7mp5r\" (UID: \"b22fe41c-cf52-4e97-81f1-2f64bcc43640\") " pod="openshift-multus/multus-additional-cni-plugins-7mp5r" Oct 30 23:41:28.350641 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.350487 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/58f7cf25-f5d0-4481-a927-4a745b692f7a-bound-sa-token\") pod \"image-registry-544b44f8d-5p6kz\" (UID: \"58f7cf25-f5d0-4481-a927-4a745b692f7a\") " pod="openshift-image-registry/image-registry-544b44f8d-5p6kz" Oct 30 23:41:28.350641 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.350502 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a8e4c4db-197e-420c-9c60-4273cdf95377-agent-certs\") pod \"konnectivity-agent-9s2fx\" (UID: \"a8e4c4db-197e-420c-9c60-4273cdf95377\") " pod="kube-system/konnectivity-agent-9s2fx" Oct 30 23:41:28.350641 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.350514 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ece07686-4d8e-4068-9fb2-5889046cb73b-iptables-alerter-script\") pod \"iptables-alerter-57xrk\" (UID: \"ece07686-4d8e-4068-9fb2-5889046cb73b\") " pod="openshift-network-operator/iptables-alerter-57xrk" Oct 30 23:41:28.350641 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.350619 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/be5540b5-7b10-45b2-b40c-723efd9f1b01-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-vl8t5\" (UID: \"be5540b5-7b10-45b2-b40c-723efd9f1b01\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vl8t5" Oct 30 23:41:28.350641 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.350630 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0d4810de-8bd2-4270-b907-341d9f840bb8-env-overrides\") pod \"ovnkube-node-26v6x\" (UID: \"0d4810de-8bd2-4270-b907-341d9f840bb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-26v6x" Oct 30 23:41:28.351142 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.350630 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/99f800b6-a38c-46ca-8234-0dbd3610c8fd-serviceca\") pod \"node-ca-7sbzf\" (UID: \"99f800b6-a38c-46ca-8234-0dbd3610c8fd\") " pod="openshift-image-registry/node-ca-7sbzf" Oct 30 23:41:28.351142 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.350657 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0796261f-298e-4cbe-9b3e-b3dcf4528e07-registration-dir\") pod \"aws-ebs-csi-driver-node-j7n2l\" (UID: \"0796261f-298e-4cbe-9b3e-b3dcf4528e07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j7n2l" Oct 30 23:41:28.351142 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.350687 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0d4810de-8bd2-4270-b907-341d9f840bb8-etc-openvswitch\") pod \"ovnkube-node-26v6x\" (UID: \"0d4810de-8bd2-4270-b907-341d9f840bb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-26v6x" Oct 30 23:41:28.351142 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.350728 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/99f800b6-a38c-46ca-8234-0dbd3610c8fd-host\") pod \"node-ca-7sbzf\" (UID: \"99f800b6-a38c-46ca-8234-0dbd3610c8fd\") " pod="openshift-image-registry/node-ca-7sbzf" Oct 30 23:41:28.351142 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.350760 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/b53ad6ed-90fd-4a0f-bf54-88d416ba19e9-nginx-conf\") pod \"networking-console-plugin-86c478bb88-rxpst\" (UID: \"b53ad6ed-90fd-4a0f-bf54-88d416ba19e9\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-rxpst" Oct 30 23:41:28.351142 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.350788 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b22fe41c-cf52-4e97-81f1-2f64bcc43640-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-7mp5r\" (UID: \"b22fe41c-cf52-4e97-81f1-2f64bcc43640\") " pod="openshift-multus/multus-additional-cni-plugins-7mp5r" Oct 30 23:41:28.351142 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.350815 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btgs8\" (UniqueName: \"kubernetes.io/projected/b22fe41c-cf52-4e97-81f1-2f64bcc43640-kube-api-access-btgs8\") pod \"multus-additional-cni-plugins-7mp5r\" (UID: \"b22fe41c-cf52-4e97-81f1-2f64bcc43640\") " pod="openshift-multus/multus-additional-cni-plugins-7mp5r" Oct 30 23:41:28.351142 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.350820 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/be5540b5-7b10-45b2-b40c-723efd9f1b01-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-vl8t5\" (UID: \"be5540b5-7b10-45b2-b40c-723efd9f1b01\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vl8t5" Oct 30 23:41:28.351142 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.350842 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0796261f-298e-4cbe-9b3e-b3dcf4528e07-sys-fs\") pod \"aws-ebs-csi-driver-node-j7n2l\" (UID: \"0796261f-298e-4cbe-9b3e-b3dcf4528e07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j7n2l" Oct 30 23:41:28.351142 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.350850 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a8e4c4db-197e-420c-9c60-4273cdf95377-agent-certs\") pod \"konnectivity-agent-9s2fx\" (UID: \"a8e4c4db-197e-420c-9c60-4273cdf95377\") " pod="kube-system/konnectivity-agent-9s2fx" Oct 30 23:41:28.351142 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.350865 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0d4810de-8bd2-4270-b907-341d9f840bb8-host-kubelet\") pod \"ovnkube-node-26v6x\" (UID: \"0d4810de-8bd2-4270-b907-341d9f840bb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-26v6x" Oct 30 23:41:28.351142 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.350913 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/953b9446-c289-48ca-b854-a14c70e5971b-tmp\") pod \"tuned-rsv7z\" (UID: \"953b9446-c289-48ca-b854-a14c70e5971b\") " pod="openshift-cluster-node-tuning-operator/tuned-rsv7z" Oct 30 23:41:28.351142 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.350977 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/b53ad6ed-90fd-4a0f-bf54-88d416ba19e9-nginx-conf\") pod \"networking-console-plugin-86c478bb88-rxpst\" (UID: \"b53ad6ed-90fd-4a0f-bf54-88d416ba19e9\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-rxpst" Oct 30 23:41:28.351142 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.350993 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/953b9446-c289-48ca-b854-a14c70e5971b-tmp\") pod \"tuned-rsv7z\" (UID: \"953b9446-c289-48ca-b854-a14c70e5971b\") " pod="openshift-cluster-node-tuning-operator/tuned-rsv7z" Oct 30 23:41:28.351142 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.351019 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlhgb\" (UniqueName: \"kubernetes.io/projected/99f800b6-a38c-46ca-8234-0dbd3610c8fd-kube-api-access-mlhgb\") pod \"node-ca-7sbzf\" (UID: \"99f800b6-a38c-46ca-8234-0dbd3610c8fd\") " pod="openshift-image-registry/node-ca-7sbzf" Oct 30 23:41:28.351142 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.351027 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b22fe41c-cf52-4e97-81f1-2f64bcc43640-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-7mp5r\" (UID: \"b22fe41c-cf52-4e97-81f1-2f64bcc43640\") " pod="openshift-multus/multus-additional-cni-plugins-7mp5r" Oct 30 23:41:28.351142 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.351047 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/97da2086-b00d-4f03-b502-56de5ee3d90d-kubelet-config\") pod \"global-pull-secret-syncer-7wrh7\" (UID: \"97da2086-b00d-4f03-b502-56de5ee3d90d\") " pod="kube-system/global-pull-secret-syncer-7wrh7" Oct 30 23:41:28.351142 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.351072 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2fa7d24b-67b0-400f-94d4-903feb268782-multus-socket-dir-parent\") pod \"multus-cb9vj\" (UID: \"2fa7d24b-67b0-400f-94d4-903feb268782\") " pod="openshift-multus/multus-cb9vj" Oct 30 23:41:28.351829 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.351212 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nsggg\" (UniqueName: \"kubernetes.io/projected/145dd323-9370-4b9d-a96e-8b93d0a47a7c-kube-api-access-nsggg\") pod \"network-metrics-daemon-pt58s\" (UID: \"145dd323-9370-4b9d-a96e-8b93d0a47a7c\") " pod="openshift-multus/network-metrics-daemon-pt58s" Oct 30 23:41:28.351829 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.351232 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vj8vx\" (UniqueName: \"kubernetes.io/projected/29b8bd1a-beeb-495c-9f61-e3326866a1d7-kube-api-access-vj8vx\") pod \"dns-default-9l5zq\" (UID: \"29b8bd1a-beeb-495c-9f61-e3326866a1d7\") " pod="openshift-dns/dns-default-9l5zq" Oct 30 23:41:28.351829 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.351249 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2cxtv\" (UniqueName: \"kubernetes.io/projected/ece07686-4d8e-4068-9fb2-5889046cb73b-kube-api-access-2cxtv\") pod \"iptables-alerter-57xrk\" (UID: \"ece07686-4d8e-4068-9fb2-5889046cb73b\") " pod="openshift-network-operator/iptables-alerter-57xrk" Oct 30 23:41:28.351829 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.351266 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/953b9446-c289-48ca-b854-a14c70e5971b-etc-kubernetes\") pod \"tuned-rsv7z\" (UID: \"953b9446-c289-48ca-b854-a14c70e5971b\") " pod="openshift-cluster-node-tuning-operator/tuned-rsv7z" Oct 30 23:41:28.351829 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.351282 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6382dfb5-a363-47a2-bc79-0b96f7bec0c3-serving-cert\") pod \"console-operator-7cd56b8dbc-pqt2c\" (UID: \"6382dfb5-a363-47a2-bc79-0b96f7bec0c3\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-pqt2c" Oct 30 23:41:28.351829 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.351304 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/145dd323-9370-4b9d-a96e-8b93d0a47a7c-metrics-certs\") pod \"network-metrics-daemon-pt58s\" (UID: \"145dd323-9370-4b9d-a96e-8b93d0a47a7c\") " pod="openshift-multus/network-metrics-daemon-pt58s" Oct 30 23:41:28.351829 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.351373 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/58f7cf25-f5d0-4481-a927-4a745b692f7a-registry-certificates\") pod \"image-registry-544b44f8d-5p6kz\" (UID: \"58f7cf25-f5d0-4481-a927-4a745b692f7a\") " pod="openshift-image-registry/image-registry-544b44f8d-5p6kz" Oct 30 23:41:28.351829 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.351460 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45fmr\" (UniqueName: \"kubernetes.io/projected/be5540b5-7b10-45b2-b40c-723efd9f1b01-kube-api-access-45fmr\") pod \"cluster-samples-operator-6ddb6c8b94-vl8t5\" (UID: \"be5540b5-7b10-45b2-b40c-723efd9f1b01\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vl8t5" Oct 30 23:41:28.351829 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.351464 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6382dfb5-a363-47a2-bc79-0b96f7bec0c3-serving-cert\") pod \"console-operator-7cd56b8dbc-pqt2c\" (UID: \"6382dfb5-a363-47a2-bc79-0b96f7bec0c3\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-pqt2c" Oct 30 23:41:28.351829 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.351498 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ece07686-4d8e-4068-9fb2-5889046cb73b-host-slash\") pod \"iptables-alerter-57xrk\" (UID: \"ece07686-4d8e-4068-9fb2-5889046cb73b\") " pod="openshift-network-operator/iptables-alerter-57xrk" Oct 30 23:41:28.351829 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.351522 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0d4810de-8bd2-4270-b907-341d9f840bb8-run-openvswitch\") pod \"ovnkube-node-26v6x\" (UID: \"0d4810de-8bd2-4270-b907-341d9f840bb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-26v6x" Oct 30 23:41:28.351829 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.351537 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/953b9446-c289-48ca-b854-a14c70e5971b-sys\") pod \"tuned-rsv7z\" (UID: \"953b9446-c289-48ca-b854-a14c70e5971b\") " pod="openshift-cluster-node-tuning-operator/tuned-rsv7z" Oct 30 23:41:28.351829 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.351580 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9zsm\" (UniqueName: \"kubernetes.io/projected/aa90814d-5b4d-4039-917d-76610b1f88e4-kube-api-access-q9zsm\") pod \"migrator-66df5d496b-8khpq\" (UID: \"aa90814d-5b4d-4039-917d-76610b1f88e4\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-8khpq" Oct 30 23:41:28.351829 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.351594 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/58f7cf25-f5d0-4481-a927-4a745b692f7a-registry-certificates\") pod \"image-registry-544b44f8d-5p6kz\" (UID: \"58f7cf25-f5d0-4481-a927-4a745b692f7a\") " pod="openshift-image-registry/image-registry-544b44f8d-5p6kz" Oct 30 23:41:28.351829 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.351608 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztdxk\" (UniqueName: \"kubernetes.io/projected/f464ef41-45c9-4ff4-b410-e2d9d6cdbdf5-kube-api-access-ztdxk\") pod \"service-ca-operator-544d749c8c-fbgth\" (UID: \"f464ef41-45c9-4ff4-b410-e2d9d6cdbdf5\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-fbgth" Oct 30 23:41:28.351829 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.351628 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/10774606-8e0c-460b-bd5f-4cb6d121b30b-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-9mqqt\" (UID: \"10774606-8e0c-460b-bd5f-4cb6d121b30b\") " pod="openshift-insights/insights-operator-7c4986d449-9mqqt" Oct 30 23:41:28.351829 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.351745 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2fa7d24b-67b0-400f-94d4-903feb268782-cnibin\") pod \"multus-cb9vj\" (UID: \"2fa7d24b-67b0-400f-94d4-903feb268782\") " pod="openshift-multus/multus-cb9vj" Oct 30 23:41:28.351829 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.351772 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4cf4abe6-da2b-41b6-9caa-9659d4f750fb-metrics-certs\") pod \"router-default-66d97c9c97-qqxj5\" (UID: \"4cf4abe6-da2b-41b6-9caa-9659d4f750fb\") " pod="openshift-ingress/router-default-66d97c9c97-qqxj5" Oct 30 23:41:28.351829 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.351823 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7p9rj\" (UniqueName: \"kubernetes.io/projected/f8b33ecc-59a4-4680-9521-299648ea5055-kube-api-access-7p9rj\") pod \"node-resolver-tfnr7\" (UID: \"f8b33ecc-59a4-4680-9521-299648ea5055\") " pod="openshift-dns/node-resolver-tfnr7" Oct 30 23:41:28.352692 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.351861 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkthh\" (UniqueName: \"kubernetes.io/projected/17461b46-7ebf-49b7-afaf-12e20e3994a3-kube-api-access-xkthh\") pod \"image-registry-dd954f877-x9h9g\" (UID: \"17461b46-7ebf-49b7-afaf-12e20e3994a3\") " pod="openshift-image-registry/image-registry-dd954f877-x9h9g" Oct 30 23:41:28.352692 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.351878 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/97da2086-b00d-4f03-b502-56de5ee3d90d-dbus\") pod \"global-pull-secret-syncer-7wrh7\" (UID: \"97da2086-b00d-4f03-b502-56de5ee3d90d\") " pod="kube-system/global-pull-secret-syncer-7wrh7" Oct 30 23:41:28.352692 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.351893 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/58f7cf25-f5d0-4481-a927-4a745b692f7a-registry-tls\") pod \"image-registry-544b44f8d-5p6kz\" (UID: \"58f7cf25-f5d0-4481-a927-4a745b692f7a\") " pod="openshift-image-registry/image-registry-544b44f8d-5p6kz" Oct 30 23:41:28.352692 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.351913 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0d4810de-8bd2-4270-b907-341d9f840bb8-host-run-netns\") pod \"ovnkube-node-26v6x\" (UID: \"0d4810de-8bd2-4270-b907-341d9f840bb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-26v6x" Oct 30 23:41:28.352692 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.351893 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/10774606-8e0c-460b-bd5f-4cb6d121b30b-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-9mqqt\" (UID: \"10774606-8e0c-460b-bd5f-4cb6d121b30b\") " pod="openshift-insights/insights-operator-7c4986d449-9mqqt" Oct 30 23:41:28.352692 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.351945 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/953b9446-c289-48ca-b854-a14c70e5971b-etc-systemd\") pod \"tuned-rsv7z\" (UID: \"953b9446-c289-48ca-b854-a14c70e5971b\") " pod="openshift-cluster-node-tuning-operator/tuned-rsv7z" Oct 30 23:41:28.352692 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.352016 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2fa7d24b-67b0-400f-94d4-903feb268782-host-var-lib-cni-multus\") pod \"multus-cb9vj\" (UID: \"2fa7d24b-67b0-400f-94d4-903feb268782\") " pod="openshift-multus/multus-cb9vj" Oct 30 23:41:28.352692 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.352065 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2fa7d24b-67b0-400f-94d4-903feb268782-multus-conf-dir\") pod \"multus-cb9vj\" (UID: \"2fa7d24b-67b0-400f-94d4-903feb268782\") " pod="openshift-multus/multus-cb9vj" Oct 30 23:41:28.352692 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.352102 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtfn9\" (UniqueName: \"kubernetes.io/projected/4cf4abe6-da2b-41b6-9caa-9659d4f750fb-kube-api-access-rtfn9\") pod \"router-default-66d97c9c97-qqxj5\" (UID: \"4cf4abe6-da2b-41b6-9caa-9659d4f750fb\") " pod="openshift-ingress/router-default-66d97c9c97-qqxj5" Oct 30 23:41:28.352692 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.352139 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67888\" (UniqueName: \"kubernetes.io/projected/a4f35692-894d-4d41-8b9f-3f5ba3c529b1-kube-api-access-67888\") pod \"kas-connection-checker-685895c7c9-l4d2m\" (UID: \"a4f35692-894d-4d41-8b9f-3f5ba3c529b1\") " pod="kube-system/kas-connection-checker-685895c7c9-l4d2m" Oct 30 23:41:28.352692 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.352167 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0d4810de-8bd2-4270-b907-341d9f840bb8-run-ovn\") pod \"ovnkube-node-26v6x\" (UID: \"0d4810de-8bd2-4270-b907-341d9f840bb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-26v6x" Oct 30 23:41:28.352692 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.352229 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2fa7d24b-67b0-400f-94d4-903feb268782-host-run-k8s-cni-cncf-io\") pod \"multus-cb9vj\" (UID: \"2fa7d24b-67b0-400f-94d4-903feb268782\") " pod="openshift-multus/multus-cb9vj" Oct 30 23:41:28.352692 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.352256 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/97da2086-b00d-4f03-b502-56de5ee3d90d-original-pull-secret\") pod \"global-pull-secret-syncer-7wrh7\" (UID: \"97da2086-b00d-4f03-b502-56de5ee3d90d\") " pod="kube-system/global-pull-secret-syncer-7wrh7" Oct 30 23:41:28.352692 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.352300 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6382dfb5-a363-47a2-bc79-0b96f7bec0c3-trusted-ca\") pod \"console-operator-7cd56b8dbc-pqt2c\" (UID: \"6382dfb5-a363-47a2-bc79-0b96f7bec0c3\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-pqt2c" Oct 30 23:41:28.352692 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.352433 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b22fe41c-cf52-4e97-81f1-2f64bcc43640-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-7mp5r\" (UID: \"b22fe41c-cf52-4e97-81f1-2f64bcc43640\") " pod="openshift-multus/multus-additional-cni-plugins-7mp5r" Oct 30 23:41:28.352692 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.352466 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/58f7cf25-f5d0-4481-a927-4a745b692f7a-image-registry-private-configuration\") pod \"image-registry-544b44f8d-5p6kz\" (UID: \"58f7cf25-f5d0-4481-a927-4a745b692f7a\") " pod="openshift-image-registry/image-registry-544b44f8d-5p6kz" Oct 30 23:41:28.352692 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.352489 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/97da2086-b00d-4f03-b502-56de5ee3d90d-original-pull-secret\") pod \"global-pull-secret-syncer-7wrh7\" (UID: \"97da2086-b00d-4f03-b502-56de5ee3d90d\") " pod="kube-system/global-pull-secret-syncer-7wrh7" Oct 30 23:41:28.352692 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.352495 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/58f7cf25-f5d0-4481-a927-4a745b692f7a-installation-pull-secrets\") pod \"image-registry-544b44f8d-5p6kz\" (UID: \"58f7cf25-f5d0-4481-a927-4a745b692f7a\") " pod="openshift-image-registry/image-registry-544b44f8d-5p6kz" Oct 30 23:41:28.352692 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.352611 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqmnp\" (UniqueName: \"kubernetes.io/projected/7728abfe-1747-422e-b8e2-c91c9de83d09-kube-api-access-jqmnp\") pod \"network-check-target-kgdzs\" (UID: \"7728abfe-1747-422e-b8e2-c91c9de83d09\") " pod="openshift-network-diagnostics/network-check-target-kgdzs" Oct 30 23:41:28.352692 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.352661 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/10774606-8e0c-460b-bd5f-4cb6d121b30b-tmp\") pod \"insights-operator-7c4986d449-9mqqt\" (UID: \"10774606-8e0c-460b-bd5f-4cb6d121b30b\") " pod="openshift-insights/insights-operator-7c4986d449-9mqqt" Oct 30 23:41:28.353519 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.352717 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/58f7cf25-f5d0-4481-a927-4a745b692f7a-image-registry-private-configuration\") pod \"image-registry-544b44f8d-5p6kz\" (UID: \"58f7cf25-f5d0-4481-a927-4a745b692f7a\") " pod="openshift-image-registry/image-registry-544b44f8d-5p6kz" Oct 30 23:41:28.353519 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.352719 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6382dfb5-a363-47a2-bc79-0b96f7bec0c3-trusted-ca\") pod \"console-operator-7cd56b8dbc-pqt2c\" (UID: \"6382dfb5-a363-47a2-bc79-0b96f7bec0c3\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-pqt2c" Oct 30 23:41:28.353519 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.352747 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b22fe41c-cf52-4e97-81f1-2f64bcc43640-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-7mp5r\" (UID: \"b22fe41c-cf52-4e97-81f1-2f64bcc43640\") " pod="openshift-multus/multus-additional-cni-plugins-7mp5r" Oct 30 23:41:28.353519 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.352757 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/58f7cf25-f5d0-4481-a927-4a745b692f7a-installation-pull-secrets\") pod \"image-registry-544b44f8d-5p6kz\" (UID: \"58f7cf25-f5d0-4481-a927-4a745b692f7a\") " pod="openshift-image-registry/image-registry-544b44f8d-5p6kz" Oct 30 23:41:28.353519 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.352794 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n65gn\" (UniqueName: \"kubernetes.io/projected/10774606-8e0c-460b-bd5f-4cb6d121b30b-kube-api-access-n65gn\") pod \"insights-operator-7c4986d449-9mqqt\" (UID: \"10774606-8e0c-460b-bd5f-4cb6d121b30b\") " pod="openshift-insights/insights-operator-7c4986d449-9mqqt" Oct 30 23:41:28.353519 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.352813 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/10774606-8e0c-460b-bd5f-4cb6d121b30b-tmp\") pod \"insights-operator-7c4986d449-9mqqt\" (UID: \"10774606-8e0c-460b-bd5f-4cb6d121b30b\") " pod="openshift-insights/insights-operator-7c4986d449-9mqqt" Oct 30 23:41:28.353519 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.352824 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2fa7d24b-67b0-400f-94d4-903feb268782-host-run-netns\") pod \"multus-cb9vj\" (UID: \"2fa7d24b-67b0-400f-94d4-903feb268782\") " pod="openshift-multus/multus-cb9vj" Oct 30 23:41:28.353519 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.352864 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/4cf4abe6-da2b-41b6-9caa-9659d4f750fb-default-certificate\") pod \"router-default-66d97c9c97-qqxj5\" (UID: \"4cf4abe6-da2b-41b6-9caa-9659d4f750fb\") " pod="openshift-ingress/router-default-66d97c9c97-qqxj5" Oct 30 23:41:28.353519 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.352908 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0796261f-298e-4cbe-9b3e-b3dcf4528e07-kubelet-dir\") pod \"aws-ebs-csi-driver-node-j7n2l\" (UID: \"0796261f-298e-4cbe-9b3e-b3dcf4528e07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j7n2l" Oct 30 23:41:28.353519 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.352938 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0d4810de-8bd2-4270-b907-341d9f840bb8-host-cni-netd\") pod \"ovnkube-node-26v6x\" (UID: \"0d4810de-8bd2-4270-b907-341d9f840bb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-26v6x" Oct 30 23:41:28.353519 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.353039 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0d4810de-8bd2-4270-b907-341d9f840bb8-ovn-node-metrics-cert\") pod \"ovnkube-node-26v6x\" (UID: \"0d4810de-8bd2-4270-b907-341d9f840bb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-26v6x" Oct 30 23:41:28.353519 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.353088 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/953b9446-c289-48ca-b854-a14c70e5971b-etc-sysconfig\") pod \"tuned-rsv7z\" (UID: \"953b9446-c289-48ca-b854-a14c70e5971b\") " pod="openshift-cluster-node-tuning-operator/tuned-rsv7z" Oct 30 23:41:28.353519 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.353115 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/4cf4abe6-da2b-41b6-9caa-9659d4f750fb-default-certificate\") pod \"router-default-66d97c9c97-qqxj5\" (UID: \"4cf4abe6-da2b-41b6-9caa-9659d4f750fb\") " pod="openshift-ingress/router-default-66d97c9c97-qqxj5" Oct 30 23:41:28.353519 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.353188 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6382dfb5-a363-47a2-bc79-0b96f7bec0c3-config\") pod \"console-operator-7cd56b8dbc-pqt2c\" (UID: \"6382dfb5-a363-47a2-bc79-0b96f7bec0c3\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-pqt2c" Oct 30 23:41:28.353519 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.353199 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0d4810de-8bd2-4270-b907-341d9f840bb8-ovn-node-metrics-cert\") pod \"ovnkube-node-26v6x\" (UID: \"0d4810de-8bd2-4270-b907-341d9f840bb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-26v6x" Oct 30 23:41:28.353519 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.353241 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sx9pp\" (UniqueName: \"kubernetes.io/projected/58f7cf25-f5d0-4481-a927-4a745b692f7a-kube-api-access-sx9pp\") pod \"image-registry-544b44f8d-5p6kz\" (UID: \"58f7cf25-f5d0-4481-a927-4a745b692f7a\") " pod="openshift-image-registry/image-registry-544b44f8d-5p6kz" Oct 30 23:41:28.353519 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.353280 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/840aaae0-a215-4ccd-b420-d6f5723b65ae-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-9qs84\" (UID: \"840aaae0-a215-4ccd-b420-d6f5723b65ae\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-9qs84" Oct 30 23:41:28.353519 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.353307 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ssd98\" (UniqueName: \"kubernetes.io/projected/2fa7d24b-67b0-400f-94d4-903feb268782-kube-api-access-ssd98\") pod \"multus-cb9vj\" (UID: \"2fa7d24b-67b0-400f-94d4-903feb268782\") " pod="openshift-multus/multus-cb9vj" Oct 30 23:41:28.353519 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.353347 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfwlk\" (UniqueName: \"kubernetes.io/projected/64ae3ff5-44eb-456b-948a-c1a8320e6b07-kube-api-access-cfwlk\") pod \"kas-connection-checker-685895c7c9-lxdsf\" (UID: \"64ae3ff5-44eb-456b-948a-c1a8320e6b07\") " pod="kube-system/kas-connection-checker-685895c7c9-lxdsf" Oct 30 23:41:28.353519 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.353373 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6382dfb5-a363-47a2-bc79-0b96f7bec0c3-config\") pod \"console-operator-7cd56b8dbc-pqt2c\" (UID: \"6382dfb5-a363-47a2-bc79-0b96f7bec0c3\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-pqt2c" Oct 30 23:41:28.353519 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.353444 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/29b8bd1a-beeb-495c-9f61-e3326866a1d7-config-volume\") pod \"dns-default-9l5zq\" (UID: \"29b8bd1a-beeb-495c-9f61-e3326866a1d7\") " pod="openshift-dns/dns-default-9l5zq" Oct 30 23:41:28.353519 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.353499 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/840aaae0-a215-4ccd-b420-d6f5723b65ae-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-9qs84\" (UID: \"840aaae0-a215-4ccd-b420-d6f5723b65ae\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-9qs84" Oct 30 23:41:28.353519 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.353512 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/17461b46-7ebf-49b7-afaf-12e20e3994a3-installation-pull-secrets\") pod \"image-registry-dd954f877-x9h9g\" (UID: \"17461b46-7ebf-49b7-afaf-12e20e3994a3\") " pod="openshift-image-registry/image-registry-dd954f877-x9h9g" Oct 30 23:41:28.355349 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.353554 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/17461b46-7ebf-49b7-afaf-12e20e3994a3-bound-sa-token\") pod \"image-registry-dd954f877-x9h9g\" (UID: \"17461b46-7ebf-49b7-afaf-12e20e3994a3\") " pod="openshift-image-registry/image-registry-dd954f877-x9h9g" Oct 30 23:41:28.355349 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.353594 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xn7l\" (UniqueName: \"kubernetes.io/projected/9700c98a-b858-45a5-9025-8d75f3c14044-kube-api-access-5xn7l\") pod \"ingress-canary-46zqw\" (UID: \"9700c98a-b858-45a5-9025-8d75f3c14044\") " pod="openshift-ingress-canary/ingress-canary-46zqw" Oct 30 23:41:28.355349 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.353628 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/10774606-8e0c-460b-bd5f-4cb6d121b30b-snapshots\") pod \"insights-operator-7c4986d449-9mqqt\" (UID: \"10774606-8e0c-460b-bd5f-4cb6d121b30b\") " pod="openshift-insights/insights-operator-7c4986d449-9mqqt" Oct 30 23:41:28.355349 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.353647 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/29b8bd1a-beeb-495c-9f61-e3326866a1d7-config-volume\") pod \"dns-default-9l5zq\" (UID: \"29b8bd1a-beeb-495c-9f61-e3326866a1d7\") " pod="openshift-dns/dns-default-9l5zq" Oct 30 23:41:28.355349 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.353655 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0796261f-298e-4cbe-9b3e-b3dcf4528e07-socket-dir\") pod \"aws-ebs-csi-driver-node-j7n2l\" (UID: \"0796261f-298e-4cbe-9b3e-b3dcf4528e07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j7n2l" Oct 30 23:41:28.355349 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.353696 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/17461b46-7ebf-49b7-afaf-12e20e3994a3-image-registry-private-configuration\") pod \"image-registry-dd954f877-x9h9g\" (UID: \"17461b46-7ebf-49b7-afaf-12e20e3994a3\") " pod="openshift-image-registry/image-registry-dd954f877-x9h9g" Oct 30 23:41:28.355349 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.353727 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0d4810de-8bd2-4270-b907-341d9f840bb8-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-26v6x\" (UID: \"0d4810de-8bd2-4270-b907-341d9f840bb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-26v6x" Oct 30 23:41:28.355349 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.353732 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/10774606-8e0c-460b-bd5f-4cb6d121b30b-snapshots\") pod \"insights-operator-7c4986d449-9mqqt\" (UID: \"10774606-8e0c-460b-bd5f-4cb6d121b30b\") " pod="openshift-insights/insights-operator-7c4986d449-9mqqt" Oct 30 23:41:28.355349 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.353733 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/17461b46-7ebf-49b7-afaf-12e20e3994a3-installation-pull-secrets\") pod \"image-registry-dd954f877-x9h9g\" (UID: \"17461b46-7ebf-49b7-afaf-12e20e3994a3\") " pod="openshift-image-registry/image-registry-dd954f877-x9h9g" Oct 30 23:41:28.355349 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.353754 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48m7s\" (UniqueName: \"kubernetes.io/projected/0d4810de-8bd2-4270-b907-341d9f840bb8-kube-api-access-48m7s\") pod \"ovnkube-node-26v6x\" (UID: \"0d4810de-8bd2-4270-b907-341d9f840bb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-26v6x" Oct 30 23:41:28.355349 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.353785 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4n58\" (UniqueName: \"kubernetes.io/projected/ec330d4f-2eea-476a-a7b8-60cd295fddd2-kube-api-access-z4n58\") pod \"insights-runtime-extractor-sqv69\" (UID: \"ec330d4f-2eea-476a-a7b8-60cd295fddd2\") " pod="openshift-insights/insights-runtime-extractor-sqv69" Oct 30 23:41:28.355349 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.353806 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9l2q\" (UniqueName: \"kubernetes.io/projected/f8eb4b4c-8225-436b-b1e8-7775414a1ca3-kube-api-access-k9l2q\") pod \"volume-data-source-validator-58c9f5f46d-tcxmv\" (UID: \"f8eb4b4c-8225-436b-b1e8-7775414a1ca3\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-tcxmv" Oct 30 23:41:28.355349 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.353825 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f464ef41-45c9-4ff4-b410-e2d9d6cdbdf5-serving-cert\") pod \"service-ca-operator-544d749c8c-fbgth\" (UID: \"f464ef41-45c9-4ff4-b410-e2d9d6cdbdf5\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-fbgth" Oct 30 23:41:28.355349 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.353841 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0796261f-298e-4cbe-9b3e-b3dcf4528e07-etc-selinux\") pod \"aws-ebs-csi-driver-node-j7n2l\" (UID: \"0796261f-298e-4cbe-9b3e-b3dcf4528e07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j7n2l" Oct 30 23:41:28.355349 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.353857 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0d4810de-8bd2-4270-b907-341d9f840bb8-systemd-units\") pod \"ovnkube-node-26v6x\" (UID: \"0d4810de-8bd2-4270-b907-341d9f840bb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-26v6x" Oct 30 23:41:28.355349 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.353876 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/953b9446-c289-48ca-b854-a14c70e5971b-etc-tuned\") pod \"tuned-rsv7z\" (UID: \"953b9446-c289-48ca-b854-a14c70e5971b\") " pod="openshift-cluster-node-tuning-operator/tuned-rsv7z" Oct 30 23:41:28.355349 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.353893 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/10774606-8e0c-460b-bd5f-4cb6d121b30b-service-ca-bundle\") pod \"insights-operator-7c4986d449-9mqqt\" (UID: \"10774606-8e0c-460b-bd5f-4cb6d121b30b\") " pod="openshift-insights/insights-operator-7c4986d449-9mqqt" Oct 30 23:41:28.355349 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.353902 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/17461b46-7ebf-49b7-afaf-12e20e3994a3-image-registry-private-configuration\") pod \"image-registry-dd954f877-x9h9g\" (UID: \"17461b46-7ebf-49b7-afaf-12e20e3994a3\") " pod="openshift-image-registry/image-registry-dd954f877-x9h9g" Oct 30 23:41:28.355349 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.353927 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/03211c6a-9761-419c-839b-21ca413d5bb2-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-httg9\" (UID: \"03211c6a-9761-419c-839b-21ca413d5bb2\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-httg9" Oct 30 23:41:28.355349 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.354000 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/10774606-8e0c-460b-bd5f-4cb6d121b30b-service-ca-bundle\") pod \"insights-operator-7c4986d449-9mqqt\" (UID: \"10774606-8e0c-460b-bd5f-4cb6d121b30b\") " pod="openshift-insights/insights-operator-7c4986d449-9mqqt" Oct 30 23:41:28.355349 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.354019 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f464ef41-45c9-4ff4-b410-e2d9d6cdbdf5-serving-cert\") pod \"service-ca-operator-544d749c8c-fbgth\" (UID: \"f464ef41-45c9-4ff4-b410-e2d9d6cdbdf5\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-fbgth" Oct 30 23:41:28.355349 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.354042 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/953b9446-c289-48ca-b854-a14c70e5971b-etc-tuned\") pod \"tuned-rsv7z\" (UID: \"953b9446-c289-48ca-b854-a14c70e5971b\") " pod="openshift-cluster-node-tuning-operator/tuned-rsv7z" Oct 30 23:41:28.355349 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.354057 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/17461b46-7ebf-49b7-afaf-12e20e3994a3-ca-trust-extracted\") pod \"image-registry-dd954f877-x9h9g\" (UID: \"17461b46-7ebf-49b7-afaf-12e20e3994a3\") " pod="openshift-image-registry/image-registry-dd954f877-x9h9g" Oct 30 23:41:28.355349 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.354121 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/17461b46-7ebf-49b7-afaf-12e20e3994a3-trusted-ca\") pod \"image-registry-dd954f877-x9h9g\" (UID: \"17461b46-7ebf-49b7-afaf-12e20e3994a3\") " pod="openshift-image-registry/image-registry-dd954f877-x9h9g" Oct 30 23:41:28.355349 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.354145 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/17461b46-7ebf-49b7-afaf-12e20e3994a3-ca-trust-extracted\") pod \"image-registry-dd954f877-x9h9g\" (UID: \"17461b46-7ebf-49b7-afaf-12e20e3994a3\") " pod="openshift-image-registry/image-registry-dd954f877-x9h9g" Oct 30 23:41:28.355349 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.354151 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0d4810de-8bd2-4270-b907-341d9f840bb8-ovnkube-config\") pod \"ovnkube-node-26v6x\" (UID: \"0d4810de-8bd2-4270-b907-341d9f840bb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-26v6x" Oct 30 23:41:28.355349 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.354184 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2fa7d24b-67b0-400f-94d4-903feb268782-system-cni-dir\") pod \"multus-cb9vj\" (UID: \"2fa7d24b-67b0-400f-94d4-903feb268782\") " pod="openshift-multus/multus-cb9vj" Oct 30 23:41:28.355349 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.354209 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2fa7d24b-67b0-400f-94d4-903feb268782-multus-daemon-config\") pod \"multus-cb9vj\" (UID: \"2fa7d24b-67b0-400f-94d4-903feb268782\") " pod="openshift-multus/multus-cb9vj" Oct 30 23:41:28.355349 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.354234 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8b33ecc-59a4-4680-9521-299648ea5055-tmp-dir\") pod \"node-resolver-tfnr7\" (UID: \"f8b33ecc-59a4-4680-9521-299648ea5055\") " pod="openshift-dns/node-resolver-tfnr7" Oct 30 23:41:28.355349 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.354253 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4cf4abe6-da2b-41b6-9caa-9659d4f750fb-service-ca-bundle\") pod \"router-default-66d97c9c97-qqxj5\" (UID: \"4cf4abe6-da2b-41b6-9caa-9659d4f750fb\") " pod="openshift-ingress/router-default-66d97c9c97-qqxj5" Oct 30 23:41:28.355349 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.354291 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vv9dt\" (UniqueName: \"kubernetes.io/projected/27ccca21-5e99-485f-8b5c-ca8ec5177668-kube-api-access-vv9dt\") pod \"network-check-source-6f78d59f99-xrk8k\" (UID: \"27ccca21-5e99-485f-8b5c-ca8ec5177668\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-xrk8k" Oct 30 23:41:28.355349 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.354345 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vsfpv\" (UniqueName: \"kubernetes.io/projected/840aaae0-a215-4ccd-b420-d6f5723b65ae-kube-api-access-vsfpv\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-9qs84\" (UID: \"840aaae0-a215-4ccd-b420-d6f5723b65ae\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-9qs84" Oct 30 23:41:28.355349 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.354368 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0d4810de-8bd2-4270-b907-341d9f840bb8-log-socket\") pod \"ovnkube-node-26v6x\" (UID: \"0d4810de-8bd2-4270-b907-341d9f840bb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-26v6x" Oct 30 23:41:28.355349 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.354378 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8b33ecc-59a4-4680-9521-299648ea5055-tmp-dir\") pod \"node-resolver-tfnr7\" (UID: \"f8b33ecc-59a4-4680-9521-299648ea5055\") " pod="openshift-dns/node-resolver-tfnr7" Oct 30 23:41:28.355349 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.354379 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0d4810de-8bd2-4270-b907-341d9f840bb8-ovnkube-config\") pod \"ovnkube-node-26v6x\" (UID: \"0d4810de-8bd2-4270-b907-341d9f840bb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-26v6x" Oct 30 23:41:28.355349 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.354386 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gtt94\" (UniqueName: \"kubernetes.io/projected/953b9446-c289-48ca-b854-a14c70e5971b-kube-api-access-gtt94\") pod \"tuned-rsv7z\" (UID: \"953b9446-c289-48ca-b854-a14c70e5971b\") " pod="openshift-cluster-node-tuning-operator/tuned-rsv7z" Oct 30 23:41:28.355349 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.354405 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2fa7d24b-67b0-400f-94d4-903feb268782-multus-daemon-config\") pod \"multus-cb9vj\" (UID: \"2fa7d24b-67b0-400f-94d4-903feb268782\") " pod="openshift-multus/multus-cb9vj" Oct 30 23:41:28.355349 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.354440 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/17461b46-7ebf-49b7-afaf-12e20e3994a3-trusted-ca\") pod \"image-registry-dd954f877-x9h9g\" (UID: \"17461b46-7ebf-49b7-afaf-12e20e3994a3\") " pod="openshift-image-registry/image-registry-dd954f877-x9h9g" Oct 30 23:41:28.355349 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.354443 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2fa7d24b-67b0-400f-94d4-903feb268782-host-run-multus-certs\") pod \"multus-cb9vj\" (UID: \"2fa7d24b-67b0-400f-94d4-903feb268782\") " pod="openshift-multus/multus-cb9vj" Oct 30 23:41:28.355349 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.354488 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ec330d4f-2eea-476a-a7b8-60cd295fddd2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sqv69\" (UID: \"ec330d4f-2eea-476a-a7b8-60cd295fddd2\") " pod="openshift-insights/insights-runtime-extractor-sqv69" Oct 30 23:41:28.355349 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.354509 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b22fe41c-cf52-4e97-81f1-2f64bcc43640-cni-binary-copy\") pod \"multus-additional-cni-plugins-7mp5r\" (UID: \"b22fe41c-cf52-4e97-81f1-2f64bcc43640\") " pod="openshift-multus/multus-additional-cni-plugins-7mp5r" Oct 30 23:41:28.355349 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.354531 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a8e4c4db-197e-420c-9c60-4273cdf95377-konnectivity-ca\") pod \"konnectivity-agent-9s2fx\" (UID: \"a8e4c4db-197e-420c-9c60-4273cdf95377\") " pod="kube-system/konnectivity-agent-9s2fx" Oct 30 23:41:28.355349 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.354569 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25rtq\" (UniqueName: \"kubernetes.io/projected/0796261f-298e-4cbe-9b3e-b3dcf4528e07-kube-api-access-25rtq\") pod \"aws-ebs-csi-driver-node-j7n2l\" (UID: \"0796261f-298e-4cbe-9b3e-b3dcf4528e07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j7n2l" Oct 30 23:41:28.355349 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.354601 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0d4810de-8bd2-4270-b907-341d9f840bb8-var-lib-openvswitch\") pod \"ovnkube-node-26v6x\" (UID: \"0d4810de-8bd2-4270-b907-341d9f840bb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-26v6x" Oct 30 23:41:28.355349 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.354627 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ec330d4f-2eea-476a-a7b8-60cd295fddd2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sqv69\" (UID: \"ec330d4f-2eea-476a-a7b8-60cd295fddd2\") " pod="openshift-insights/insights-runtime-extractor-sqv69" Oct 30 23:41:28.355349 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.354672 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5x8x6\" (UniqueName: \"kubernetes.io/projected/03211c6a-9761-419c-839b-21ca413d5bb2-kube-api-access-5x8x6\") pod \"cluster-monitoring-operator-884576c57-httg9\" (UID: \"03211c6a-9761-419c-839b-21ca413d5bb2\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-httg9" Oct 30 23:41:28.355349 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.354682 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b22fe41c-cf52-4e97-81f1-2f64bcc43640-cni-binary-copy\") pod \"multus-additional-cni-plugins-7mp5r\" (UID: \"b22fe41c-cf52-4e97-81f1-2f64bcc43640\") " pod="openshift-multus/multus-additional-cni-plugins-7mp5r" Oct 30 23:41:28.355349 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.354690 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a8e4c4db-197e-420c-9c60-4273cdf95377-konnectivity-ca\") pod \"konnectivity-agent-9s2fx\" (UID: \"a8e4c4db-197e-420c-9c60-4273cdf95377\") " pod="kube-system/konnectivity-agent-9s2fx" Oct 30 23:41:28.355349 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.354765 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/58f7cf25-f5d0-4481-a927-4a745b692f7a-trusted-ca\") pod \"image-registry-544b44f8d-5p6kz\" (UID: \"58f7cf25-f5d0-4481-a927-4a745b692f7a\") " pod="openshift-image-registry/image-registry-544b44f8d-5p6kz" Oct 30 23:41:28.355349 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.354786 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/840aaae0-a215-4ccd-b420-d6f5723b65ae-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-9qs84\" (UID: \"840aaae0-a215-4ccd-b420-d6f5723b65ae\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-9qs84" Oct 30 23:41:28.355349 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.354877 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/840aaae0-a215-4ccd-b420-d6f5723b65ae-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-9qs84\" (UID: \"840aaae0-a215-4ccd-b420-d6f5723b65ae\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-9qs84" Oct 30 23:41:28.355349 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.354939 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/17461b46-7ebf-49b7-afaf-12e20e3994a3-registry-tls\") pod \"image-registry-dd954f877-x9h9g\" (UID: \"17461b46-7ebf-49b7-afaf-12e20e3994a3\") " pod="openshift-image-registry/image-registry-dd954f877-x9h9g" Oct 30 23:41:28.355349 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.354965 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0d4810de-8bd2-4270-b907-341d9f840bb8-ovnkube-script-lib\") pod \"ovnkube-node-26v6x\" (UID: \"0d4810de-8bd2-4270-b907-341d9f840bb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-26v6x" Oct 30 23:41:28.355349 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.354983 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qg8hn\" (UniqueName: \"kubernetes.io/projected/6382dfb5-a363-47a2-bc79-0b96f7bec0c3-kube-api-access-qg8hn\") pod \"console-operator-7cd56b8dbc-pqt2c\" (UID: \"6382dfb5-a363-47a2-bc79-0b96f7bec0c3\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-pqt2c" Oct 30 23:41:28.355349 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.355144 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/58f7cf25-f5d0-4481-a927-4a745b692f7a-trusted-ca\") pod \"image-registry-544b44f8d-5p6kz\" (UID: \"58f7cf25-f5d0-4481-a927-4a745b692f7a\") " pod="openshift-image-registry/image-registry-544b44f8d-5p6kz" Oct 30 23:41:28.355349 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.355307 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0d4810de-8bd2-4270-b907-341d9f840bb8-ovnkube-script-lib\") pod \"ovnkube-node-26v6x\" (UID: \"0d4810de-8bd2-4270-b907-341d9f840bb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-26v6x" Oct 30 23:41:28.409889 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.409863 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdj76\" (UniqueName: \"kubernetes.io/projected/a78879b4-333e-4f1a-88bc-e96aee0e6d6b-kube-api-access-sdj76\") pod \"periodic-gathering-r9crl-vqcgk\" (UID: \"a78879b4-333e-4f1a-88bc-e96aee0e6d6b\") " pod="openshift-insights/periodic-gathering-r9crl-vqcgk" Oct 30 23:41:28.410570 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.410546 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/58f7cf25-f5d0-4481-a927-4a745b692f7a-bound-sa-token\") pod \"image-registry-544b44f8d-5p6kz\" (UID: \"58f7cf25-f5d0-4481-a927-4a745b692f7a\") " pod="openshift-image-registry/image-registry-544b44f8d-5p6kz" Oct 30 23:41:28.410702 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.410552 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xfx9\" (UniqueName: \"kubernetes.io/projected/3af6ccf2-97f1-4598-b8ef-12eac377133b-kube-api-access-9xfx9\") pod \"kas-connection-checker-685895c7c9-f7zjp\" (UID: \"3af6ccf2-97f1-4598-b8ef-12eac377133b\") " pod="kube-system/kas-connection-checker-685895c7c9-f7zjp" Oct 30 23:41:28.410989 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.410969 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mlhgb\" (UniqueName: \"kubernetes.io/projected/99f800b6-a38c-46ca-8234-0dbd3610c8fd-kube-api-access-mlhgb\") pod \"node-ca-7sbzf\" (UID: \"99f800b6-a38c-46ca-8234-0dbd3610c8fd\") " pod="openshift-image-registry/node-ca-7sbzf" Oct 30 23:41:28.411208 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.411191 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-btgs8\" (UniqueName: \"kubernetes.io/projected/b22fe41c-cf52-4e97-81f1-2f64bcc43640-kube-api-access-btgs8\") pod \"multus-additional-cni-plugins-7mp5r\" (UID: \"b22fe41c-cf52-4e97-81f1-2f64bcc43640\") " pod="openshift-multus/multus-additional-cni-plugins-7mp5r" Oct 30 23:41:28.422893 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.422874 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9zsm\" (UniqueName: \"kubernetes.io/projected/aa90814d-5b4d-4039-917d-76610b1f88e4-kube-api-access-q9zsm\") pod \"migrator-66df5d496b-8khpq\" (UID: \"aa90814d-5b4d-4039-917d-76610b1f88e4\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-8khpq" Oct 30 23:41:28.423138 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.423123 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nsggg\" (UniqueName: \"kubernetes.io/projected/145dd323-9370-4b9d-a96e-8b93d0a47a7c-kube-api-access-nsggg\") pod \"network-metrics-daemon-pt58s\" (UID: \"145dd323-9370-4b9d-a96e-8b93d0a47a7c\") " pod="openshift-multus/network-metrics-daemon-pt58s" Oct 30 23:41:28.423263 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.423245 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7p9rj\" (UniqueName: \"kubernetes.io/projected/f8b33ecc-59a4-4680-9521-299648ea5055-kube-api-access-7p9rj\") pod \"node-resolver-tfnr7\" (UID: \"f8b33ecc-59a4-4680-9521-299648ea5055\") " pod="openshift-dns/node-resolver-tfnr7" Oct 30 23:41:28.424497 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.424482 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtfn9\" (UniqueName: \"kubernetes.io/projected/4cf4abe6-da2b-41b6-9caa-9659d4f750fb-kube-api-access-rtfn9\") pod \"router-default-66d97c9c97-qqxj5\" (UID: \"4cf4abe6-da2b-41b6-9caa-9659d4f750fb\") " pod="openshift-ingress/router-default-66d97c9c97-qqxj5" Oct 30 23:41:28.425836 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.425820 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vj8vx\" (UniqueName: \"kubernetes.io/projected/29b8bd1a-beeb-495c-9f61-e3326866a1d7-kube-api-access-vj8vx\") pod \"dns-default-9l5zq\" (UID: \"29b8bd1a-beeb-495c-9f61-e3326866a1d7\") " pod="openshift-dns/dns-default-9l5zq" Oct 30 23:41:28.426149 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.426133 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ztdxk\" (UniqueName: \"kubernetes.io/projected/f464ef41-45c9-4ff4-b410-e2d9d6cdbdf5-kube-api-access-ztdxk\") pod \"service-ca-operator-544d749c8c-fbgth\" (UID: \"f464ef41-45c9-4ff4-b410-e2d9d6cdbdf5\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-fbgth" Oct 30 23:41:28.428262 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.428240 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2cxtv\" (UniqueName: \"kubernetes.io/projected/ece07686-4d8e-4068-9fb2-5889046cb73b-kube-api-access-2cxtv\") pod \"iptables-alerter-57xrk\" (UID: \"ece07686-4d8e-4068-9fb2-5889046cb73b\") " pod="openshift-network-operator/iptables-alerter-57xrk" Oct 30 23:41:28.428639 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.428617 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-67888\" (UniqueName: \"kubernetes.io/projected/a4f35692-894d-4d41-8b9f-3f5ba3c529b1-kube-api-access-67888\") pod \"kas-connection-checker-685895c7c9-l4d2m\" (UID: \"a4f35692-894d-4d41-8b9f-3f5ba3c529b1\") " pod="kube-system/kas-connection-checker-685895c7c9-l4d2m" Oct 30 23:41:28.428718 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.428661 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqmnp\" (UniqueName: \"kubernetes.io/projected/7728abfe-1747-422e-b8e2-c91c9de83d09-kube-api-access-jqmnp\") pod \"network-check-target-kgdzs\" (UID: \"7728abfe-1747-422e-b8e2-c91c9de83d09\") " pod="openshift-network-diagnostics/network-check-target-kgdzs" Oct 30 23:41:28.428718 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.428671 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-45fmr\" (UniqueName: \"kubernetes.io/projected/be5540b5-7b10-45b2-b40c-723efd9f1b01-kube-api-access-45fmr\") pod \"cluster-samples-operator-6ddb6c8b94-vl8t5\" (UID: \"be5540b5-7b10-45b2-b40c-723efd9f1b01\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vl8t5" Oct 30 23:41:28.428718 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.428687 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xkthh\" (UniqueName: \"kubernetes.io/projected/17461b46-7ebf-49b7-afaf-12e20e3994a3-kube-api-access-xkthh\") pod \"image-registry-dd954f877-x9h9g\" (UID: \"17461b46-7ebf-49b7-afaf-12e20e3994a3\") " pod="openshift-image-registry/image-registry-dd954f877-x9h9g" Oct 30 23:41:28.430080 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.430061 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/17461b46-7ebf-49b7-afaf-12e20e3994a3-bound-sa-token\") pod \"image-registry-dd954f877-x9h9g\" (UID: \"17461b46-7ebf-49b7-afaf-12e20e3994a3\") " pod="openshift-image-registry/image-registry-dd954f877-x9h9g" Oct 30 23:41:28.431155 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.431132 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vsfpv\" (UniqueName: \"kubernetes.io/projected/840aaae0-a215-4ccd-b420-d6f5723b65ae-kube-api-access-vsfpv\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-9qs84\" (UID: \"840aaae0-a215-4ccd-b420-d6f5723b65ae\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-9qs84" Oct 30 23:41:28.431249 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.431182 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gtt94\" (UniqueName: \"kubernetes.io/projected/953b9446-c289-48ca-b854-a14c70e5971b-kube-api-access-gtt94\") pod \"tuned-rsv7z\" (UID: \"953b9446-c289-48ca-b854-a14c70e5971b\") " pod="openshift-cluster-node-tuning-operator/tuned-rsv7z" Oct 30 23:41:28.431442 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.431422 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xn7l\" (UniqueName: \"kubernetes.io/projected/9700c98a-b858-45a5-9025-8d75f3c14044-kube-api-access-5xn7l\") pod \"ingress-canary-46zqw\" (UID: \"9700c98a-b858-45a5-9025-8d75f3c14044\") " pod="openshift-ingress-canary/ingress-canary-46zqw" Oct 30 23:41:28.431496 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.431464 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n65gn\" (UniqueName: \"kubernetes.io/projected/10774606-8e0c-460b-bd5f-4cb6d121b30b-kube-api-access-n65gn\") pod \"insights-operator-7c4986d449-9mqqt\" (UID: \"10774606-8e0c-460b-bd5f-4cb6d121b30b\") " pod="openshift-insights/insights-operator-7c4986d449-9mqqt" Oct 30 23:41:28.431549 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.431534 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vv9dt\" (UniqueName: \"kubernetes.io/projected/27ccca21-5e99-485f-8b5c-ca8ec5177668-kube-api-access-vv9dt\") pod \"network-check-source-6f78d59f99-xrk8k\" (UID: \"27ccca21-5e99-485f-8b5c-ca8ec5177668\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-xrk8k" Oct 30 23:41:28.433116 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.433094 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9l2q\" (UniqueName: \"kubernetes.io/projected/f8eb4b4c-8225-436b-b1e8-7775414a1ca3-kube-api-access-k9l2q\") pod \"volume-data-source-validator-58c9f5f46d-tcxmv\" (UID: \"f8eb4b4c-8225-436b-b1e8-7775414a1ca3\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-tcxmv" Oct 30 23:41:28.433291 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.433277 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfwlk\" (UniqueName: \"kubernetes.io/projected/64ae3ff5-44eb-456b-948a-c1a8320e6b07-kube-api-access-cfwlk\") pod \"kas-connection-checker-685895c7c9-lxdsf\" (UID: \"64ae3ff5-44eb-456b-948a-c1a8320e6b07\") " pod="kube-system/kas-connection-checker-685895c7c9-lxdsf" Oct 30 23:41:28.433365 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.433346 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4n58\" (UniqueName: \"kubernetes.io/projected/ec330d4f-2eea-476a-a7b8-60cd295fddd2-kube-api-access-z4n58\") pod \"insights-runtime-extractor-sqv69\" (UID: \"ec330d4f-2eea-476a-a7b8-60cd295fddd2\") " pod="openshift-insights/insights-runtime-extractor-sqv69" Oct 30 23:41:28.433432 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.433419 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-48m7s\" (UniqueName: \"kubernetes.io/projected/0d4810de-8bd2-4270-b907-341d9f840bb8-kube-api-access-48m7s\") pod \"ovnkube-node-26v6x\" (UID: \"0d4810de-8bd2-4270-b907-341d9f840bb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-26v6x" Oct 30 23:41:28.433727 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.433713 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qg8hn\" (UniqueName: \"kubernetes.io/projected/6382dfb5-a363-47a2-bc79-0b96f7bec0c3-kube-api-access-qg8hn\") pod \"console-operator-7cd56b8dbc-pqt2c\" (UID: \"6382dfb5-a363-47a2-bc79-0b96f7bec0c3\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-pqt2c" Oct 30 23:41:28.435970 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.435955 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ssd98\" (UniqueName: \"kubernetes.io/projected/2fa7d24b-67b0-400f-94d4-903feb268782-kube-api-access-ssd98\") pod \"multus-cb9vj\" (UID: \"2fa7d24b-67b0-400f-94d4-903feb268782\") " pod="openshift-multus/multus-cb9vj" Oct 30 23:41:28.436014 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.435935 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5x8x6\" (UniqueName: \"kubernetes.io/projected/03211c6a-9761-419c-839b-21ca413d5bb2-kube-api-access-5x8x6\") pod \"cluster-monitoring-operator-884576c57-httg9\" (UID: \"03211c6a-9761-419c-839b-21ca413d5bb2\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-httg9" Oct 30 23:41:28.436080 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.436058 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-25rtq\" (UniqueName: \"kubernetes.io/projected/0796261f-298e-4cbe-9b3e-b3dcf4528e07-kube-api-access-25rtq\") pod \"aws-ebs-csi-driver-node-j7n2l\" (UID: \"0796261f-298e-4cbe-9b3e-b3dcf4528e07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j7n2l" Oct 30 23:41:28.436123 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.436095 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sx9pp\" (UniqueName: \"kubernetes.io/projected/58f7cf25-f5d0-4481-a927-4a745b692f7a-kube-api-access-sx9pp\") pod \"image-registry-544b44f8d-5p6kz\" (UID: \"58f7cf25-f5d0-4481-a927-4a745b692f7a\") " pod="openshift-image-registry/image-registry-544b44f8d-5p6kz" Oct 30 23:41:28.455822 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.455797 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4cf4abe6-da2b-41b6-9caa-9659d4f750fb-metrics-certs\") pod \"router-default-66d97c9c97-qqxj5\" (UID: \"4cf4abe6-da2b-41b6-9caa-9659d4f750fb\") " pod="openshift-ingress/router-default-66d97c9c97-qqxj5" Oct 30 23:41:28.455883 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.455829 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/97da2086-b00d-4f03-b502-56de5ee3d90d-dbus\") pod \"global-pull-secret-syncer-7wrh7\" (UID: \"97da2086-b00d-4f03-b502-56de5ee3d90d\") " pod="kube-system/global-pull-secret-syncer-7wrh7" Oct 30 23:41:28.455883 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.455846 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/58f7cf25-f5d0-4481-a927-4a745b692f7a-registry-tls\") pod \"image-registry-544b44f8d-5p6kz\" (UID: \"58f7cf25-f5d0-4481-a927-4a745b692f7a\") " pod="openshift-image-registry/image-registry-544b44f8d-5p6kz" Oct 30 23:41:28.455883 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.455867 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0d4810de-8bd2-4270-b907-341d9f840bb8-host-run-netns\") pod \"ovnkube-node-26v6x\" (UID: \"0d4810de-8bd2-4270-b907-341d9f840bb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-26v6x" Oct 30 23:41:28.455994 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.455889 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/953b9446-c289-48ca-b854-a14c70e5971b-etc-systemd\") pod \"tuned-rsv7z\" (UID: \"953b9446-c289-48ca-b854-a14c70e5971b\") " pod="openshift-cluster-node-tuning-operator/tuned-rsv7z" Oct 30 23:41:28.455994 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.455930 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2fa7d24b-67b0-400f-94d4-903feb268782-host-var-lib-cni-multus\") pod \"multus-cb9vj\" (UID: \"2fa7d24b-67b0-400f-94d4-903feb268782\") " pod="openshift-multus/multus-cb9vj" Oct 30 23:41:28.455994 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.455943 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/953b9446-c289-48ca-b854-a14c70e5971b-etc-systemd\") pod \"tuned-rsv7z\" (UID: \"953b9446-c289-48ca-b854-a14c70e5971b\") " pod="openshift-cluster-node-tuning-operator/tuned-rsv7z" Oct 30 23:41:28.455994 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.455932 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/97da2086-b00d-4f03-b502-56de5ee3d90d-dbus\") pod \"global-pull-secret-syncer-7wrh7\" (UID: \"97da2086-b00d-4f03-b502-56de5ee3d90d\") " pod="kube-system/global-pull-secret-syncer-7wrh7" Oct 30 23:41:28.455994 ip-10-0-141-180 kubenswrapper[4927]: E1030 23:41:28.455953 4927 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 23:41:28.455994 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.455966 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2fa7d24b-67b0-400f-94d4-903feb268782-host-var-lib-cni-multus\") pod \"multus-cb9vj\" (UID: \"2fa7d24b-67b0-400f-94d4-903feb268782\") " pod="openshift-multus/multus-cb9vj" Oct 30 23:41:28.455994 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.455974 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2fa7d24b-67b0-400f-94d4-903feb268782-multus-conf-dir\") pod \"multus-cb9vj\" (UID: \"2fa7d24b-67b0-400f-94d4-903feb268782\") " pod="openshift-multus/multus-cb9vj" Oct 30 23:41:28.455994 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.455990 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0d4810de-8bd2-4270-b907-341d9f840bb8-host-run-netns\") pod \"ovnkube-node-26v6x\" (UID: \"0d4810de-8bd2-4270-b907-341d9f840bb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-26v6x" Oct 30 23:41:28.456254 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.456002 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0d4810de-8bd2-4270-b907-341d9f840bb8-run-ovn\") pod \"ovnkube-node-26v6x\" (UID: \"0d4810de-8bd2-4270-b907-341d9f840bb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-26v6x" Oct 30 23:41:28.456254 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.456018 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2fa7d24b-67b0-400f-94d4-903feb268782-host-run-k8s-cni-cncf-io\") pod \"multus-cb9vj\" (UID: \"2fa7d24b-67b0-400f-94d4-903feb268782\") " pod="openshift-multus/multus-cb9vj" Oct 30 23:41:28.456254 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.456026 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2fa7d24b-67b0-400f-94d4-903feb268782-multus-conf-dir\") pod \"multus-cb9vj\" (UID: \"2fa7d24b-67b0-400f-94d4-903feb268782\") " pod="openshift-multus/multus-cb9vj" Oct 30 23:41:28.456254 ip-10-0-141-180 kubenswrapper[4927]: E1030 23:41:28.455956 4927 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 23:41:28.456254 ip-10-0-141-180 kubenswrapper[4927]: E1030 23:41:28.456050 4927 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4cf4abe6-da2b-41b6-9caa-9659d4f750fb-metrics-certs podName:4cf4abe6-da2b-41b6-9caa-9659d4f750fb nodeName:}" failed. No retries permitted until 2026-10-30 23:41:28.956028527 +0000 UTC m=+1.992359357 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4cf4abe6-da2b-41b6-9caa-9659d4f750fb-metrics-certs") pod "router-default-66d97c9c97-qqxj5" (UID: "4cf4abe6-da2b-41b6-9caa-9659d4f750fb") : secret "router-metrics-certs-default" not found Oct 30 23:41:28.456254 ip-10-0-141-180 kubenswrapper[4927]: E1030 23:41:28.456054 4927 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-544b44f8d-5p6kz: secret "image-registry-tls" not found Oct 30 23:41:28.456254 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.456060 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0d4810de-8bd2-4270-b907-341d9f840bb8-run-ovn\") pod \"ovnkube-node-26v6x\" (UID: \"0d4810de-8bd2-4270-b907-341d9f840bb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-26v6x" Oct 30 23:41:28.456254 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.456076 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2fa7d24b-67b0-400f-94d4-903feb268782-host-run-netns\") pod \"multus-cb9vj\" (UID: \"2fa7d24b-67b0-400f-94d4-903feb268782\") " pod="openshift-multus/multus-cb9vj" Oct 30 23:41:28.456254 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.456104 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2fa7d24b-67b0-400f-94d4-903feb268782-host-run-netns\") pod \"multus-cb9vj\" (UID: \"2fa7d24b-67b0-400f-94d4-903feb268782\") " pod="openshift-multus/multus-cb9vj" Oct 30 23:41:28.456254 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.456105 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0796261f-298e-4cbe-9b3e-b3dcf4528e07-kubelet-dir\") pod \"aws-ebs-csi-driver-node-j7n2l\" (UID: \"0796261f-298e-4cbe-9b3e-b3dcf4528e07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j7n2l" Oct 30 23:41:28.456254 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.456052 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2fa7d24b-67b0-400f-94d4-903feb268782-host-run-k8s-cni-cncf-io\") pod \"multus-cb9vj\" (UID: \"2fa7d24b-67b0-400f-94d4-903feb268782\") " pod="openshift-multus/multus-cb9vj" Oct 30 23:41:28.456254 ip-10-0-141-180 kubenswrapper[4927]: E1030 23:41:28.456132 4927 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/58f7cf25-f5d0-4481-a927-4a745b692f7a-registry-tls podName:58f7cf25-f5d0-4481-a927-4a745b692f7a nodeName:}" failed. No retries permitted until 2026-10-30 23:41:28.956116478 +0000 UTC m=+1.992447305 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/58f7cf25-f5d0-4481-a927-4a745b692f7a-registry-tls") pod "image-registry-544b44f8d-5p6kz" (UID: "58f7cf25-f5d0-4481-a927-4a745b692f7a") : secret "image-registry-tls" not found Oct 30 23:41:28.456254 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.456152 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0796261f-298e-4cbe-9b3e-b3dcf4528e07-kubelet-dir\") pod \"aws-ebs-csi-driver-node-j7n2l\" (UID: \"0796261f-298e-4cbe-9b3e-b3dcf4528e07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j7n2l" Oct 30 23:41:28.456254 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.456158 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0d4810de-8bd2-4270-b907-341d9f840bb8-host-cni-netd\") pod \"ovnkube-node-26v6x\" (UID: \"0d4810de-8bd2-4270-b907-341d9f840bb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-26v6x" Oct 30 23:41:28.456254 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.456179 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0d4810de-8bd2-4270-b907-341d9f840bb8-host-cni-netd\") pod \"ovnkube-node-26v6x\" (UID: \"0d4810de-8bd2-4270-b907-341d9f840bb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-26v6x" Oct 30 23:41:28.456254 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.456185 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/953b9446-c289-48ca-b854-a14c70e5971b-etc-sysconfig\") pod \"tuned-rsv7z\" (UID: \"953b9446-c289-48ca-b854-a14c70e5971b\") " pod="openshift-cluster-node-tuning-operator/tuned-rsv7z" Oct 30 23:41:28.456254 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.456222 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0796261f-298e-4cbe-9b3e-b3dcf4528e07-socket-dir\") pod \"aws-ebs-csi-driver-node-j7n2l\" (UID: \"0796261f-298e-4cbe-9b3e-b3dcf4528e07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j7n2l" Oct 30 23:41:28.456254 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.456235 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/953b9446-c289-48ca-b854-a14c70e5971b-etc-sysconfig\") pod \"tuned-rsv7z\" (UID: \"953b9446-c289-48ca-b854-a14c70e5971b\") " pod="openshift-cluster-node-tuning-operator/tuned-rsv7z" Oct 30 23:41:28.456254 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.456242 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0d4810de-8bd2-4270-b907-341d9f840bb8-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-26v6x\" (UID: \"0d4810de-8bd2-4270-b907-341d9f840bb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-26v6x" Oct 30 23:41:28.456982 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.456272 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0d4810de-8bd2-4270-b907-341d9f840bb8-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-26v6x\" (UID: \"0d4810de-8bd2-4270-b907-341d9f840bb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-26v6x" Oct 30 23:41:28.456982 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.456276 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0796261f-298e-4cbe-9b3e-b3dcf4528e07-etc-selinux\") pod \"aws-ebs-csi-driver-node-j7n2l\" (UID: \"0796261f-298e-4cbe-9b3e-b3dcf4528e07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j7n2l" Oct 30 23:41:28.456982 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.456308 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0d4810de-8bd2-4270-b907-341d9f840bb8-systemd-units\") pod \"ovnkube-node-26v6x\" (UID: \"0d4810de-8bd2-4270-b907-341d9f840bb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-26v6x" Oct 30 23:41:28.456982 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.456345 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0d4810de-8bd2-4270-b907-341d9f840bb8-systemd-units\") pod \"ovnkube-node-26v6x\" (UID: \"0d4810de-8bd2-4270-b907-341d9f840bb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-26v6x" Oct 30 23:41:28.456982 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.456309 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0796261f-298e-4cbe-9b3e-b3dcf4528e07-socket-dir\") pod \"aws-ebs-csi-driver-node-j7n2l\" (UID: \"0796261f-298e-4cbe-9b3e-b3dcf4528e07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j7n2l" Oct 30 23:41:28.456982 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.456355 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/03211c6a-9761-419c-839b-21ca413d5bb2-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-httg9\" (UID: \"03211c6a-9761-419c-839b-21ca413d5bb2\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-httg9" Oct 30 23:41:28.456982 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.456310 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0796261f-298e-4cbe-9b3e-b3dcf4528e07-etc-selinux\") pod \"aws-ebs-csi-driver-node-j7n2l\" (UID: \"0796261f-298e-4cbe-9b3e-b3dcf4528e07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j7n2l" Oct 30 23:41:28.456982 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.456384 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2fa7d24b-67b0-400f-94d4-903feb268782-system-cni-dir\") pod \"multus-cb9vj\" (UID: \"2fa7d24b-67b0-400f-94d4-903feb268782\") " pod="openshift-multus/multus-cb9vj" Oct 30 23:41:28.456982 ip-10-0-141-180 kubenswrapper[4927]: E1030 23:41:28.456406 4927 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 23:41:28.456982 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.456438 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2fa7d24b-67b0-400f-94d4-903feb268782-system-cni-dir\") pod \"multus-cb9vj\" (UID: \"2fa7d24b-67b0-400f-94d4-903feb268782\") " pod="openshift-multus/multus-cb9vj" Oct 30 23:41:28.456982 ip-10-0-141-180 kubenswrapper[4927]: E1030 23:41:28.456445 4927 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/03211c6a-9761-419c-839b-21ca413d5bb2-cluster-monitoring-operator-tls podName:03211c6a-9761-419c-839b-21ca413d5bb2 nodeName:}" failed. No retries permitted until 2026-10-30 23:41:28.956436505 +0000 UTC m=+1.992767308 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/03211c6a-9761-419c-839b-21ca413d5bb2-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-httg9" (UID: "03211c6a-9761-419c-839b-21ca413d5bb2") : secret "cluster-monitoring-operator-tls" not found Oct 30 23:41:28.456982 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.456465 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4cf4abe6-da2b-41b6-9caa-9659d4f750fb-service-ca-bundle\") pod \"router-default-66d97c9c97-qqxj5\" (UID: \"4cf4abe6-da2b-41b6-9caa-9659d4f750fb\") " pod="openshift-ingress/router-default-66d97c9c97-qqxj5" Oct 30 23:41:28.456982 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.456480 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0d4810de-8bd2-4270-b907-341d9f840bb8-log-socket\") pod \"ovnkube-node-26v6x\" (UID: \"0d4810de-8bd2-4270-b907-341d9f840bb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-26v6x" Oct 30 23:41:28.456982 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.456495 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2fa7d24b-67b0-400f-94d4-903feb268782-host-run-multus-certs\") pod \"multus-cb9vj\" (UID: \"2fa7d24b-67b0-400f-94d4-903feb268782\") " pod="openshift-multus/multus-cb9vj" Oct 30 23:41:28.456982 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.456510 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ec330d4f-2eea-476a-a7b8-60cd295fddd2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sqv69\" (UID: \"ec330d4f-2eea-476a-a7b8-60cd295fddd2\") " pod="openshift-insights/insights-runtime-extractor-sqv69" Oct 30 23:41:28.456982 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.456528 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0d4810de-8bd2-4270-b907-341d9f840bb8-var-lib-openvswitch\") pod \"ovnkube-node-26v6x\" (UID: \"0d4810de-8bd2-4270-b907-341d9f840bb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-26v6x" Oct 30 23:41:28.456982 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.456545 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0d4810de-8bd2-4270-b907-341d9f840bb8-log-socket\") pod \"ovnkube-node-26v6x\" (UID: \"0d4810de-8bd2-4270-b907-341d9f840bb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-26v6x" Oct 30 23:41:28.456982 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.456549 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ec330d4f-2eea-476a-a7b8-60cd295fddd2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sqv69\" (UID: \"ec330d4f-2eea-476a-a7b8-60cd295fddd2\") " pod="openshift-insights/insights-runtime-extractor-sqv69" Oct 30 23:41:28.456982 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.456574 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2fa7d24b-67b0-400f-94d4-903feb268782-host-run-multus-certs\") pod \"multus-cb9vj\" (UID: \"2fa7d24b-67b0-400f-94d4-903feb268782\") " pod="openshift-multus/multus-cb9vj" Oct 30 23:41:28.456982 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.456582 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0d4810de-8bd2-4270-b907-341d9f840bb8-var-lib-openvswitch\") pod \"ovnkube-node-26v6x\" (UID: \"0d4810de-8bd2-4270-b907-341d9f840bb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-26v6x" Oct 30 23:41:28.456982 ip-10-0-141-180 kubenswrapper[4927]: E1030 23:41:28.456581 4927 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 23:41:28.456982 ip-10-0-141-180 kubenswrapper[4927]: E1030 23:41:28.456614 4927 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4cf4abe6-da2b-41b6-9caa-9659d4f750fb-service-ca-bundle podName:4cf4abe6-da2b-41b6-9caa-9659d4f750fb nodeName:}" failed. No retries permitted until 2026-10-30 23:41:28.956599123 +0000 UTC m=+1.992929951 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/4cf4abe6-da2b-41b6-9caa-9659d4f750fb-service-ca-bundle") pod "router-default-66d97c9c97-qqxj5" (UID: "4cf4abe6-da2b-41b6-9caa-9659d4f750fb") : configmap references non-existent config key: service-ca.crt Oct 30 23:41:28.456982 ip-10-0-141-180 kubenswrapper[4927]: E1030 23:41:28.456634 4927 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ec330d4f-2eea-476a-a7b8-60cd295fddd2-kube-rbac-proxy-cm podName:ec330d4f-2eea-476a-a7b8-60cd295fddd2 nodeName:}" failed. No retries permitted until 2026-10-30 23:41:28.956622052 +0000 UTC m=+1.992952878 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ec330d4f-2eea-476a-a7b8-60cd295fddd2-kube-rbac-proxy-cm") pod "insights-runtime-extractor-sqv69" (UID: "ec330d4f-2eea-476a-a7b8-60cd295fddd2") : configmap "kube-rbac-proxy" not found Oct 30 23:41:28.456982 ip-10-0-141-180 kubenswrapper[4927]: E1030 23:41:28.456650 4927 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 23:41:28.456982 ip-10-0-141-180 kubenswrapper[4927]: E1030 23:41:28.456687 4927 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 23:41:28.456982 ip-10-0-141-180 kubenswrapper[4927]: E1030 23:41:28.456694 4927 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-dd954f877-x9h9g: secret "image-registry-tls" not found Oct 30 23:41:28.456982 ip-10-0-141-180 kubenswrapper[4927]: E1030 23:41:28.456695 4927 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec330d4f-2eea-476a-a7b8-60cd295fddd2-insights-runtime-extractor-tls podName:ec330d4f-2eea-476a-a7b8-60cd295fddd2 nodeName:}" failed. No retries permitted until 2026-10-30 23:41:28.95667931 +0000 UTC m=+1.993010132 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ec330d4f-2eea-476a-a7b8-60cd295fddd2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-sqv69" (UID: "ec330d4f-2eea-476a-a7b8-60cd295fddd2") : secret "insights-runtime-extractor-tls" not found Oct 30 23:41:28.456982 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.456650 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/17461b46-7ebf-49b7-afaf-12e20e3994a3-registry-tls\") pod \"image-registry-dd954f877-x9h9g\" (UID: \"17461b46-7ebf-49b7-afaf-12e20e3994a3\") " pod="openshift-image-registry/image-registry-dd954f877-x9h9g" Oct 30 23:41:28.456982 ip-10-0-141-180 kubenswrapper[4927]: E1030 23:41:28.456714 4927 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/17461b46-7ebf-49b7-afaf-12e20e3994a3-registry-tls podName:17461b46-7ebf-49b7-afaf-12e20e3994a3 nodeName:}" failed. No retries permitted until 2026-10-30 23:41:28.956707807 +0000 UTC m=+1.993038606 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/17461b46-7ebf-49b7-afaf-12e20e3994a3-registry-tls") pod "image-registry-dd954f877-x9h9g" (UID: "17461b46-7ebf-49b7-afaf-12e20e3994a3") : secret "image-registry-tls" not found Oct 30 23:41:28.456982 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.456731 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f8b33ecc-59a4-4680-9521-299648ea5055-hosts-file\") pod \"node-resolver-tfnr7\" (UID: \"f8b33ecc-59a4-4680-9521-299648ea5055\") " pod="openshift-dns/node-resolver-tfnr7" Oct 30 23:41:28.456982 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.456746 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0d4810de-8bd2-4270-b907-341d9f840bb8-node-log\") pod \"ovnkube-node-26v6x\" (UID: \"0d4810de-8bd2-4270-b907-341d9f840bb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-26v6x" Oct 30 23:41:28.456982 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.456761 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0d4810de-8bd2-4270-b907-341d9f840bb8-host-run-ovn-kubernetes\") pod \"ovnkube-node-26v6x\" (UID: \"0d4810de-8bd2-4270-b907-341d9f840bb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-26v6x" Oct 30 23:41:28.456982 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.456777 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0d4810de-8bd2-4270-b907-341d9f840bb8-host-cni-bin\") pod \"ovnkube-node-26v6x\" (UID: \"0d4810de-8bd2-4270-b907-341d9f840bb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-26v6x" Oct 30 23:41:28.456982 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.456790 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2fa7d24b-67b0-400f-94d4-903feb268782-multus-cni-dir\") pod \"multus-cb9vj\" (UID: \"2fa7d24b-67b0-400f-94d4-903feb268782\") " pod="openshift-multus/multus-cb9vj" Oct 30 23:41:28.456982 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.456805 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0d4810de-8bd2-4270-b907-341d9f840bb8-host-slash\") pod \"ovnkube-node-26v6x\" (UID: \"0d4810de-8bd2-4270-b907-341d9f840bb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-26v6x" Oct 30 23:41:28.456982 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.456811 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0d4810de-8bd2-4270-b907-341d9f840bb8-node-log\") pod \"ovnkube-node-26v6x\" (UID: \"0d4810de-8bd2-4270-b907-341d9f840bb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-26v6x" Oct 30 23:41:28.456982 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.456820 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2fa7d24b-67b0-400f-94d4-903feb268782-host-var-lib-cni-bin\") pod \"multus-cb9vj\" (UID: \"2fa7d24b-67b0-400f-94d4-903feb268782\") " pod="openshift-multus/multus-cb9vj" Oct 30 23:41:28.456982 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.456831 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0d4810de-8bd2-4270-b907-341d9f840bb8-host-run-ovn-kubernetes\") pod \"ovnkube-node-26v6x\" (UID: \"0d4810de-8bd2-4270-b907-341d9f840bb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-26v6x" Oct 30 23:41:28.456982 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.456840 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0d4810de-8bd2-4270-b907-341d9f840bb8-host-cni-bin\") pod \"ovnkube-node-26v6x\" (UID: \"0d4810de-8bd2-4270-b907-341d9f840bb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-26v6x" Oct 30 23:41:28.456982 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.456841 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b53ad6ed-90fd-4a0f-bf54-88d416ba19e9-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-rxpst\" (UID: \"b53ad6ed-90fd-4a0f-bf54-88d416ba19e9\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-rxpst" Oct 30 23:41:28.456982 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.456863 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2fa7d24b-67b0-400f-94d4-903feb268782-multus-cni-dir\") pod \"multus-cb9vj\" (UID: \"2fa7d24b-67b0-400f-94d4-903feb268782\") " pod="openshift-multus/multus-cb9vj" Oct 30 23:41:28.456982 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.456864 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f8b33ecc-59a4-4680-9521-299648ea5055-hosts-file\") pod \"node-resolver-tfnr7\" (UID: \"f8b33ecc-59a4-4680-9521-299648ea5055\") " pod="openshift-dns/node-resolver-tfnr7" Oct 30 23:41:28.456982 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.456868 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0d4810de-8bd2-4270-b907-341d9f840bb8-host-slash\") pod \"ovnkube-node-26v6x\" (UID: \"0d4810de-8bd2-4270-b907-341d9f840bb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-26v6x" Oct 30 23:41:28.456982 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.456875 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2fa7d24b-67b0-400f-94d4-903feb268782-host-var-lib-cni-bin\") pod \"multus-cb9vj\" (UID: \"2fa7d24b-67b0-400f-94d4-903feb268782\") " pod="openshift-multus/multus-cb9vj" Oct 30 23:41:28.456982 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.456883 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ec330d4f-2eea-476a-a7b8-60cd295fddd2-crio-socket\") pod \"insights-runtime-extractor-sqv69\" (UID: \"ec330d4f-2eea-476a-a7b8-60cd295fddd2\") " pod="openshift-insights/insights-runtime-extractor-sqv69" Oct 30 23:41:28.456982 ip-10-0-141-180 kubenswrapper[4927]: E1030 23:41:28.456905 4927 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 23:41:28.456982 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.456907 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0d4810de-8bd2-4270-b907-341d9f840bb8-run-systemd\") pod \"ovnkube-node-26v6x\" (UID: \"0d4810de-8bd2-4270-b907-341d9f840bb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-26v6x" Oct 30 23:41:28.456982 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.456917 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ec330d4f-2eea-476a-a7b8-60cd295fddd2-crio-socket\") pod \"insights-runtime-extractor-sqv69\" (UID: \"ec330d4f-2eea-476a-a7b8-60cd295fddd2\") " pod="openshift-insights/insights-runtime-extractor-sqv69" Oct 30 23:41:28.456982 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.456923 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/953b9446-c289-48ca-b854-a14c70e5971b-lib-modules\") pod \"tuned-rsv7z\" (UID: \"953b9446-c289-48ca-b854-a14c70e5971b\") " pod="openshift-cluster-node-tuning-operator/tuned-rsv7z" Oct 30 23:41:28.456982 ip-10-0-141-180 kubenswrapper[4927]: E1030 23:41:28.456953 4927 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b53ad6ed-90fd-4a0f-bf54-88d416ba19e9-networking-console-plugin-cert podName:b53ad6ed-90fd-4a0f-bf54-88d416ba19e9 nodeName:}" failed. No retries permitted until 2026-10-30 23:41:28.956942185 +0000 UTC m=+1.993273000 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/b53ad6ed-90fd-4a0f-bf54-88d416ba19e9-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-rxpst" (UID: "b53ad6ed-90fd-4a0f-bf54-88d416ba19e9") : secret "networking-console-plugin-cert" not found Oct 30 23:41:28.456982 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.456966 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0d4810de-8bd2-4270-b907-341d9f840bb8-run-systemd\") pod \"ovnkube-node-26v6x\" (UID: \"0d4810de-8bd2-4270-b907-341d9f840bb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-26v6x" Oct 30 23:41:28.456982 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.456972 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2fa7d24b-67b0-400f-94d4-903feb268782-etc-kubernetes\") pod \"multus-cb9vj\" (UID: \"2fa7d24b-67b0-400f-94d4-903feb268782\") " pod="openshift-multus/multus-cb9vj" Oct 30 23:41:28.456982 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.457000 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2fa7d24b-67b0-400f-94d4-903feb268782-etc-kubernetes\") pod \"multus-cb9vj\" (UID: \"2fa7d24b-67b0-400f-94d4-903feb268782\") " pod="openshift-multus/multus-cb9vj" Oct 30 23:41:28.456982 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.457006 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/953b9446-c289-48ca-b854-a14c70e5971b-etc-modprobe-d\") pod \"tuned-rsv7z\" (UID: \"953b9446-c289-48ca-b854-a14c70e5971b\") " pod="openshift-cluster-node-tuning-operator/tuned-rsv7z" Oct 30 23:41:28.459100 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.457024 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/953b9446-c289-48ca-b854-a14c70e5971b-etc-sysctl-conf\") pod \"tuned-rsv7z\" (UID: \"953b9446-c289-48ca-b854-a14c70e5971b\") " pod="openshift-cluster-node-tuning-operator/tuned-rsv7z" Oct 30 23:41:28.459100 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.457031 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/953b9446-c289-48ca-b854-a14c70e5971b-lib-modules\") pod \"tuned-rsv7z\" (UID: \"953b9446-c289-48ca-b854-a14c70e5971b\") " pod="openshift-cluster-node-tuning-operator/tuned-rsv7z" Oct 30 23:41:28.459100 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.457047 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/953b9446-c289-48ca-b854-a14c70e5971b-var-lib-kubelet\") pod \"tuned-rsv7z\" (UID: \"953b9446-c289-48ca-b854-a14c70e5971b\") " pod="openshift-cluster-node-tuning-operator/tuned-rsv7z" Oct 30 23:41:28.459100 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.457077 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/be5540b5-7b10-45b2-b40c-723efd9f1b01-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-vl8t5\" (UID: \"be5540b5-7b10-45b2-b40c-723efd9f1b01\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vl8t5" Oct 30 23:41:28.459100 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.457097 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/953b9446-c289-48ca-b854-a14c70e5971b-run\") pod \"tuned-rsv7z\" (UID: \"953b9446-c289-48ca-b854-a14c70e5971b\") " pod="openshift-cluster-node-tuning-operator/tuned-rsv7z" Oct 30 23:41:28.459100 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.457118 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/953b9446-c289-48ca-b854-a14c70e5971b-var-lib-kubelet\") pod \"tuned-rsv7z\" (UID: \"953b9446-c289-48ca-b854-a14c70e5971b\") " pod="openshift-cluster-node-tuning-operator/tuned-rsv7z" Oct 30 23:41:28.459100 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.457122 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/953b9446-c289-48ca-b854-a14c70e5971b-host\") pod \"tuned-rsv7z\" (UID: \"953b9446-c289-48ca-b854-a14c70e5971b\") " pod="openshift-cluster-node-tuning-operator/tuned-rsv7z" Oct 30 23:41:28.459100 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.457158 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/953b9446-c289-48ca-b854-a14c70e5971b-etc-sysctl-d\") pod \"tuned-rsv7z\" (UID: \"953b9446-c289-48ca-b854-a14c70e5971b\") " pod="openshift-cluster-node-tuning-operator/tuned-rsv7z" Oct 30 23:41:28.459100 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.457120 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/953b9446-c289-48ca-b854-a14c70e5971b-etc-modprobe-d\") pod \"tuned-rsv7z\" (UID: \"953b9446-c289-48ca-b854-a14c70e5971b\") " pod="openshift-cluster-node-tuning-operator/tuned-rsv7z" Oct 30 23:41:28.459100 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.457160 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/953b9446-c289-48ca-b854-a14c70e5971b-host\") pod \"tuned-rsv7z\" (UID: \"953b9446-c289-48ca-b854-a14c70e5971b\") " pod="openshift-cluster-node-tuning-operator/tuned-rsv7z" Oct 30 23:41:28.459100 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.457184 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b22fe41c-cf52-4e97-81f1-2f64bcc43640-cnibin\") pod \"multus-additional-cni-plugins-7mp5r\" (UID: \"b22fe41c-cf52-4e97-81f1-2f64bcc43640\") " pod="openshift-multus/multus-additional-cni-plugins-7mp5r" Oct 30 23:41:28.459100 ip-10-0-141-180 kubenswrapper[4927]: E1030 23:41:28.457185 4927 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 23:41:28.459100 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.457124 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/953b9446-c289-48ca-b854-a14c70e5971b-etc-sysctl-conf\") pod \"tuned-rsv7z\" (UID: \"953b9446-c289-48ca-b854-a14c70e5971b\") " pod="openshift-cluster-node-tuning-operator/tuned-rsv7z" Oct 30 23:41:28.459100 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.457214 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0796261f-298e-4cbe-9b3e-b3dcf4528e07-device-dir\") pod \"aws-ebs-csi-driver-node-j7n2l\" (UID: \"0796261f-298e-4cbe-9b3e-b3dcf4528e07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j7n2l" Oct 30 23:41:28.459100 ip-10-0-141-180 kubenswrapper[4927]: E1030 23:41:28.457237 4927 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/be5540b5-7b10-45b2-b40c-723efd9f1b01-samples-operator-tls podName:be5540b5-7b10-45b2-b40c-723efd9f1b01 nodeName:}" failed. No retries permitted until 2026-10-30 23:41:28.957226775 +0000 UTC m=+1.993557608 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/be5540b5-7b10-45b2-b40c-723efd9f1b01-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-vl8t5" (UID: "be5540b5-7b10-45b2-b40c-723efd9f1b01") : secret "samples-operator-tls" not found Oct 30 23:41:28.459100 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.457241 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/953b9446-c289-48ca-b854-a14c70e5971b-run\") pod \"tuned-rsv7z\" (UID: \"953b9446-c289-48ca-b854-a14c70e5971b\") " pod="openshift-cluster-node-tuning-operator/tuned-rsv7z" Oct 30 23:41:28.459100 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.457251 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/953b9446-c289-48ca-b854-a14c70e5971b-etc-sysctl-d\") pod \"tuned-rsv7z\" (UID: \"953b9446-c289-48ca-b854-a14c70e5971b\") " pod="openshift-cluster-node-tuning-operator/tuned-rsv7z" Oct 30 23:41:28.459100 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.457245 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b22fe41c-cf52-4e97-81f1-2f64bcc43640-cnibin\") pod \"multus-additional-cni-plugins-7mp5r\" (UID: \"b22fe41c-cf52-4e97-81f1-2f64bcc43640\") " pod="openshift-multus/multus-additional-cni-plugins-7mp5r" Oct 30 23:41:28.459100 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.457253 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2fa7d24b-67b0-400f-94d4-903feb268782-hostroot\") pod \"multus-cb9vj\" (UID: \"2fa7d24b-67b0-400f-94d4-903feb268782\") " pod="openshift-multus/multus-cb9vj" Oct 30 23:41:28.459100 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.457276 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2fa7d24b-67b0-400f-94d4-903feb268782-hostroot\") pod \"multus-cb9vj\" (UID: \"2fa7d24b-67b0-400f-94d4-903feb268782\") " pod="openshift-multus/multus-cb9vj" Oct 30 23:41:28.459100 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.457286 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/29b8bd1a-beeb-495c-9f61-e3326866a1d7-metrics-tls\") pod \"dns-default-9l5zq\" (UID: \"29b8bd1a-beeb-495c-9f61-e3326866a1d7\") " pod="openshift-dns/dns-default-9l5zq" Oct 30 23:41:28.459100 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.457265 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0796261f-298e-4cbe-9b3e-b3dcf4528e07-device-dir\") pod \"aws-ebs-csi-driver-node-j7n2l\" (UID: \"0796261f-298e-4cbe-9b3e-b3dcf4528e07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j7n2l" Oct 30 23:41:28.459100 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.457314 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b22fe41c-cf52-4e97-81f1-2f64bcc43640-tuning-conf-dir\") pod \"multus-additional-cni-plugins-7mp5r\" (UID: \"b22fe41c-cf52-4e97-81f1-2f64bcc43640\") " pod="openshift-multus/multus-additional-cni-plugins-7mp5r" Oct 30 23:41:28.459100 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.457362 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2fa7d24b-67b0-400f-94d4-903feb268782-host-var-lib-kubelet\") pod \"multus-cb9vj\" (UID: \"2fa7d24b-67b0-400f-94d4-903feb268782\") " pod="openshift-multus/multus-cb9vj" Oct 30 23:41:28.459100 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.457371 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b22fe41c-cf52-4e97-81f1-2f64bcc43640-tuning-conf-dir\") pod \"multus-additional-cni-plugins-7mp5r\" (UID: \"b22fe41c-cf52-4e97-81f1-2f64bcc43640\") " pod="openshift-multus/multus-additional-cni-plugins-7mp5r" Oct 30 23:41:28.459100 ip-10-0-141-180 kubenswrapper[4927]: E1030 23:41:28.457380 4927 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 23:41:28.459100 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.457396 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2fa7d24b-67b0-400f-94d4-903feb268782-host-var-lib-kubelet\") pod \"multus-cb9vj\" (UID: \"2fa7d24b-67b0-400f-94d4-903feb268782\") " pod="openshift-multus/multus-cb9vj" Oct 30 23:41:28.459100 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.457398 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b22fe41c-cf52-4e97-81f1-2f64bcc43640-system-cni-dir\") pod \"multus-additional-cni-plugins-7mp5r\" (UID: \"b22fe41c-cf52-4e97-81f1-2f64bcc43640\") " pod="openshift-multus/multus-additional-cni-plugins-7mp5r" Oct 30 23:41:28.459100 ip-10-0-141-180 kubenswrapper[4927]: E1030 23:41:28.457409 4927 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/29b8bd1a-beeb-495c-9f61-e3326866a1d7-metrics-tls podName:29b8bd1a-beeb-495c-9f61-e3326866a1d7 nodeName:}" failed. No retries permitted until 2026-10-30 23:41:28.957400481 +0000 UTC m=+1.993731282 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/29b8bd1a-beeb-495c-9f61-e3326866a1d7-metrics-tls") pod "dns-default-9l5zq" (UID: "29b8bd1a-beeb-495c-9f61-e3326866a1d7") : secret "dns-default-metrics-tls" not found Oct 30 23:41:28.459100 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.457382 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b22fe41c-cf52-4e97-81f1-2f64bcc43640-system-cni-dir\") pod \"multus-additional-cni-plugins-7mp5r\" (UID: \"b22fe41c-cf52-4e97-81f1-2f64bcc43640\") " pod="openshift-multus/multus-additional-cni-plugins-7mp5r" Oct 30 23:41:28.459100 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.457436 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0796261f-298e-4cbe-9b3e-b3dcf4528e07-registration-dir\") pod \"aws-ebs-csi-driver-node-j7n2l\" (UID: \"0796261f-298e-4cbe-9b3e-b3dcf4528e07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j7n2l" Oct 30 23:41:28.459100 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.457451 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0d4810de-8bd2-4270-b907-341d9f840bb8-etc-openvswitch\") pod \"ovnkube-node-26v6x\" (UID: \"0d4810de-8bd2-4270-b907-341d9f840bb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-26v6x" Oct 30 23:41:28.459100 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.457465 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/99f800b6-a38c-46ca-8234-0dbd3610c8fd-host\") pod \"node-ca-7sbzf\" (UID: \"99f800b6-a38c-46ca-8234-0dbd3610c8fd\") " pod="openshift-image-registry/node-ca-7sbzf" Oct 30 23:41:28.459100 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.457488 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0796261f-298e-4cbe-9b3e-b3dcf4528e07-sys-fs\") pod \"aws-ebs-csi-driver-node-j7n2l\" (UID: \"0796261f-298e-4cbe-9b3e-b3dcf4528e07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j7n2l" Oct 30 23:41:28.459100 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.457498 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0796261f-298e-4cbe-9b3e-b3dcf4528e07-registration-dir\") pod \"aws-ebs-csi-driver-node-j7n2l\" (UID: \"0796261f-298e-4cbe-9b3e-b3dcf4528e07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j7n2l" Oct 30 23:41:28.459100 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.457506 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0d4810de-8bd2-4270-b907-341d9f840bb8-etc-openvswitch\") pod \"ovnkube-node-26v6x\" (UID: \"0d4810de-8bd2-4270-b907-341d9f840bb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-26v6x" Oct 30 23:41:28.459100 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.457523 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/99f800b6-a38c-46ca-8234-0dbd3610c8fd-host\") pod \"node-ca-7sbzf\" (UID: \"99f800b6-a38c-46ca-8234-0dbd3610c8fd\") " pod="openshift-image-registry/node-ca-7sbzf" Oct 30 23:41:28.459100 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.457536 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0796261f-298e-4cbe-9b3e-b3dcf4528e07-sys-fs\") pod \"aws-ebs-csi-driver-node-j7n2l\" (UID: \"0796261f-298e-4cbe-9b3e-b3dcf4528e07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j7n2l" Oct 30 23:41:28.459100 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.457544 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0d4810de-8bd2-4270-b907-341d9f840bb8-host-kubelet\") pod \"ovnkube-node-26v6x\" (UID: \"0d4810de-8bd2-4270-b907-341d9f840bb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-26v6x" Oct 30 23:41:28.459100 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.457566 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/97da2086-b00d-4f03-b502-56de5ee3d90d-kubelet-config\") pod \"global-pull-secret-syncer-7wrh7\" (UID: \"97da2086-b00d-4f03-b502-56de5ee3d90d\") " pod="kube-system/global-pull-secret-syncer-7wrh7" Oct 30 23:41:28.459100 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.457591 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2fa7d24b-67b0-400f-94d4-903feb268782-multus-socket-dir-parent\") pod \"multus-cb9vj\" (UID: \"2fa7d24b-67b0-400f-94d4-903feb268782\") " pod="openshift-multus/multus-cb9vj" Oct 30 23:41:28.459100 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.457597 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0d4810de-8bd2-4270-b907-341d9f840bb8-host-kubelet\") pod \"ovnkube-node-26v6x\" (UID: \"0d4810de-8bd2-4270-b907-341d9f840bb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-26v6x" Oct 30 23:41:28.459100 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.457595 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/97da2086-b00d-4f03-b502-56de5ee3d90d-kubelet-config\") pod \"global-pull-secret-syncer-7wrh7\" (UID: \"97da2086-b00d-4f03-b502-56de5ee3d90d\") " pod="kube-system/global-pull-secret-syncer-7wrh7" Oct 30 23:41:28.459100 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.457620 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2fa7d24b-67b0-400f-94d4-903feb268782-multus-socket-dir-parent\") pod \"multus-cb9vj\" (UID: \"2fa7d24b-67b0-400f-94d4-903feb268782\") " pod="openshift-multus/multus-cb9vj" Oct 30 23:41:28.459100 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.457619 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/953b9446-c289-48ca-b854-a14c70e5971b-etc-kubernetes\") pod \"tuned-rsv7z\" (UID: \"953b9446-c289-48ca-b854-a14c70e5971b\") " pod="openshift-cluster-node-tuning-operator/tuned-rsv7z" Oct 30 23:41:28.459100 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.457642 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/145dd323-9370-4b9d-a96e-8b93d0a47a7c-metrics-certs\") pod \"network-metrics-daemon-pt58s\" (UID: \"145dd323-9370-4b9d-a96e-8b93d0a47a7c\") " pod="openshift-multus/network-metrics-daemon-pt58s" Oct 30 23:41:28.459100 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.457649 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/953b9446-c289-48ca-b854-a14c70e5971b-etc-kubernetes\") pod \"tuned-rsv7z\" (UID: \"953b9446-c289-48ca-b854-a14c70e5971b\") " pod="openshift-cluster-node-tuning-operator/tuned-rsv7z" Oct 30 23:41:28.459100 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.457660 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ece07686-4d8e-4068-9fb2-5889046cb73b-host-slash\") pod \"iptables-alerter-57xrk\" (UID: \"ece07686-4d8e-4068-9fb2-5889046cb73b\") " pod="openshift-network-operator/iptables-alerter-57xrk" Oct 30 23:41:28.459100 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.457682 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0d4810de-8bd2-4270-b907-341d9f840bb8-run-openvswitch\") pod \"ovnkube-node-26v6x\" (UID: \"0d4810de-8bd2-4270-b907-341d9f840bb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-26v6x" Oct 30 23:41:28.459100 ip-10-0-141-180 kubenswrapper[4927]: E1030 23:41:28.457691 4927 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 30 23:41:28.459100 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.457705 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0d4810de-8bd2-4270-b907-341d9f840bb8-run-openvswitch\") pod \"ovnkube-node-26v6x\" (UID: \"0d4810de-8bd2-4270-b907-341d9f840bb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-26v6x" Oct 30 23:41:28.459100 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.457705 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ece07686-4d8e-4068-9fb2-5889046cb73b-host-slash\") pod \"iptables-alerter-57xrk\" (UID: \"ece07686-4d8e-4068-9fb2-5889046cb73b\") " pod="openshift-network-operator/iptables-alerter-57xrk" Oct 30 23:41:28.459100 ip-10-0-141-180 kubenswrapper[4927]: E1030 23:41:28.457722 4927 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/145dd323-9370-4b9d-a96e-8b93d0a47a7c-metrics-certs podName:145dd323-9370-4b9d-a96e-8b93d0a47a7c nodeName:}" failed. No retries permitted until 2026-10-30 23:41:28.957710355 +0000 UTC m=+1.994041159 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/145dd323-9370-4b9d-a96e-8b93d0a47a7c-metrics-certs") pod "network-metrics-daemon-pt58s" (UID: "145dd323-9370-4b9d-a96e-8b93d0a47a7c") : secret "metrics-daemon-secret" not found Oct 30 23:41:28.459100 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.457746 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/953b9446-c289-48ca-b854-a14c70e5971b-sys\") pod \"tuned-rsv7z\" (UID: \"953b9446-c289-48ca-b854-a14c70e5971b\") " pod="openshift-cluster-node-tuning-operator/tuned-rsv7z" Oct 30 23:41:28.459100 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.457765 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2fa7d24b-67b0-400f-94d4-903feb268782-cnibin\") pod \"multus-cb9vj\" (UID: \"2fa7d24b-67b0-400f-94d4-903feb268782\") " pod="openshift-multus/multus-cb9vj" Oct 30 23:41:28.459100 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.457802 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/953b9446-c289-48ca-b854-a14c70e5971b-sys\") pod \"tuned-rsv7z\" (UID: \"953b9446-c289-48ca-b854-a14c70e5971b\") " pod="openshift-cluster-node-tuning-operator/tuned-rsv7z" Oct 30 23:41:28.459100 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.457821 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2fa7d24b-67b0-400f-94d4-903feb268782-cnibin\") pod \"multus-cb9vj\" (UID: \"2fa7d24b-67b0-400f-94d4-903feb268782\") " pod="openshift-multus/multus-cb9vj" Oct 30 23:41:28.486207 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.486179 4927 scope.go:122] "RemoveContainer" containerID="e373ab1ed0edd47d38ff51aab784dee07bf119717f872d40609595f835e4357c" Oct 30 23:41:28.501830 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.501813 4927 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 23:41:28.649846 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.649814 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:41:28.868643 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.868574 4927 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-tfnr7_f8b33ecc-59a4-4680-9521-299648ea5055/dns-node-resolver/0.log" Oct 30 23:41:28.962346 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.962292 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/be5540b5-7b10-45b2-b40c-723efd9f1b01-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-vl8t5\" (UID: \"be5540b5-7b10-45b2-b40c-723efd9f1b01\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vl8t5" Oct 30 23:41:28.962545 ip-10-0-141-180 kubenswrapper[4927]: E1030 23:41:28.962438 4927 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 23:41:28.962545 ip-10-0-141-180 kubenswrapper[4927]: E1030 23:41:28.962504 4927 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/be5540b5-7b10-45b2-b40c-723efd9f1b01-samples-operator-tls podName:be5540b5-7b10-45b2-b40c-723efd9f1b01 nodeName:}" failed. No retries permitted until 2026-10-30 23:41:29.962489654 +0000 UTC m=+2.998820454 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/be5540b5-7b10-45b2-b40c-723efd9f1b01-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-vl8t5" (UID: "be5540b5-7b10-45b2-b40c-723efd9f1b01") : secret "samples-operator-tls" not found Oct 30 23:41:28.962545 ip-10-0-141-180 kubenswrapper[4927]: E1030 23:41:28.962504 4927 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 23:41:28.962545 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.962446 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/29b8bd1a-beeb-495c-9f61-e3326866a1d7-metrics-tls\") pod \"dns-default-9l5zq\" (UID: \"29b8bd1a-beeb-495c-9f61-e3326866a1d7\") " pod="openshift-dns/dns-default-9l5zq" Oct 30 23:41:28.962545 ip-10-0-141-180 kubenswrapper[4927]: E1030 23:41:28.962547 4927 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/29b8bd1a-beeb-495c-9f61-e3326866a1d7-metrics-tls podName:29b8bd1a-beeb-495c-9f61-e3326866a1d7 nodeName:}" failed. No retries permitted until 2026-10-30 23:41:29.962537314 +0000 UTC m=+2.998868114 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/29b8bd1a-beeb-495c-9f61-e3326866a1d7-metrics-tls") pod "dns-default-9l5zq" (UID: "29b8bd1a-beeb-495c-9f61-e3326866a1d7") : secret "dns-default-metrics-tls" not found Oct 30 23:41:28.962831 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.962569 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/145dd323-9370-4b9d-a96e-8b93d0a47a7c-metrics-certs\") pod \"network-metrics-daemon-pt58s\" (UID: \"145dd323-9370-4b9d-a96e-8b93d0a47a7c\") " pod="openshift-multus/network-metrics-daemon-pt58s" Oct 30 23:41:28.962831 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.962594 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4cf4abe6-da2b-41b6-9caa-9659d4f750fb-metrics-certs\") pod \"router-default-66d97c9c97-qqxj5\" (UID: \"4cf4abe6-da2b-41b6-9caa-9659d4f750fb\") " pod="openshift-ingress/router-default-66d97c9c97-qqxj5" Oct 30 23:41:28.962831 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.962611 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/58f7cf25-f5d0-4481-a927-4a745b692f7a-registry-tls\") pod \"image-registry-544b44f8d-5p6kz\" (UID: \"58f7cf25-f5d0-4481-a927-4a745b692f7a\") " pod="openshift-image-registry/image-registry-544b44f8d-5p6kz" Oct 30 23:41:28.962831 ip-10-0-141-180 kubenswrapper[4927]: E1030 23:41:28.962653 4927 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 30 23:41:28.962831 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.962654 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/03211c6a-9761-419c-839b-21ca413d5bb2-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-httg9\" (UID: \"03211c6a-9761-419c-839b-21ca413d5bb2\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-httg9" Oct 30 23:41:28.962831 ip-10-0-141-180 kubenswrapper[4927]: E1030 23:41:28.962671 4927 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 23:41:28.962831 ip-10-0-141-180 kubenswrapper[4927]: E1030 23:41:28.962709 4927 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 23:41:28.962831 ip-10-0-141-180 kubenswrapper[4927]: E1030 23:41:28.962678 4927 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/145dd323-9370-4b9d-a96e-8b93d0a47a7c-metrics-certs podName:145dd323-9370-4b9d-a96e-8b93d0a47a7c nodeName:}" failed. No retries permitted until 2026-10-30 23:41:29.962671176 +0000 UTC m=+2.999001975 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/145dd323-9370-4b9d-a96e-8b93d0a47a7c-metrics-certs") pod "network-metrics-daemon-pt58s" (UID: "145dd323-9370-4b9d-a96e-8b93d0a47a7c") : secret "metrics-daemon-secret" not found Oct 30 23:41:28.962831 ip-10-0-141-180 kubenswrapper[4927]: E1030 23:41:28.962722 4927 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 23:41:28.962831 ip-10-0-141-180 kubenswrapper[4927]: E1030 23:41:28.962735 4927 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-544b44f8d-5p6kz: secret "image-registry-tls" not found Oct 30 23:41:28.962831 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.962740 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4cf4abe6-da2b-41b6-9caa-9659d4f750fb-service-ca-bundle\") pod \"router-default-66d97c9c97-qqxj5\" (UID: \"4cf4abe6-da2b-41b6-9caa-9659d4f750fb\") " pod="openshift-ingress/router-default-66d97c9c97-qqxj5" Oct 30 23:41:28.962831 ip-10-0-141-180 kubenswrapper[4927]: E1030 23:41:28.962771 4927 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/58f7cf25-f5d0-4481-a927-4a745b692f7a-registry-tls podName:58f7cf25-f5d0-4481-a927-4a745b692f7a nodeName:}" failed. No retries permitted until 2026-10-30 23:41:29.962758837 +0000 UTC m=+2.999089656 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/58f7cf25-f5d0-4481-a927-4a745b692f7a-registry-tls") pod "image-registry-544b44f8d-5p6kz" (UID: "58f7cf25-f5d0-4481-a927-4a745b692f7a") : secret "image-registry-tls" not found Oct 30 23:41:28.962831 ip-10-0-141-180 kubenswrapper[4927]: E1030 23:41:28.962800 4927 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4cf4abe6-da2b-41b6-9caa-9659d4f750fb-metrics-certs podName:4cf4abe6-da2b-41b6-9caa-9659d4f750fb nodeName:}" failed. No retries permitted until 2026-10-30 23:41:29.96279462 +0000 UTC m=+2.999125423 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4cf4abe6-da2b-41b6-9caa-9659d4f750fb-metrics-certs") pod "router-default-66d97c9c97-qqxj5" (UID: "4cf4abe6-da2b-41b6-9caa-9659d4f750fb") : secret "router-metrics-certs-default" not found Oct 30 23:41:28.962831 ip-10-0-141-180 kubenswrapper[4927]: E1030 23:41:28.962810 4927 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/03211c6a-9761-419c-839b-21ca413d5bb2-cluster-monitoring-operator-tls podName:03211c6a-9761-419c-839b-21ca413d5bb2 nodeName:}" failed. No retries permitted until 2026-10-30 23:41:29.962805264 +0000 UTC m=+2.999136064 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/03211c6a-9761-419c-839b-21ca413d5bb2-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-httg9" (UID: "03211c6a-9761-419c-839b-21ca413d5bb2") : secret "cluster-monitoring-operator-tls" not found Oct 30 23:41:28.962831 ip-10-0-141-180 kubenswrapper[4927]: E1030 23:41:28.962825 4927 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4cf4abe6-da2b-41b6-9caa-9659d4f750fb-service-ca-bundle podName:4cf4abe6-da2b-41b6-9caa-9659d4f750fb nodeName:}" failed. No retries permitted until 2026-10-30 23:41:29.962818024 +0000 UTC m=+2.999148851 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/4cf4abe6-da2b-41b6-9caa-9659d4f750fb-service-ca-bundle") pod "router-default-66d97c9c97-qqxj5" (UID: "4cf4abe6-da2b-41b6-9caa-9659d4f750fb") : configmap references non-existent config key: service-ca.crt Oct 30 23:41:28.963543 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.962846 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ec330d4f-2eea-476a-a7b8-60cd295fddd2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sqv69\" (UID: \"ec330d4f-2eea-476a-a7b8-60cd295fddd2\") " pod="openshift-insights/insights-runtime-extractor-sqv69" Oct 30 23:41:28.963543 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.962883 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ec330d4f-2eea-476a-a7b8-60cd295fddd2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sqv69\" (UID: \"ec330d4f-2eea-476a-a7b8-60cd295fddd2\") " pod="openshift-insights/insights-runtime-extractor-sqv69" Oct 30 23:41:28.963543 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.962965 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/17461b46-7ebf-49b7-afaf-12e20e3994a3-registry-tls\") pod \"image-registry-dd954f877-x9h9g\" (UID: \"17461b46-7ebf-49b7-afaf-12e20e3994a3\") " pod="openshift-image-registry/image-registry-dd954f877-x9h9g" Oct 30 23:41:28.963543 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.963010 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b53ad6ed-90fd-4a0f-bf54-88d416ba19e9-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-rxpst\" (UID: \"b53ad6ed-90fd-4a0f-bf54-88d416ba19e9\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-rxpst" Oct 30 23:41:28.963543 ip-10-0-141-180 kubenswrapper[4927]: E1030 23:41:28.963034 4927 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 23:41:28.963543 ip-10-0-141-180 kubenswrapper[4927]: E1030 23:41:28.963105 4927 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 23:41:28.963543 ip-10-0-141-180 kubenswrapper[4927]: E1030 23:41:28.963114 4927 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec330d4f-2eea-476a-a7b8-60cd295fddd2-insights-runtime-extractor-tls podName:ec330d4f-2eea-476a-a7b8-60cd295fddd2 nodeName:}" failed. No retries permitted until 2026-10-30 23:41:29.963102948 +0000 UTC m=+2.999433747 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ec330d4f-2eea-476a-a7b8-60cd295fddd2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-sqv69" (UID: "ec330d4f-2eea-476a-a7b8-60cd295fddd2") : secret "insights-runtime-extractor-tls" not found Oct 30 23:41:28.963543 ip-10-0-141-180 kubenswrapper[4927]: E1030 23:41:28.963075 4927 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 23:41:28.963543 ip-10-0-141-180 kubenswrapper[4927]: E1030 23:41:28.963134 4927 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-dd954f877-x9h9g: secret "image-registry-tls" not found Oct 30 23:41:28.963543 ip-10-0-141-180 kubenswrapper[4927]: E1030 23:41:28.963151 4927 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b53ad6ed-90fd-4a0f-bf54-88d416ba19e9-networking-console-plugin-cert podName:b53ad6ed-90fd-4a0f-bf54-88d416ba19e9 nodeName:}" failed. No retries permitted until 2026-10-30 23:41:29.963138318 +0000 UTC m=+2.999469132 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/b53ad6ed-90fd-4a0f-bf54-88d416ba19e9-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-rxpst" (UID: "b53ad6ed-90fd-4a0f-bf54-88d416ba19e9") : secret "networking-console-plugin-cert" not found Oct 30 23:41:28.963543 ip-10-0-141-180 kubenswrapper[4927]: E1030 23:41:28.963176 4927 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/17461b46-7ebf-49b7-afaf-12e20e3994a3-registry-tls podName:17461b46-7ebf-49b7-afaf-12e20e3994a3 nodeName:}" failed. No retries permitted until 2026-10-30 23:41:29.963166578 +0000 UTC m=+2.999497378 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/17461b46-7ebf-49b7-afaf-12e20e3994a3-registry-tls") pod "image-registry-dd954f877-x9h9g" (UID: "17461b46-7ebf-49b7-afaf-12e20e3994a3") : secret "image-registry-tls" not found Oct 30 23:41:28.963543 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:28.963441 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ec330d4f-2eea-476a-a7b8-60cd295fddd2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sqv69\" (UID: \"ec330d4f-2eea-476a-a7b8-60cd295fddd2\") " pod="openshift-insights/insights-runtime-extractor-sqv69" Oct 30 23:41:29.141623 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:29.141550 4927 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-pqt2c_6382dfb5-a363-47a2-bc79-0b96f7bec0c3/console-operator/2.log" Oct 30 23:41:29.142040 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:29.141965 4927 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-pqt2c_6382dfb5-a363-47a2-bc79-0b96f7bec0c3/console-operator/1.log" Oct 30 23:41:29.142040 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:29.142004 4927 generic.go:356] "Generic (PLEG): container finished" podID="6382dfb5-a363-47a2-bc79-0b96f7bec0c3" containerID="64081a16fba6223deec2f4d14ac5189bbc15017349d35d38b83aae316589b8e4" exitCode=255 Oct 30 23:41:29.142160 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:29.142076 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-pqt2c" event={"ID":"6382dfb5-a363-47a2-bc79-0b96f7bec0c3","Type":"ContainerDied","Data":"64081a16fba6223deec2f4d14ac5189bbc15017349d35d38b83aae316589b8e4"} Oct 30 23:41:29.142213 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:29.142178 4927 scope.go:122] "RemoveContainer" containerID="e373ab1ed0edd47d38ff51aab784dee07bf119717f872d40609595f835e4357c" Oct 30 23:41:29.142653 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:29.142634 4927 scope.go:122] "RemoveContainer" containerID="64081a16fba6223deec2f4d14ac5189bbc15017349d35d38b83aae316589b8e4" Oct 30 23:41:29.142804 ip-10-0-141-180 kubenswrapper[4927]: E1030 23:41:29.142783 4927 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-pqt2c_openshift-console-operator(6382dfb5-a363-47a2-bc79-0b96f7bec0c3)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-pqt2c" podUID="6382dfb5-a363-47a2-bc79-0b96f7bec0c3" Oct 30 23:41:29.345244 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:29.345216 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-pqt2c" Oct 30 23:41:29.380962 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:29.380939 4927 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-7sbzf_99f800b6-a38c-46ca-8234-0dbd3610c8fd/node-ca/0.log" Oct 30 23:41:29.972188 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:29.972155 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/29b8bd1a-beeb-495c-9f61-e3326866a1d7-metrics-tls\") pod \"dns-default-9l5zq\" (UID: \"29b8bd1a-beeb-495c-9f61-e3326866a1d7\") " pod="openshift-dns/dns-default-9l5zq" Oct 30 23:41:29.972397 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:29.972205 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/145dd323-9370-4b9d-a96e-8b93d0a47a7c-metrics-certs\") pod \"network-metrics-daemon-pt58s\" (UID: \"145dd323-9370-4b9d-a96e-8b93d0a47a7c\") " pod="openshift-multus/network-metrics-daemon-pt58s" Oct 30 23:41:29.972397 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:29.972238 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4cf4abe6-da2b-41b6-9caa-9659d4f750fb-metrics-certs\") pod \"router-default-66d97c9c97-qqxj5\" (UID: \"4cf4abe6-da2b-41b6-9caa-9659d4f750fb\") " pod="openshift-ingress/router-default-66d97c9c97-qqxj5" Oct 30 23:41:29.972397 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:29.972263 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/58f7cf25-f5d0-4481-a927-4a745b692f7a-registry-tls\") pod \"image-registry-544b44f8d-5p6kz\" (UID: \"58f7cf25-f5d0-4481-a927-4a745b692f7a\") " pod="openshift-image-registry/image-registry-544b44f8d-5p6kz" Oct 30 23:41:29.972397 ip-10-0-141-180 kubenswrapper[4927]: E1030 23:41:29.972313 4927 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 23:41:29.972397 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:29.972319 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/03211c6a-9761-419c-839b-21ca413d5bb2-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-httg9\" (UID: \"03211c6a-9761-419c-839b-21ca413d5bb2\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-httg9" Oct 30 23:41:29.972397 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:29.972370 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4cf4abe6-da2b-41b6-9caa-9659d4f750fb-service-ca-bundle\") pod \"router-default-66d97c9c97-qqxj5\" (UID: \"4cf4abe6-da2b-41b6-9caa-9659d4f750fb\") " pod="openshift-ingress/router-default-66d97c9c97-qqxj5" Oct 30 23:41:29.972397 ip-10-0-141-180 kubenswrapper[4927]: E1030 23:41:29.972391 4927 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/29b8bd1a-beeb-495c-9f61-e3326866a1d7-metrics-tls podName:29b8bd1a-beeb-495c-9f61-e3326866a1d7 nodeName:}" failed. No retries permitted until 2026-10-30 23:41:31.972373979 +0000 UTC m=+5.008704795 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/29b8bd1a-beeb-495c-9f61-e3326866a1d7-metrics-tls") pod "dns-default-9l5zq" (UID: "29b8bd1a-beeb-495c-9f61-e3326866a1d7") : secret "dns-default-metrics-tls" not found Oct 30 23:41:29.972810 ip-10-0-141-180 kubenswrapper[4927]: E1030 23:41:29.972403 4927 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 30 23:41:29.972810 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:29.972428 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ec330d4f-2eea-476a-a7b8-60cd295fddd2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sqv69\" (UID: \"ec330d4f-2eea-476a-a7b8-60cd295fddd2\") " pod="openshift-insights/insights-runtime-extractor-sqv69" Oct 30 23:41:29.972810 ip-10-0-141-180 kubenswrapper[4927]: E1030 23:41:29.972441 4927 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4cf4abe6-da2b-41b6-9caa-9659d4f750fb-service-ca-bundle podName:4cf4abe6-da2b-41b6-9caa-9659d4f750fb nodeName:}" failed. No retries permitted until 2026-10-30 23:41:31.972429878 +0000 UTC m=+5.008760684 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/4cf4abe6-da2b-41b6-9caa-9659d4f750fb-service-ca-bundle") pod "router-default-66d97c9c97-qqxj5" (UID: "4cf4abe6-da2b-41b6-9caa-9659d4f750fb") : configmap references non-existent config key: service-ca.crt Oct 30 23:41:29.972810 ip-10-0-141-180 kubenswrapper[4927]: E1030 23:41:29.972490 4927 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 23:41:29.972810 ip-10-0-141-180 kubenswrapper[4927]: E1030 23:41:29.972494 4927 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 23:41:29.972810 ip-10-0-141-180 kubenswrapper[4927]: E1030 23:41:29.972563 4927 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 23:41:29.972810 ip-10-0-141-180 kubenswrapper[4927]: E1030 23:41:29.972566 4927 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 23:41:29.972810 ip-10-0-141-180 kubenswrapper[4927]: E1030 23:41:29.972576 4927 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-544b44f8d-5p6kz: secret "image-registry-tls" not found Oct 30 23:41:29.972810 ip-10-0-141-180 kubenswrapper[4927]: E1030 23:41:29.972606 4927 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/145dd323-9370-4b9d-a96e-8b93d0a47a7c-metrics-certs podName:145dd323-9370-4b9d-a96e-8b93d0a47a7c nodeName:}" failed. No retries permitted until 2026-10-30 23:41:31.972595504 +0000 UTC m=+5.008926310 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/145dd323-9370-4b9d-a96e-8b93d0a47a7c-metrics-certs") pod "network-metrics-daemon-pt58s" (UID: "145dd323-9370-4b9d-a96e-8b93d0a47a7c") : secret "metrics-daemon-secret" not found Oct 30 23:41:29.972810 ip-10-0-141-180 kubenswrapper[4927]: E1030 23:41:29.972621 4927 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec330d4f-2eea-476a-a7b8-60cd295fddd2-insights-runtime-extractor-tls podName:ec330d4f-2eea-476a-a7b8-60cd295fddd2 nodeName:}" failed. No retries permitted until 2026-10-30 23:41:31.972615025 +0000 UTC m=+5.008945824 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ec330d4f-2eea-476a-a7b8-60cd295fddd2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-sqv69" (UID: "ec330d4f-2eea-476a-a7b8-60cd295fddd2") : secret "insights-runtime-extractor-tls" not found Oct 30 23:41:29.972810 ip-10-0-141-180 kubenswrapper[4927]: E1030 23:41:29.972631 4927 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4cf4abe6-da2b-41b6-9caa-9659d4f750fb-metrics-certs podName:4cf4abe6-da2b-41b6-9caa-9659d4f750fb nodeName:}" failed. No retries permitted until 2026-10-30 23:41:31.972626432 +0000 UTC m=+5.008957231 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4cf4abe6-da2b-41b6-9caa-9659d4f750fb-metrics-certs") pod "router-default-66d97c9c97-qqxj5" (UID: "4cf4abe6-da2b-41b6-9caa-9659d4f750fb") : secret "router-metrics-certs-default" not found Oct 30 23:41:29.972810 ip-10-0-141-180 kubenswrapper[4927]: E1030 23:41:29.972667 4927 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/03211c6a-9761-419c-839b-21ca413d5bb2-cluster-monitoring-operator-tls podName:03211c6a-9761-419c-839b-21ca413d5bb2 nodeName:}" failed. No retries permitted until 2026-10-30 23:41:31.972642986 +0000 UTC m=+5.008973786 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/03211c6a-9761-419c-839b-21ca413d5bb2-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-httg9" (UID: "03211c6a-9761-419c-839b-21ca413d5bb2") : secret "cluster-monitoring-operator-tls" not found Oct 30 23:41:29.972810 ip-10-0-141-180 kubenswrapper[4927]: E1030 23:41:29.972677 4927 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/58f7cf25-f5d0-4481-a927-4a745b692f7a-registry-tls podName:58f7cf25-f5d0-4481-a927-4a745b692f7a nodeName:}" failed. No retries permitted until 2026-10-30 23:41:31.972672332 +0000 UTC m=+5.009003131 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/58f7cf25-f5d0-4481-a927-4a745b692f7a-registry-tls") pod "image-registry-544b44f8d-5p6kz" (UID: "58f7cf25-f5d0-4481-a927-4a745b692f7a") : secret "image-registry-tls" not found Oct 30 23:41:29.972810 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:29.972689 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/17461b46-7ebf-49b7-afaf-12e20e3994a3-registry-tls\") pod \"image-registry-dd954f877-x9h9g\" (UID: \"17461b46-7ebf-49b7-afaf-12e20e3994a3\") " pod="openshift-image-registry/image-registry-dd954f877-x9h9g" Oct 30 23:41:29.972810 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:29.972720 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b53ad6ed-90fd-4a0f-bf54-88d416ba19e9-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-rxpst\" (UID: \"b53ad6ed-90fd-4a0f-bf54-88d416ba19e9\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-rxpst" Oct 30 23:41:29.972810 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:29.972759 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/be5540b5-7b10-45b2-b40c-723efd9f1b01-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-vl8t5\" (UID: \"be5540b5-7b10-45b2-b40c-723efd9f1b01\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vl8t5" Oct 30 23:41:29.973588 ip-10-0-141-180 kubenswrapper[4927]: E1030 23:41:29.972820 4927 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 23:41:29.973588 ip-10-0-141-180 kubenswrapper[4927]: E1030 23:41:29.972838 4927 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-dd954f877-x9h9g: secret "image-registry-tls" not found Oct 30 23:41:29.973588 ip-10-0-141-180 kubenswrapper[4927]: E1030 23:41:29.972865 4927 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 23:41:29.973588 ip-10-0-141-180 kubenswrapper[4927]: E1030 23:41:29.972827 4927 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 23:41:29.973588 ip-10-0-141-180 kubenswrapper[4927]: E1030 23:41:29.972891 4927 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/17461b46-7ebf-49b7-afaf-12e20e3994a3-registry-tls podName:17461b46-7ebf-49b7-afaf-12e20e3994a3 nodeName:}" failed. No retries permitted until 2026-10-30 23:41:31.972878994 +0000 UTC m=+5.009209812 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/17461b46-7ebf-49b7-afaf-12e20e3994a3-registry-tls") pod "image-registry-dd954f877-x9h9g" (UID: "17461b46-7ebf-49b7-afaf-12e20e3994a3") : secret "image-registry-tls" not found Oct 30 23:41:29.973588 ip-10-0-141-180 kubenswrapper[4927]: E1030 23:41:29.972906 4927 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/be5540b5-7b10-45b2-b40c-723efd9f1b01-samples-operator-tls podName:be5540b5-7b10-45b2-b40c-723efd9f1b01 nodeName:}" failed. No retries permitted until 2026-10-30 23:41:31.972897223 +0000 UTC m=+5.009228062 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/be5540b5-7b10-45b2-b40c-723efd9f1b01-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-vl8t5" (UID: "be5540b5-7b10-45b2-b40c-723efd9f1b01") : secret "samples-operator-tls" not found Oct 30 23:41:29.973588 ip-10-0-141-180 kubenswrapper[4927]: E1030 23:41:29.972929 4927 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b53ad6ed-90fd-4a0f-bf54-88d416ba19e9-networking-console-plugin-cert podName:b53ad6ed-90fd-4a0f-bf54-88d416ba19e9 nodeName:}" failed. No retries permitted until 2026-10-30 23:41:31.972921406 +0000 UTC m=+5.009252205 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/b53ad6ed-90fd-4a0f-bf54-88d416ba19e9-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-rxpst" (UID: "b53ad6ed-90fd-4a0f-bf54-88d416ba19e9") : secret "networking-console-plugin-cert" not found Oct 30 23:41:30.146956 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:30.146924 4927 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-pqt2c_6382dfb5-a363-47a2-bc79-0b96f7bec0c3/console-operator/2.log" Oct 30 23:41:30.147365 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:30.147244 4927 scope.go:122] "RemoveContainer" containerID="64081a16fba6223deec2f4d14ac5189bbc15017349d35d38b83aae316589b8e4" Oct 30 23:41:30.147471 ip-10-0-141-180 kubenswrapper[4927]: E1030 23:41:30.147448 4927 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-pqt2c_openshift-console-operator(6382dfb5-a363-47a2-bc79-0b96f7bec0c3)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-pqt2c" podUID="6382dfb5-a363-47a2-bc79-0b96f7bec0c3" Oct 30 23:41:30.201852 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:30.201823 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:41:30.381875 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:30.381807 4927 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-46zqw_9700c98a-b858-45a5-9025-8d75f3c14044/serve-healthcheck-canary/0.log" Oct 30 23:41:30.918018 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:30.917988 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-kgdzs" Oct 30 23:41:30.985120 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:30.985090 4927 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7d4bc45dd6-9qs84_840aaae0-a215-4ccd-b420-d6f5723b65ae/kube-storage-version-migrator-operator/0.log" Oct 30 23:41:31.151320 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:31.151246 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-8khpq" event={"ID":"aa90814d-5b4d-4039-917d-76610b1f88e4","Type":"ContainerStarted","Data":"79fe6eb1345c6e655f01befb6fc200dd13d5771744113c6570f1f708d8206144"} Oct 30 23:41:31.421382 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:31.421353 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:41:31.566082 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:31.566046 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:41:31.639709 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:31.639685 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-26v6x" Oct 30 23:41:31.992164 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:31.992125 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/03211c6a-9761-419c-839b-21ca413d5bb2-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-httg9\" (UID: \"03211c6a-9761-419c-839b-21ca413d5bb2\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-httg9" Oct 30 23:41:31.992354 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:31.992171 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4cf4abe6-da2b-41b6-9caa-9659d4f750fb-service-ca-bundle\") pod \"router-default-66d97c9c97-qqxj5\" (UID: \"4cf4abe6-da2b-41b6-9caa-9659d4f750fb\") " pod="openshift-ingress/router-default-66d97c9c97-qqxj5" Oct 30 23:41:31.992354 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:31.992205 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ec330d4f-2eea-476a-a7b8-60cd295fddd2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sqv69\" (UID: \"ec330d4f-2eea-476a-a7b8-60cd295fddd2\") " pod="openshift-insights/insights-runtime-extractor-sqv69" Oct 30 23:41:31.992354 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:31.992234 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/17461b46-7ebf-49b7-afaf-12e20e3994a3-registry-tls\") pod \"image-registry-dd954f877-x9h9g\" (UID: \"17461b46-7ebf-49b7-afaf-12e20e3994a3\") " pod="openshift-image-registry/image-registry-dd954f877-x9h9g" Oct 30 23:41:31.992354 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:31.992268 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b53ad6ed-90fd-4a0f-bf54-88d416ba19e9-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-rxpst\" (UID: \"b53ad6ed-90fd-4a0f-bf54-88d416ba19e9\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-rxpst" Oct 30 23:41:31.992354 ip-10-0-141-180 kubenswrapper[4927]: E1030 23:41:31.992283 4927 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 23:41:31.992354 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:31.992306 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/be5540b5-7b10-45b2-b40c-723efd9f1b01-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-vl8t5\" (UID: \"be5540b5-7b10-45b2-b40c-723efd9f1b01\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vl8t5" Oct 30 23:41:31.992658 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:31.992360 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/29b8bd1a-beeb-495c-9f61-e3326866a1d7-metrics-tls\") pod \"dns-default-9l5zq\" (UID: \"29b8bd1a-beeb-495c-9f61-e3326866a1d7\") " pod="openshift-dns/dns-default-9l5zq" Oct 30 23:41:31.992658 ip-10-0-141-180 kubenswrapper[4927]: E1030 23:41:31.992376 4927 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/03211c6a-9761-419c-839b-21ca413d5bb2-cluster-monitoring-operator-tls podName:03211c6a-9761-419c-839b-21ca413d5bb2 nodeName:}" failed. No retries permitted until 2026-10-30 23:41:35.992355233 +0000 UTC m=+9.028686053 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/03211c6a-9761-419c-839b-21ca413d5bb2-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-httg9" (UID: "03211c6a-9761-419c-839b-21ca413d5bb2") : secret "cluster-monitoring-operator-tls" not found Oct 30 23:41:31.992658 ip-10-0-141-180 kubenswrapper[4927]: E1030 23:41:31.992387 4927 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 23:41:31.992658 ip-10-0-141-180 kubenswrapper[4927]: E1030 23:41:31.992412 4927 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-dd954f877-x9h9g: secret "image-registry-tls" not found Oct 30 23:41:31.992658 ip-10-0-141-180 kubenswrapper[4927]: E1030 23:41:31.992420 4927 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4cf4abe6-da2b-41b6-9caa-9659d4f750fb-service-ca-bundle podName:4cf4abe6-da2b-41b6-9caa-9659d4f750fb nodeName:}" failed. No retries permitted until 2026-10-30 23:41:35.992401863 +0000 UTC m=+9.028732663 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/4cf4abe6-da2b-41b6-9caa-9659d4f750fb-service-ca-bundle") pod "router-default-66d97c9c97-qqxj5" (UID: "4cf4abe6-da2b-41b6-9caa-9659d4f750fb") : configmap references non-existent config key: service-ca.crt Oct 30 23:41:31.992658 ip-10-0-141-180 kubenswrapper[4927]: E1030 23:41:31.992431 4927 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 23:41:31.992658 ip-10-0-141-180 kubenswrapper[4927]: E1030 23:41:31.992422 4927 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 23:41:31.992658 ip-10-0-141-180 kubenswrapper[4927]: E1030 23:41:31.992463 4927 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/17461b46-7ebf-49b7-afaf-12e20e3994a3-registry-tls podName:17461b46-7ebf-49b7-afaf-12e20e3994a3 nodeName:}" failed. No retries permitted until 2026-10-30 23:41:35.992448712 +0000 UTC m=+9.028779539 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/17461b46-7ebf-49b7-afaf-12e20e3994a3-registry-tls") pod "image-registry-dd954f877-x9h9g" (UID: "17461b46-7ebf-49b7-afaf-12e20e3994a3") : secret "image-registry-tls" not found Oct 30 23:41:31.992658 ip-10-0-141-180 kubenswrapper[4927]: E1030 23:41:31.992488 4927 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/be5540b5-7b10-45b2-b40c-723efd9f1b01-samples-operator-tls podName:be5540b5-7b10-45b2-b40c-723efd9f1b01 nodeName:}" failed. No retries permitted until 2026-10-30 23:41:35.992482208 +0000 UTC m=+9.028813012 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/be5540b5-7b10-45b2-b40c-723efd9f1b01-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-vl8t5" (UID: "be5540b5-7b10-45b2-b40c-723efd9f1b01") : secret "samples-operator-tls" not found Oct 30 23:41:31.992658 ip-10-0-141-180 kubenswrapper[4927]: E1030 23:41:31.992423 4927 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 23:41:31.992658 ip-10-0-141-180 kubenswrapper[4927]: E1030 23:41:31.992387 4927 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 23:41:31.992658 ip-10-0-141-180 kubenswrapper[4927]: E1030 23:41:31.992521 4927 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/29b8bd1a-beeb-495c-9f61-e3326866a1d7-metrics-tls podName:29b8bd1a-beeb-495c-9f61-e3326866a1d7 nodeName:}" failed. No retries permitted until 2026-10-30 23:41:35.99250754 +0000 UTC m=+9.028838354 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/29b8bd1a-beeb-495c-9f61-e3326866a1d7-metrics-tls") pod "dns-default-9l5zq" (UID: "29b8bd1a-beeb-495c-9f61-e3326866a1d7") : secret "dns-default-metrics-tls" not found Oct 30 23:41:31.992658 ip-10-0-141-180 kubenswrapper[4927]: E1030 23:41:31.992637 4927 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b53ad6ed-90fd-4a0f-bf54-88d416ba19e9-networking-console-plugin-cert podName:b53ad6ed-90fd-4a0f-bf54-88d416ba19e9 nodeName:}" failed. No retries permitted until 2026-10-30 23:41:35.992625467 +0000 UTC m=+9.028956269 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/b53ad6ed-90fd-4a0f-bf54-88d416ba19e9-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-rxpst" (UID: "b53ad6ed-90fd-4a0f-bf54-88d416ba19e9") : secret "networking-console-plugin-cert" not found Oct 30 23:41:31.992658 ip-10-0-141-180 kubenswrapper[4927]: E1030 23:41:31.992652 4927 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec330d4f-2eea-476a-a7b8-60cd295fddd2-insights-runtime-extractor-tls podName:ec330d4f-2eea-476a-a7b8-60cd295fddd2 nodeName:}" failed. No retries permitted until 2026-10-30 23:41:35.992643841 +0000 UTC m=+9.028974641 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ec330d4f-2eea-476a-a7b8-60cd295fddd2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-sqv69" (UID: "ec330d4f-2eea-476a-a7b8-60cd295fddd2") : secret "insights-runtime-extractor-tls" not found Oct 30 23:41:31.993075 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:31.992672 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/145dd323-9370-4b9d-a96e-8b93d0a47a7c-metrics-certs\") pod \"network-metrics-daemon-pt58s\" (UID: \"145dd323-9370-4b9d-a96e-8b93d0a47a7c\") " pod="openshift-multus/network-metrics-daemon-pt58s" Oct 30 23:41:31.993075 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:31.992708 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4cf4abe6-da2b-41b6-9caa-9659d4f750fb-metrics-certs\") pod \"router-default-66d97c9c97-qqxj5\" (UID: \"4cf4abe6-da2b-41b6-9caa-9659d4f750fb\") " pod="openshift-ingress/router-default-66d97c9c97-qqxj5" Oct 30 23:41:31.993075 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:31.992736 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/58f7cf25-f5d0-4481-a927-4a745b692f7a-registry-tls\") pod \"image-registry-544b44f8d-5p6kz\" (UID: \"58f7cf25-f5d0-4481-a927-4a745b692f7a\") " pod="openshift-image-registry/image-registry-544b44f8d-5p6kz" Oct 30 23:41:31.993075 ip-10-0-141-180 kubenswrapper[4927]: E1030 23:41:31.992763 4927 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 30 23:41:31.993075 ip-10-0-141-180 kubenswrapper[4927]: E1030 23:41:31.992799 4927 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/145dd323-9370-4b9d-a96e-8b93d0a47a7c-metrics-certs podName:145dd323-9370-4b9d-a96e-8b93d0a47a7c nodeName:}" failed. No retries permitted until 2026-10-30 23:41:35.992787371 +0000 UTC m=+9.029118202 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/145dd323-9370-4b9d-a96e-8b93d0a47a7c-metrics-certs") pod "network-metrics-daemon-pt58s" (UID: "145dd323-9370-4b9d-a96e-8b93d0a47a7c") : secret "metrics-daemon-secret" not found Oct 30 23:41:31.993075 ip-10-0-141-180 kubenswrapper[4927]: E1030 23:41:31.992805 4927 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 23:41:31.993075 ip-10-0-141-180 kubenswrapper[4927]: E1030 23:41:31.992838 4927 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 23:41:31.993075 ip-10-0-141-180 kubenswrapper[4927]: E1030 23:41:31.992848 4927 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-544b44f8d-5p6kz: secret "image-registry-tls" not found Oct 30 23:41:31.993075 ip-10-0-141-180 kubenswrapper[4927]: E1030 23:41:31.992859 4927 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4cf4abe6-da2b-41b6-9caa-9659d4f750fb-metrics-certs podName:4cf4abe6-da2b-41b6-9caa-9659d4f750fb nodeName:}" failed. No retries permitted until 2026-10-30 23:41:35.992848734 +0000 UTC m=+9.029179537 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4cf4abe6-da2b-41b6-9caa-9659d4f750fb-metrics-certs") pod "router-default-66d97c9c97-qqxj5" (UID: "4cf4abe6-da2b-41b6-9caa-9659d4f750fb") : secret "router-metrics-certs-default" not found Oct 30 23:41:31.993075 ip-10-0-141-180 kubenswrapper[4927]: E1030 23:41:31.992875 4927 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/58f7cf25-f5d0-4481-a927-4a745b692f7a-registry-tls podName:58f7cf25-f5d0-4481-a927-4a745b692f7a nodeName:}" failed. No retries permitted until 2026-10-30 23:41:35.992868488 +0000 UTC m=+9.029199287 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/58f7cf25-f5d0-4481-a927-4a745b692f7a-registry-tls") pod "image-registry-544b44f8d-5p6kz" (UID: "58f7cf25-f5d0-4481-a927-4a745b692f7a") : secret "image-registry-tls" not found Oct 30 23:41:32.300541 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:32.300458 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:41:32.706025 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:32.705983 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:41:32.890044 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:32.890005 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:41:33.198276 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:33.198245 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:41:33.693779 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:33.693735 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:41:33.843416 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:33.843384 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:41:33.998523 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:33.998452 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:41:34.398920 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:34.398843 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:41:34.510623 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:34.510590 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-26v6x" Oct 30 23:41:34.510908 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:34.510890 4927 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 23:41:34.596325 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:34.596288 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:41:34.689639 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:34.689607 4927 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-pqt2c" Oct 30 23:41:34.690034 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:34.690015 4927 scope.go:122] "RemoveContainer" containerID="64081a16fba6223deec2f4d14ac5189bbc15017349d35d38b83aae316589b8e4" Oct 30 23:41:34.690248 ip-10-0-141-180 kubenswrapper[4927]: E1030 23:41:34.690226 4927 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-pqt2c_openshift-console-operator(6382dfb5-a363-47a2-bc79-0b96f7bec0c3)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-pqt2c" podUID="6382dfb5-a363-47a2-bc79-0b96f7bec0c3" Oct 30 23:41:34.750349 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:34.750241 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:41:34.897697 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:34.897657 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:41:35.195174 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:35.195143 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:41:35.689850 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:35.689799 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:41:35.994346 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:35.994261 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:41:36.030880 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:36.030846 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/145dd323-9370-4b9d-a96e-8b93d0a47a7c-metrics-certs\") pod \"network-metrics-daemon-pt58s\" (UID: \"145dd323-9370-4b9d-a96e-8b93d0a47a7c\") " pod="openshift-multus/network-metrics-daemon-pt58s" Oct 30 23:41:36.031044 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:36.030888 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4cf4abe6-da2b-41b6-9caa-9659d4f750fb-metrics-certs\") pod \"router-default-66d97c9c97-qqxj5\" (UID: \"4cf4abe6-da2b-41b6-9caa-9659d4f750fb\") " pod="openshift-ingress/router-default-66d97c9c97-qqxj5" Oct 30 23:41:36.031044 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:36.030919 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/58f7cf25-f5d0-4481-a927-4a745b692f7a-registry-tls\") pod \"image-registry-544b44f8d-5p6kz\" (UID: \"58f7cf25-f5d0-4481-a927-4a745b692f7a\") " pod="openshift-image-registry/image-registry-544b44f8d-5p6kz" Oct 30 23:41:36.031044 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:36.030974 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/03211c6a-9761-419c-839b-21ca413d5bb2-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-httg9\" (UID: \"03211c6a-9761-419c-839b-21ca413d5bb2\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-httg9" Oct 30 23:41:36.031044 ip-10-0-141-180 kubenswrapper[4927]: E1030 23:41:36.030991 4927 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 23:41:36.031044 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:36.031005 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4cf4abe6-da2b-41b6-9caa-9659d4f750fb-service-ca-bundle\") pod \"router-default-66d97c9c97-qqxj5\" (UID: \"4cf4abe6-da2b-41b6-9caa-9659d4f750fb\") " pod="openshift-ingress/router-default-66d97c9c97-qqxj5" Oct 30 23:41:36.031044 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:36.031035 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ec330d4f-2eea-476a-a7b8-60cd295fddd2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sqv69\" (UID: \"ec330d4f-2eea-476a-a7b8-60cd295fddd2\") " pod="openshift-insights/insights-runtime-extractor-sqv69" Oct 30 23:41:36.031391 ip-10-0-141-180 kubenswrapper[4927]: E1030 23:41:36.031054 4927 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4cf4abe6-da2b-41b6-9caa-9659d4f750fb-metrics-certs podName:4cf4abe6-da2b-41b6-9caa-9659d4f750fb nodeName:}" failed. No retries permitted until 2026-10-30 23:41:44.031039801 +0000 UTC m=+17.067370616 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4cf4abe6-da2b-41b6-9caa-9659d4f750fb-metrics-certs") pod "router-default-66d97c9c97-qqxj5" (UID: "4cf4abe6-da2b-41b6-9caa-9659d4f750fb") : secret "router-metrics-certs-default" not found Oct 30 23:41:36.031391 ip-10-0-141-180 kubenswrapper[4927]: E1030 23:41:36.031112 4927 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 23:41:36.031391 ip-10-0-141-180 kubenswrapper[4927]: E1030 23:41:36.031130 4927 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4cf4abe6-da2b-41b6-9caa-9659d4f750fb-service-ca-bundle podName:4cf4abe6-da2b-41b6-9caa-9659d4f750fb nodeName:}" failed. No retries permitted until 2026-10-30 23:41:44.031114812 +0000 UTC m=+17.067445620 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/4cf4abe6-da2b-41b6-9caa-9659d4f750fb-service-ca-bundle") pod "router-default-66d97c9c97-qqxj5" (UID: "4cf4abe6-da2b-41b6-9caa-9659d4f750fb") : configmap references non-existent config key: service-ca.crt Oct 30 23:41:36.031391 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:36.031156 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/17461b46-7ebf-49b7-afaf-12e20e3994a3-registry-tls\") pod \"image-registry-dd954f877-x9h9g\" (UID: \"17461b46-7ebf-49b7-afaf-12e20e3994a3\") " pod="openshift-image-registry/image-registry-dd954f877-x9h9g" Oct 30 23:41:36.031391 ip-10-0-141-180 kubenswrapper[4927]: E1030 23:41:36.031166 4927 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/03211c6a-9761-419c-839b-21ca413d5bb2-cluster-monitoring-operator-tls podName:03211c6a-9761-419c-839b-21ca413d5bb2 nodeName:}" failed. No retries permitted until 2026-10-30 23:41:44.031150048 +0000 UTC m=+17.067480863 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/03211c6a-9761-419c-839b-21ca413d5bb2-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-httg9" (UID: "03211c6a-9761-419c-839b-21ca413d5bb2") : secret "cluster-monitoring-operator-tls" not found Oct 30 23:41:36.031391 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:36.031218 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b53ad6ed-90fd-4a0f-bf54-88d416ba19e9-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-rxpst\" (UID: \"b53ad6ed-90fd-4a0f-bf54-88d416ba19e9\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-rxpst" Oct 30 23:41:36.031391 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:36.031269 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/be5540b5-7b10-45b2-b40c-723efd9f1b01-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-vl8t5\" (UID: \"be5540b5-7b10-45b2-b40c-723efd9f1b01\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vl8t5" Oct 30 23:41:36.031391 ip-10-0-141-180 kubenswrapper[4927]: E1030 23:41:36.031226 4927 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 23:41:36.031391 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:36.031307 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/29b8bd1a-beeb-495c-9f61-e3326866a1d7-metrics-tls\") pod \"dns-default-9l5zq\" (UID: \"29b8bd1a-beeb-495c-9f61-e3326866a1d7\") " pod="openshift-dns/dns-default-9l5zq" Oct 30 23:41:36.031391 ip-10-0-141-180 kubenswrapper[4927]: E1030 23:41:36.031387 4927 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 23:41:36.031893 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:36.031309 4927 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 23:41:36.031893 ip-10-0-141-180 kubenswrapper[4927]: E1030 23:41:36.031406 4927 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec330d4f-2eea-476a-a7b8-60cd295fddd2-insights-runtime-extractor-tls podName:ec330d4f-2eea-476a-a7b8-60cd295fddd2 nodeName:}" failed. No retries permitted until 2026-10-30 23:41:44.031387715 +0000 UTC m=+17.067718532 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ec330d4f-2eea-476a-a7b8-60cd295fddd2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-sqv69" (UID: "ec330d4f-2eea-476a-a7b8-60cd295fddd2") : secret "insights-runtime-extractor-tls" not found Oct 30 23:41:36.031893 ip-10-0-141-180 kubenswrapper[4927]: E1030 23:41:36.031426 4927 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/29b8bd1a-beeb-495c-9f61-e3326866a1d7-metrics-tls podName:29b8bd1a-beeb-495c-9f61-e3326866a1d7 nodeName:}" failed. No retries permitted until 2026-10-30 23:41:44.031415105 +0000 UTC m=+17.067745906 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/29b8bd1a-beeb-495c-9f61-e3326866a1d7-metrics-tls") pod "dns-default-9l5zq" (UID: "29b8bd1a-beeb-495c-9f61-e3326866a1d7") : secret "dns-default-metrics-tls" not found Oct 30 23:41:36.033857 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:36.033826 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/17461b46-7ebf-49b7-afaf-12e20e3994a3-registry-tls\") pod \"image-registry-dd954f877-x9h9g\" (UID: \"17461b46-7ebf-49b7-afaf-12e20e3994a3\") " pod="openshift-image-registry/image-registry-dd954f877-x9h9g" Oct 30 23:41:36.033966 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:36.033851 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/be5540b5-7b10-45b2-b40c-723efd9f1b01-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-vl8t5\" (UID: \"be5540b5-7b10-45b2-b40c-723efd9f1b01\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vl8t5" Oct 30 23:41:36.033966 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:36.033916 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/145dd323-9370-4b9d-a96e-8b93d0a47a7c-metrics-certs\") pod \"network-metrics-daemon-pt58s\" (UID: \"145dd323-9370-4b9d-a96e-8b93d0a47a7c\") " pod="openshift-multus/network-metrics-daemon-pt58s" Oct 30 23:41:36.034044 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:36.033993 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/58f7cf25-f5d0-4481-a927-4a745b692f7a-registry-tls\") pod \"image-registry-544b44f8d-5p6kz\" (UID: \"58f7cf25-f5d0-4481-a927-4a745b692f7a\") " pod="openshift-image-registry/image-registry-544b44f8d-5p6kz" Oct 30 23:41:36.034085 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:36.034062 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b53ad6ed-90fd-4a0f-bf54-88d416ba19e9-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-rxpst\" (UID: \"b53ad6ed-90fd-4a0f-bf54-88d416ba19e9\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-rxpst" Oct 30 23:41:36.285549 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:36.285475 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vl8t5" Oct 30 23:41:36.286590 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:36.286569 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-dd954f877-x9h9g" Oct 30 23:41:36.288047 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:36.288033 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-rxpst" Oct 30 23:41:36.290323 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:36.290209 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pt58s" Oct 30 23:41:36.298826 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:36.298803 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-544b44f8d-5p6kz" Oct 30 23:41:36.398133 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:36.398102 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:41:36.597414 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:36.597317 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:41:36.678841 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:36.678810 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vl8t5"] Oct 30 23:41:36.728419 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:36.728384 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-pt58s"] Oct 30 23:41:36.731492 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:36.731206 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-rxpst"] Oct 30 23:41:36.773327 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:36.773300 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb53ad6ed_90fd_4a0f_bf54_88d416ba19e9.slice/crio-a943303a74887a9d9b37f2691bb4e033e907feed48c6914d086c7bc3da0eb912 WatchSource:0}: Error finding container a943303a74887a9d9b37f2691bb4e033e907feed48c6914d086c7bc3da0eb912: Status 404 returned error can't find the container with id a943303a74887a9d9b37f2691bb4e033e907feed48c6914d086c7bc3da0eb912 Oct 30 23:41:36.773979 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:36.773909 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17461b46_7ebf_49b7_afaf_12e20e3994a3.slice/crio-7a6060d10f1c9eee9db00524cfd3379928a176a560d3cadb832a71aace5e50a4 WatchSource:0}: Error finding container 7a6060d10f1c9eee9db00524cfd3379928a176a560d3cadb832a71aace5e50a4: Status 404 returned error can't find the container with id 7a6060d10f1c9eee9db00524cfd3379928a176a560d3cadb832a71aace5e50a4 Oct 30 23:41:36.779911 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:36.779887 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-dd954f877-x9h9g"] Oct 30 23:41:36.785181 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:36.785157 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-544b44f8d-5p6kz"] Oct 30 23:41:36.788597 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:36.788576 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58f7cf25_f5d0_4481_a927_4a745b692f7a.slice/crio-031752aff7fa5ac6deb70609bad56693e9a040632954724c65c7ac153132fae8 WatchSource:0}: Error finding container 031752aff7fa5ac6deb70609bad56693e9a040632954724c65c7ac153132fae8: Status 404 returned error can't find the container with id 031752aff7fa5ac6deb70609bad56693e9a040632954724c65c7ac153132fae8 Oct 30 23:41:36.797358 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:36.797305 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:41:36.942611 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:36.942574 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:41:37.170184 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:37.170152 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vl8t5" event={"ID":"be5540b5-7b10-45b2-b40c-723efd9f1b01","Type":"ContainerStarted","Data":"3dccbe10481f566703890f75518345e4ac98e9a3095411aa1d972ec0086483bc"} Oct 30 23:41:37.171647 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:37.171623 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-544b44f8d-5p6kz" event={"ID":"58f7cf25-f5d0-4481-a927-4a745b692f7a","Type":"ContainerStarted","Data":"b66ee2127fd8d0b08162e3f6ef34c3504266c597e197a2c00d3d6f32108a93d4"} Oct 30 23:41:37.171724 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:37.171668 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-544b44f8d-5p6kz" event={"ID":"58f7cf25-f5d0-4481-a927-4a745b692f7a","Type":"ContainerStarted","Data":"031752aff7fa5ac6deb70609bad56693e9a040632954724c65c7ac153132fae8"} Oct 30 23:41:37.172826 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:37.172805 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-rxpst" event={"ID":"b53ad6ed-90fd-4a0f-bf54-88d416ba19e9","Type":"ContainerStarted","Data":"a943303a74887a9d9b37f2691bb4e033e907feed48c6914d086c7bc3da0eb912"} Oct 30 23:41:37.174222 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:37.174203 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-dd954f877-x9h9g" event={"ID":"17461b46-7ebf-49b7-afaf-12e20e3994a3","Type":"ContainerStarted","Data":"4464107bc2478344a72f7f65a202b81bebc426570b6749c04f9b8d1986cb66c7"} Oct 30 23:41:37.174316 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:37.174253 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-dd954f877-x9h9g" event={"ID":"17461b46-7ebf-49b7-afaf-12e20e3994a3","Type":"ContainerStarted","Data":"7a6060d10f1c9eee9db00524cfd3379928a176a560d3cadb832a71aace5e50a4"} Oct 30 23:41:37.175292 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:37.175271 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-pt58s" event={"ID":"145dd323-9370-4b9d-a96e-8b93d0a47a7c","Type":"ContainerStarted","Data":"d96dd2dcea078cec34a0fc37b22864bd9cd0dd97208025dc5cfe742d9ee6d1f8"} Oct 30 23:41:37.196530 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:37.196453 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:41:37.443774 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:37.443740 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:41:37.591458 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:37.591377 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:41:37.998541 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:37.998505 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:41:38.179918 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:38.179891 4927 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 23:41:38.180397 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:38.179891 4927 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 23:41:38.397574 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:38.397488 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:41:38.593791 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:38.593763 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:41:38.795854 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:38.795811 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:41:39.189219 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:39.189185 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-pt58s" event={"ID":"145dd323-9370-4b9d-a96e-8b93d0a47a7c","Type":"ContainerStarted","Data":"0ec9bacb3032f45b40272c8fc7ae084f2782c6638bb38bba1b81b6bbeff3d047"} Oct 30 23:41:39.189630 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:39.189276 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-pt58s" event={"ID":"145dd323-9370-4b9d-a96e-8b93d0a47a7c","Type":"ContainerStarted","Data":"ebe542d25d34f801073ec0997cde576401f7bffb510a2ce8f9761184bdb7fe20"} Oct 30 23:41:39.190604 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:39.190579 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-rxpst" event={"ID":"b53ad6ed-90fd-4a0f-bf54-88d416ba19e9","Type":"ContainerStarted","Data":"1d02f0ef023ec7f391522b818f6af541a99d5cc03fcc2eeccac1a56b9be533e5"} Oct 30 23:41:39.194232 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:39.194203 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:41:39.997995 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:39.997959 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:41:40.079401 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:40.079376 4927 scope.go:122] "RemoveContainer" containerID="64081a16fba6223deec2f4d14ac5189bbc15017349d35d38b83aae316589b8e4" Oct 30 23:41:40.195387 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:40.195361 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vl8t5" event={"ID":"be5540b5-7b10-45b2-b40c-723efd9f1b01","Type":"ContainerStarted","Data":"225d071608b65d24acfe2150df6f8ea3ceceecb6dcffc141f90b8104cf9a304b"} Oct 30 23:41:40.195702 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:40.195412 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vl8t5" event={"ID":"be5540b5-7b10-45b2-b40c-723efd9f1b01","Type":"ContainerStarted","Data":"ed35143c89c849f64132cf15626f3af7e5fe60307f8d8d40c47847f297ef57ca"} Oct 30 23:41:40.196857 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:40.196840 4927 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-pqt2c_6382dfb5-a363-47a2-bc79-0b96f7bec0c3/console-operator/3.log" Oct 30 23:41:40.197207 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:40.197193 4927 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-pqt2c_6382dfb5-a363-47a2-bc79-0b96f7bec0c3/console-operator/2.log" Oct 30 23:41:40.197279 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:40.197220 4927 generic.go:356] "Generic (PLEG): container finished" podID="6382dfb5-a363-47a2-bc79-0b96f7bec0c3" containerID="34476b86c3dda2468569ec7952c120dcf223d91119e81cde8a6a6404e1fa9378" exitCode=255 Oct 30 23:41:40.197314 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:40.197295 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-pqt2c" event={"ID":"6382dfb5-a363-47a2-bc79-0b96f7bec0c3","Type":"ContainerDied","Data":"34476b86c3dda2468569ec7952c120dcf223d91119e81cde8a6a6404e1fa9378"} Oct 30 23:41:40.197381 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:40.197368 4927 scope.go:122] "RemoveContainer" containerID="64081a16fba6223deec2f4d14ac5189bbc15017349d35d38b83aae316589b8e4" Oct 30 23:41:40.197778 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:40.197764 4927 scope.go:122] "RemoveContainer" containerID="34476b86c3dda2468569ec7952c120dcf223d91119e81cde8a6a6404e1fa9378" Oct 30 23:41:40.197928 ip-10-0-141-180 kubenswrapper[4927]: E1030 23:41:40.197910 4927 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-pqt2c_openshift-console-operator(6382dfb5-a363-47a2-bc79-0b96f7bec0c3)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-pqt2c" podUID="6382dfb5-a363-47a2-bc79-0b96f7bec0c3" Oct 30 23:41:40.396119 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:40.396042 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:41:40.590391 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:40.590354 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:41:40.736012 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:40.735981 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:41:40.881682 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:40.881648 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:41:41.195475 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:41.195440 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:41:41.202450 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:41.202431 4927 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-pqt2c_6382dfb5-a363-47a2-bc79-0b96f7bec0c3/console-operator/3.log" Oct 30 23:41:41.996731 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:41.996696 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:41:42.395975 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:42.395897 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:41:42.596579 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:42.596542 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:41:42.795059 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:42.795024 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:41:42.993159 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:42.993126 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:41:43.218435 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:43.218404 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-dd954f877-x9h9g" Oct 30 23:41:44.106156 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:44.106112 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/29b8bd1a-beeb-495c-9f61-e3326866a1d7-metrics-tls\") pod \"dns-default-9l5zq\" (UID: \"29b8bd1a-beeb-495c-9f61-e3326866a1d7\") " pod="openshift-dns/dns-default-9l5zq" Oct 30 23:41:44.106663 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:44.106171 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4cf4abe6-da2b-41b6-9caa-9659d4f750fb-metrics-certs\") pod \"router-default-66d97c9c97-qqxj5\" (UID: \"4cf4abe6-da2b-41b6-9caa-9659d4f750fb\") " pod="openshift-ingress/router-default-66d97c9c97-qqxj5" Oct 30 23:41:44.106663 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:44.106210 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/03211c6a-9761-419c-839b-21ca413d5bb2-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-httg9\" (UID: \"03211c6a-9761-419c-839b-21ca413d5bb2\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-httg9" Oct 30 23:41:44.106663 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:44.106230 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4cf4abe6-da2b-41b6-9caa-9659d4f750fb-service-ca-bundle\") pod \"router-default-66d97c9c97-qqxj5\" (UID: \"4cf4abe6-da2b-41b6-9caa-9659d4f750fb\") " pod="openshift-ingress/router-default-66d97c9c97-qqxj5" Oct 30 23:41:44.106663 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:44.106250 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ec330d4f-2eea-476a-a7b8-60cd295fddd2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sqv69\" (UID: \"ec330d4f-2eea-476a-a7b8-60cd295fddd2\") " pod="openshift-insights/insights-runtime-extractor-sqv69" Oct 30 23:41:44.107003 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:44.106977 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4cf4abe6-da2b-41b6-9caa-9659d4f750fb-service-ca-bundle\") pod \"router-default-66d97c9c97-qqxj5\" (UID: \"4cf4abe6-da2b-41b6-9caa-9659d4f750fb\") " pod="openshift-ingress/router-default-66d97c9c97-qqxj5" Oct 30 23:41:44.108628 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:44.108600 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ec330d4f-2eea-476a-a7b8-60cd295fddd2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sqv69\" (UID: \"ec330d4f-2eea-476a-a7b8-60cd295fddd2\") " pod="openshift-insights/insights-runtime-extractor-sqv69" Oct 30 23:41:44.108747 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:44.108652 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/03211c6a-9761-419c-839b-21ca413d5bb2-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-httg9\" (UID: \"03211c6a-9761-419c-839b-21ca413d5bb2\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-httg9" Oct 30 23:41:44.109025 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:44.109001 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/29b8bd1a-beeb-495c-9f61-e3326866a1d7-metrics-tls\") pod \"dns-default-9l5zq\" (UID: \"29b8bd1a-beeb-495c-9f61-e3326866a1d7\") " pod="openshift-dns/dns-default-9l5zq" Oct 30 23:41:44.109084 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:44.109075 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4cf4abe6-da2b-41b6-9caa-9659d4f750fb-metrics-certs\") pod \"router-default-66d97c9c97-qqxj5\" (UID: \"4cf4abe6-da2b-41b6-9caa-9659d4f750fb\") " pod="openshift-ingress/router-default-66d97c9c97-qqxj5" Oct 30 23:41:44.148876 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:44.148853 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-544b44f8d-5p6kz" Oct 30 23:41:44.385463 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:44.385387 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-66d97c9c97-qqxj5" Oct 30 23:41:44.388488 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:44.388469 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-httg9" Oct 30 23:41:44.399621 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:44.399498 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-9l5zq" Oct 30 23:41:44.400593 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:44.400571 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-sqv69" Oct 30 23:41:44.690064 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:44.690037 4927 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-pqt2c" Oct 30 23:41:44.690396 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:44.690379 4927 scope.go:122] "RemoveContainer" containerID="34476b86c3dda2468569ec7952c120dcf223d91119e81cde8a6a6404e1fa9378" Oct 30 23:41:44.690579 ip-10-0-141-180 kubenswrapper[4927]: E1030 23:41:44.690561 4927 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-pqt2c_openshift-console-operator(6382dfb5-a363-47a2-bc79-0b96f7bec0c3)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-pqt2c" podUID="6382dfb5-a363-47a2-bc79-0b96f7bec0c3" Oct 30 23:41:44.795900 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:44.795869 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-httg9"] Oct 30 23:41:44.810535 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:44.810495 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03211c6a_9761_419c_839b_21ca413d5bb2.slice/crio-b15415a19503c400acfceeaaaf9337438faa54fd1a89efe04da722d04f9ab96b WatchSource:0}: Error finding container b15415a19503c400acfceeaaaf9337438faa54fd1a89efe04da722d04f9ab96b: Status 404 returned error can't find the container with id b15415a19503c400acfceeaaaf9337438faa54fd1a89efe04da722d04f9ab96b Oct 30 23:41:44.829010 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:44.828987 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4cf4abe6_da2b_41b6_9caa_9659d4f750fb.slice/crio-ef89e6c4d33f273fd95c9dbcda18e516540425e626a13c587173a9d68c7b6ea2 WatchSource:0}: Error finding container ef89e6c4d33f273fd95c9dbcda18e516540425e626a13c587173a9d68c7b6ea2: Status 404 returned error can't find the container with id ef89e6c4d33f273fd95c9dbcda18e516540425e626a13c587173a9d68c7b6ea2 Oct 30 23:41:44.847617 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:44.847596 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-66d97c9c97-qqxj5"] Oct 30 23:41:44.861451 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:44.861427 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-9l5zq"] Oct 30 23:41:44.862319 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:44.862298 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29b8bd1a_beeb_495c_9f61_e3326866a1d7.slice/crio-c9b7f7ae6ab3c8462a035137fc9386615d41b5e415d1f7ccc112fba5a22fcf9b WatchSource:0}: Error finding container c9b7f7ae6ab3c8462a035137fc9386615d41b5e415d1f7ccc112fba5a22fcf9b: Status 404 returned error can't find the container with id c9b7f7ae6ab3c8462a035137fc9386615d41b5e415d1f7ccc112fba5a22fcf9b Oct 30 23:41:44.872061 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:44.872042 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-sqv69"] Oct 30 23:41:44.875271 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:41:44.875248 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec330d4f_2eea_476a_a7b8_60cd295fddd2.slice/crio-c961cbdda860205632d9c4fc97e46e3efb8ca28f701d290b06a922acf2666ac1 WatchSource:0}: Error finding container c961cbdda860205632d9c4fc97e46e3efb8ca28f701d290b06a922acf2666ac1: Status 404 returned error can't find the container with id c961cbdda860205632d9c4fc97e46e3efb8ca28f701d290b06a922acf2666ac1 Oct 30 23:41:45.127199 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:45.127122 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-dd954f877-x9h9g"] Oct 30 23:41:45.181212 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:45.181182 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:41:45.220733 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:45.220694 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-9l5zq" event={"ID":"29b8bd1a-beeb-495c-9f61-e3326866a1d7","Type":"ContainerStarted","Data":"c9b7f7ae6ab3c8462a035137fc9386615d41b5e415d1f7ccc112fba5a22fcf9b"} Oct 30 23:41:45.224902 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:45.224867 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-sqv69" event={"ID":"ec330d4f-2eea-476a-a7b8-60cd295fddd2","Type":"ContainerStarted","Data":"3726bc8b56ac700208c5e42950341a94cd2e43dcdc4b2d3dea80155c39e6dcda"} Oct 30 23:41:45.225027 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:45.224933 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-sqv69" event={"ID":"ec330d4f-2eea-476a-a7b8-60cd295fddd2","Type":"ContainerStarted","Data":"c961cbdda860205632d9c4fc97e46e3efb8ca28f701d290b06a922acf2666ac1"} Oct 30 23:41:45.226454 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:45.226430 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-httg9" event={"ID":"03211c6a-9761-419c-839b-21ca413d5bb2","Type":"ContainerStarted","Data":"b15415a19503c400acfceeaaaf9337438faa54fd1a89efe04da722d04f9ab96b"} Oct 30 23:41:45.228060 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:45.228041 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-66d97c9c97-qqxj5" event={"ID":"4cf4abe6-da2b-41b6-9caa-9659d4f750fb","Type":"ContainerStarted","Data":"2c446e0b644b15e5b1b274d442c5fc17e7251152605535d186ef169e92be95ce"} Oct 30 23:41:45.228136 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:45.228073 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-66d97c9c97-qqxj5" event={"ID":"4cf4abe6-da2b-41b6-9caa-9659d4f750fb","Type":"ContainerStarted","Data":"ef89e6c4d33f273fd95c9dbcda18e516540425e626a13c587173a9d68c7b6ea2"} Oct 30 23:41:45.386067 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:45.385992 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-66d97c9c97-qqxj5" Oct 30 23:41:45.388097 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:45.388071 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-66d97c9c97-qqxj5" Oct 30 23:41:45.478835 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:45.478795 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:41:45.876030 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:45.875922 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:41:46.024180 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:46.024152 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:41:46.233556 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:46.233435 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-66d97c9c97-qqxj5" Oct 30 23:41:46.235459 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:46.235317 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-66d97c9c97-qqxj5" Oct 30 23:41:47.238218 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:47.238181 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-sqv69" event={"ID":"ec330d4f-2eea-476a-a7b8-60cd295fddd2","Type":"ContainerStarted","Data":"9fadb33e0251c3bcc70ad642c4aa640c96c7e4c0212c1a0764263d62c0808a77"} Oct 30 23:41:48.243886 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:48.243846 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-httg9" event={"ID":"03211c6a-9761-419c-839b-21ca413d5bb2","Type":"ContainerStarted","Data":"47c0d1f449f86c6ba61eab9c0c2155dd574a558bb4383327ee3b49d6dcb00613"} Oct 30 23:41:48.245903 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:48.245862 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-9l5zq" event={"ID":"29b8bd1a-beeb-495c-9f61-e3326866a1d7","Type":"ContainerStarted","Data":"9222e422beb4102d960b77c56ecda38b1b85ce4009d446326a969968bcf807ac"} Oct 30 23:41:48.246019 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:48.245929 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-9l5zq" event={"ID":"29b8bd1a-beeb-495c-9f61-e3326866a1d7","Type":"ContainerStarted","Data":"f4dfc261590928611455bdcef9c1b1a4d015b50acf7f1b131a12b7dea324f7d1"} Oct 30 23:41:48.537145 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:48.537044 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:41:48.728026 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:48.728001 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:41:48.878168 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:48.878092 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:41:49.252057 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:49.252020 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-sqv69" event={"ID":"ec330d4f-2eea-476a-a7b8-60cd295fddd2","Type":"ContainerStarted","Data":"157922df608894dc52febadb57403125b6cad7c25e6df8bfdab6a6d9f5d7fc46"} Oct 30 23:41:49.252453 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:49.252301 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-9l5zq" Oct 30 23:41:49.345257 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:49.345230 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-pqt2c" Oct 30 23:41:49.345626 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:49.345612 4927 scope.go:122] "RemoveContainer" containerID="34476b86c3dda2468569ec7952c120dcf223d91119e81cde8a6a6404e1fa9378" Oct 30 23:41:49.345815 ip-10-0-141-180 kubenswrapper[4927]: E1030 23:41:49.345797 4927 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-pqt2c_openshift-console-operator(6382dfb5-a363-47a2-bc79-0b96f7bec0c3)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-pqt2c" podUID="6382dfb5-a363-47a2-bc79-0b96f7bec0c3" Oct 30 23:41:49.625008 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:49.624956 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:41:49.677309 ip-10-0-141-180 kubenswrapper[4927]: E1030 23:41:49.677274 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sqv69" containerName="extractor" Oct 30 23:41:49.970494 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:49.970462 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:41:52.482192 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:52.482155 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:41:52.628893 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:52.628851 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:41:53.907347 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:53.907292 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:41:54.092840 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:54.092804 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:41:54.153155 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:54.153109 4927 patch_prober.go:29] interesting pod/image-registry-544b44f8d-5p6kz container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Oct 30 23:41:54.153309 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:54.153177 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-544b44f8d-5p6kz" podUID="58f7cf25-f5d0-4481-a927-4a745b692f7a" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 23:41:54.244804 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:54.244717 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:41:54.397608 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:54.397565 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:41:54.589390 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:54.589301 4927 patch_prober.go:29] interesting pod/image-registry-544b44f8d-5p6kz container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Oct 30 23:41:54.589390 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:54.589368 4927 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-544b44f8d-5p6kz" podUID="58f7cf25-f5d0-4481-a927-4a745b692f7a" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 23:41:55.133598 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:55.133566 4927 patch_prober.go:29] interesting pod/image-registry-dd954f877-x9h9g container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Oct 30 23:41:55.133990 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:55.133613 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-dd954f877-x9h9g" podUID="17461b46-7ebf-49b7-afaf-12e20e3994a3" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 23:41:55.585805 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:55.585764 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:41:56.233123 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:56.233089 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-9s2fx" Oct 30 23:41:57.841632 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:57.841603 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-26v6x" Oct 30 23:41:59.258637 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:59.258602 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-9l5zq" Oct 30 23:41:59.839820 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:41:59.839783 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:42:00.169457 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:42:00.169418 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:42:00.402288 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:42:00.402250 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:42:00.563523 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:42:00.563437 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:42:00.780024 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:42:00.779988 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:42:00.920752 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:42:00.920721 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-kgdzs" Oct 30 23:42:00.926122 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:42:00.926094 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:42:01.036306 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:42:01.036271 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-bcb6l"] Oct 30 23:42:01.075024 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:42:01.074991 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-bcb6l" Oct 30 23:42:01.086915 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:42:01.086886 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:42:01.128857 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:42:01.128823 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 30 23:42:01.128857 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:42:01.128826 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 30 23:42:01.129064 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:42:01.128823 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 30 23:42:01.129064 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:42:01.128839 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-4224x\"" Oct 30 23:42:01.129433 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:42:01.129368 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 30 23:42:01.239478 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:42:01.239397 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:42:01.246392 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:42:01.246361 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/81575ea6-1389-4b06-98b9-d10271205350-sys\") pod \"node-exporter-bcb6l\" (UID: \"81575ea6-1389-4b06-98b9-d10271205350\") " pod="openshift-monitoring/node-exporter-bcb6l" Oct 30 23:42:01.246517 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:42:01.246406 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/81575ea6-1389-4b06-98b9-d10271205350-node-exporter-accelerators-collector-config\") pod \"node-exporter-bcb6l\" (UID: \"81575ea6-1389-4b06-98b9-d10271205350\") " pod="openshift-monitoring/node-exporter-bcb6l" Oct 30 23:42:01.246580 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:42:01.246543 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/81575ea6-1389-4b06-98b9-d10271205350-node-exporter-textfile\") pod \"node-exporter-bcb6l\" (UID: \"81575ea6-1389-4b06-98b9-d10271205350\") " pod="openshift-monitoring/node-exporter-bcb6l" Oct 30 23:42:01.246648 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:42:01.246597 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/81575ea6-1389-4b06-98b9-d10271205350-root\") pod \"node-exporter-bcb6l\" (UID: \"81575ea6-1389-4b06-98b9-d10271205350\") " pod="openshift-monitoring/node-exporter-bcb6l" Oct 30 23:42:01.246648 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:42:01.246627 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/81575ea6-1389-4b06-98b9-d10271205350-metrics-client-ca\") pod \"node-exporter-bcb6l\" (UID: \"81575ea6-1389-4b06-98b9-d10271205350\") " pod="openshift-monitoring/node-exporter-bcb6l" Oct 30 23:42:01.246737 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:42:01.246717 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/81575ea6-1389-4b06-98b9-d10271205350-node-exporter-tls\") pod \"node-exporter-bcb6l\" (UID: \"81575ea6-1389-4b06-98b9-d10271205350\") " pod="openshift-monitoring/node-exporter-bcb6l" Oct 30 23:42:01.246778 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:42:01.246746 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/81575ea6-1389-4b06-98b9-d10271205350-node-exporter-wtmp\") pod \"node-exporter-bcb6l\" (UID: \"81575ea6-1389-4b06-98b9-d10271205350\") " pod="openshift-monitoring/node-exporter-bcb6l" Oct 30 23:42:01.246833 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:42:01.246795 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fff6x\" (UniqueName: \"kubernetes.io/projected/81575ea6-1389-4b06-98b9-d10271205350-kube-api-access-fff6x\") pod \"node-exporter-bcb6l\" (UID: \"81575ea6-1389-4b06-98b9-d10271205350\") " pod="openshift-monitoring/node-exporter-bcb6l" Oct 30 23:42:01.246883 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:42:01.246866 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/81575ea6-1389-4b06-98b9-d10271205350-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-bcb6l\" (UID: \"81575ea6-1389-4b06-98b9-d10271205350\") " pod="openshift-monitoring/node-exporter-bcb6l" Oct 30 23:42:01.347373 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:42:01.347314 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/81575ea6-1389-4b06-98b9-d10271205350-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-bcb6l\" (UID: \"81575ea6-1389-4b06-98b9-d10271205350\") " pod="openshift-monitoring/node-exporter-bcb6l" Oct 30 23:42:01.347554 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:42:01.347464 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/81575ea6-1389-4b06-98b9-d10271205350-sys\") pod \"node-exporter-bcb6l\" (UID: \"81575ea6-1389-4b06-98b9-d10271205350\") " pod="openshift-monitoring/node-exporter-bcb6l" Oct 30 23:42:01.347554 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:42:01.347498 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/81575ea6-1389-4b06-98b9-d10271205350-node-exporter-accelerators-collector-config\") pod \"node-exporter-bcb6l\" (UID: \"81575ea6-1389-4b06-98b9-d10271205350\") " pod="openshift-monitoring/node-exporter-bcb6l" Oct 30 23:42:01.347673 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:42:01.347574 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/81575ea6-1389-4b06-98b9-d10271205350-sys\") pod \"node-exporter-bcb6l\" (UID: \"81575ea6-1389-4b06-98b9-d10271205350\") " pod="openshift-monitoring/node-exporter-bcb6l" Oct 30 23:42:01.347673 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:42:01.347632 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/81575ea6-1389-4b06-98b9-d10271205350-node-exporter-textfile\") pod \"node-exporter-bcb6l\" (UID: \"81575ea6-1389-4b06-98b9-d10271205350\") " pod="openshift-monitoring/node-exporter-bcb6l" Oct 30 23:42:01.347673 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:42:01.347669 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/81575ea6-1389-4b06-98b9-d10271205350-root\") pod \"node-exporter-bcb6l\" (UID: \"81575ea6-1389-4b06-98b9-d10271205350\") " pod="openshift-monitoring/node-exporter-bcb6l" Oct 30 23:42:01.347832 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:42:01.347694 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/81575ea6-1389-4b06-98b9-d10271205350-metrics-client-ca\") pod \"node-exporter-bcb6l\" (UID: \"81575ea6-1389-4b06-98b9-d10271205350\") " pod="openshift-monitoring/node-exporter-bcb6l" Oct 30 23:42:01.347832 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:42:01.347728 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/81575ea6-1389-4b06-98b9-d10271205350-node-exporter-tls\") pod \"node-exporter-bcb6l\" (UID: \"81575ea6-1389-4b06-98b9-d10271205350\") " pod="openshift-monitoring/node-exporter-bcb6l" Oct 30 23:42:01.347832 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:42:01.347741 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/81575ea6-1389-4b06-98b9-d10271205350-root\") pod \"node-exporter-bcb6l\" (UID: \"81575ea6-1389-4b06-98b9-d10271205350\") " pod="openshift-monitoring/node-exporter-bcb6l" Oct 30 23:42:01.347832 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:42:01.347758 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/81575ea6-1389-4b06-98b9-d10271205350-node-exporter-wtmp\") pod \"node-exporter-bcb6l\" (UID: \"81575ea6-1389-4b06-98b9-d10271205350\") " pod="openshift-monitoring/node-exporter-bcb6l" Oct 30 23:42:01.347832 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:42:01.347804 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fff6x\" (UniqueName: \"kubernetes.io/projected/81575ea6-1389-4b06-98b9-d10271205350-kube-api-access-fff6x\") pod \"node-exporter-bcb6l\" (UID: \"81575ea6-1389-4b06-98b9-d10271205350\") " pod="openshift-monitoring/node-exporter-bcb6l" Oct 30 23:42:01.348127 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:42:01.347974 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/81575ea6-1389-4b06-98b9-d10271205350-node-exporter-textfile\") pod \"node-exporter-bcb6l\" (UID: \"81575ea6-1389-4b06-98b9-d10271205350\") " pod="openshift-monitoring/node-exporter-bcb6l" Oct 30 23:42:01.348127 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:42:01.347996 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/81575ea6-1389-4b06-98b9-d10271205350-node-exporter-wtmp\") pod \"node-exporter-bcb6l\" (UID: \"81575ea6-1389-4b06-98b9-d10271205350\") " pod="openshift-monitoring/node-exporter-bcb6l" Oct 30 23:42:01.348238 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:42:01.348189 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/81575ea6-1389-4b06-98b9-d10271205350-node-exporter-accelerators-collector-config\") pod \"node-exporter-bcb6l\" (UID: \"81575ea6-1389-4b06-98b9-d10271205350\") " pod="openshift-monitoring/node-exporter-bcb6l" Oct 30 23:42:01.348277 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:42:01.348262 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/81575ea6-1389-4b06-98b9-d10271205350-metrics-client-ca\") pod \"node-exporter-bcb6l\" (UID: \"81575ea6-1389-4b06-98b9-d10271205350\") " pod="openshift-monitoring/node-exporter-bcb6l" Oct 30 23:42:01.349710 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:42:01.349689 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/81575ea6-1389-4b06-98b9-d10271205350-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-bcb6l\" (UID: \"81575ea6-1389-4b06-98b9-d10271205350\") " pod="openshift-monitoring/node-exporter-bcb6l" Oct 30 23:42:01.349945 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:42:01.349928 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/81575ea6-1389-4b06-98b9-d10271205350-node-exporter-tls\") pod \"node-exporter-bcb6l\" (UID: \"81575ea6-1389-4b06-98b9-d10271205350\") " pod="openshift-monitoring/node-exporter-bcb6l" Oct 30 23:42:01.411348 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:42:01.411308 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fff6x\" (UniqueName: \"kubernetes.io/projected/81575ea6-1389-4b06-98b9-d10271205350-kube-api-access-fff6x\") pod \"node-exporter-bcb6l\" (UID: \"81575ea6-1389-4b06-98b9-d10271205350\") " pod="openshift-monitoring/node-exporter-bcb6l" Oct 30 23:42:01.427251 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:42:01.427218 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-bcb6l" Oct 30 23:42:01.434174 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:42:01.434146 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:42:01.467074 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:42:01.467046 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81575ea6_1389_4b06_98b9_d10271205350.slice/crio-0811d798f55a4bd117ece017482e59f5cff7a1506ea0164425f4604b91cf199f WatchSource:0}: Error finding container 0811d798f55a4bd117ece017482e59f5cff7a1506ea0164425f4604b91cf199f: Status 404 returned error can't find the container with id 0811d798f55a4bd117ece017482e59f5cff7a1506ea0164425f4604b91cf199f Oct 30 23:42:01.835940 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:42:01.835901 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:42:01.990234 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:42:01.990199 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:42:02.232988 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:42:02.232949 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:42:02.305417 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:42:02.305381 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-bcb6l" event={"ID":"81575ea6-1389-4b06-98b9-d10271205350","Type":"ContainerStarted","Data":"0811d798f55a4bd117ece017482e59f5cff7a1506ea0164425f4604b91cf199f"} Oct 30 23:42:02.401214 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:42:02.401177 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:42:03.100787 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:42:03.100753 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:42:03.379741 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:42:03.379655 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:42:03.526412 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:42:03.526378 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:42:03.676854 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:42:03.676825 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:42:04.079405 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:42:04.079374 4927 scope.go:122] "RemoveContainer" containerID="34476b86c3dda2468569ec7952c120dcf223d91119e81cde8a6a6404e1fa9378" Oct 30 23:42:04.152836 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:42:04.152814 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-544b44f8d-5p6kz" Oct 30 23:42:04.315445 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:42:04.315419 4927 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-pqt2c_6382dfb5-a363-47a2-bc79-0b96f7bec0c3/console-operator/3.log" Oct 30 23:42:04.315626 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:42:04.315507 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-pqt2c" event={"ID":"6382dfb5-a363-47a2-bc79-0b96f7bec0c3","Type":"ContainerStarted","Data":"d2caec88d5e02bede60e01f4f63d43e85f0ec75250f4a9d59fc942a274604790"} Oct 30 23:42:04.315794 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:42:04.315773 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-pqt2c" Oct 30 23:42:04.317067 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:42:04.317040 4927 patch_prober.go:29] interesting pod/console-operator-7cd56b8dbc-pqt2c container/console-operator namespace/openshift-console-operator: Readiness probe status=failure output="Get \"https://10.132.0.9:8443/readyz\": dial tcp 10.132.0.9:8443: connect: connection refused" start-of-body= Oct 30 23:42:04.317209 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:42:04.317089 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-console-operator/console-operator-7cd56b8dbc-pqt2c" podUID="6382dfb5-a363-47a2-bc79-0b96f7bec0c3" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.9:8443/readyz\": dial tcp 10.132.0.9:8443: connect: connection refused" Oct 30 23:42:04.317402 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:42:04.317379 4927 generic.go:356] "Generic (PLEG): container finished" podID="81575ea6-1389-4b06-98b9-d10271205350" containerID="cb58068c133792f31c38314ad78552638fef0ff0ec3f9935e0d5c831cd8ea628" exitCode=0 Oct 30 23:42:04.317482 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:42:04.317449 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-bcb6l" event={"ID":"81575ea6-1389-4b06-98b9-d10271205350","Type":"ContainerDied","Data":"cb58068c133792f31c38314ad78552638fef0ff0ec3f9935e0d5c831cd8ea628"} Oct 30 23:42:04.788836 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:42:04.788802 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:42:04.990035 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:42:04.989998 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:42:05.132152 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:42:05.132077 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-dd954f877-x9h9g" Oct 30 23:42:05.138630 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:42:05.138599 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:42:05.323649 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:42:05.323609 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-bcb6l" event={"ID":"81575ea6-1389-4b06-98b9-d10271205350","Type":"ContainerStarted","Data":"caec047ce5a8a516dd477edad2e109c6aae9cdd6508d2b3feaa5b3a25980f0a6"} Oct 30 23:42:05.324134 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:42:05.323666 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-bcb6l" event={"ID":"81575ea6-1389-4b06-98b9-d10271205350","Type":"ContainerStarted","Data":"cba243f08d8d3144291dd11874d3872e601678cf8847dbc06d42ed5b0ec55bb2"} Oct 30 23:42:05.327521 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:42:05.327493 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-pqt2c" Oct 30 23:42:05.341222 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:42:05.341192 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:42:05.445932 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:42:05.445871 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-bcb6l" podStartSLOduration=3.239751353 podStartE2EDuration="5.445855321s" totalImagesPullingTime="2.123063338s" totalInitContainerRuntime="83.04063ms" isStatefulPod=true podCreationTimestamp="2026-10-30 23:42:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:42:05.444917612 +0000 UTC m=+38.481248434" watchObservedRunningTime="2026-10-30 23:42:05.445855321 +0000 UTC m=+38.482186137" Oct 30 23:42:05.491138 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:42:05.491104 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:42:05.653837 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:42:05.653792 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:42:05.809399 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:42:05.809307 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:42:05.996025 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:42:05.995988 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:42:06.149072 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:42:06.148990 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:42:06.590899 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:42:06.590865 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:42:07.443951 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:42:07.443914 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:42:07.596656 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:42:07.596615 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:42:07.874454 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:42:07.874373 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:42:08.131754 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:42:08.131679 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-544b44f8d-5p6kz"] Oct 30 23:42:08.216810 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:42:08.216781 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:42:08.365796 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:42:08.365753 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:42:08.787481 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:42:08.787440 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:42:08.939578 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:42:08.939538 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:42:10.006993 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:42:10.006955 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:42:10.146478 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:42:10.146419 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-dd954f877-x9h9g" podUID="17461b46-7ebf-49b7-afaf-12e20e3994a3" containerName="registry" containerID="cri-o://4464107bc2478344a72f7f65a202b81bebc426570b6749c04f9b8d1986cb66c7" gracePeriod=30 Oct 30 23:42:10.345547 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:42:10.345518 4927 generic.go:356] "Generic (PLEG): container finished" podID="17461b46-7ebf-49b7-afaf-12e20e3994a3" containerID="4464107bc2478344a72f7f65a202b81bebc426570b6749c04f9b8d1986cb66c7" exitCode=0 Oct 30 23:42:10.345696 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:42:10.345591 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-dd954f877-x9h9g" event={"ID":"17461b46-7ebf-49b7-afaf-12e20e3994a3","Type":"ContainerDied","Data":"4464107bc2478344a72f7f65a202b81bebc426570b6749c04f9b8d1986cb66c7"} Oct 30 23:42:10.386869 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:42:10.386842 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-dd954f877-x9h9g" Oct 30 23:42:10.523862 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:42:10.523780 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/17461b46-7ebf-49b7-afaf-12e20e3994a3-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/17461b46-7ebf-49b7-afaf-12e20e3994a3-trusted-ca\") pod \"17461b46-7ebf-49b7-afaf-12e20e3994a3\" (UID: \"17461b46-7ebf-49b7-afaf-12e20e3994a3\") " Oct 30 23:42:10.523862 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:42:10.523855 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/17461b46-7ebf-49b7-afaf-12e20e3994a3-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/17461b46-7ebf-49b7-afaf-12e20e3994a3-image-registry-private-configuration\") pod \"17461b46-7ebf-49b7-afaf-12e20e3994a3\" (UID: \"17461b46-7ebf-49b7-afaf-12e20e3994a3\") " Oct 30 23:42:10.524095 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:42:10.523883 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17461b46-7ebf-49b7-afaf-12e20e3994a3-registry-tls\" (UniqueName: \"kubernetes.io/projected/17461b46-7ebf-49b7-afaf-12e20e3994a3-registry-tls\") pod \"17461b46-7ebf-49b7-afaf-12e20e3994a3\" (UID: \"17461b46-7ebf-49b7-afaf-12e20e3994a3\") " Oct 30 23:42:10.524095 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:42:10.523970 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/17461b46-7ebf-49b7-afaf-12e20e3994a3-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/17461b46-7ebf-49b7-afaf-12e20e3994a3-installation-pull-secrets\") pod \"17461b46-7ebf-49b7-afaf-12e20e3994a3\" (UID: \"17461b46-7ebf-49b7-afaf-12e20e3994a3\") " Oct 30 23:42:10.524095 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:42:10.524037 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17461b46-7ebf-49b7-afaf-12e20e3994a3-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/17461b46-7ebf-49b7-afaf-12e20e3994a3-bound-sa-token\") pod \"17461b46-7ebf-49b7-afaf-12e20e3994a3\" (UID: \"17461b46-7ebf-49b7-afaf-12e20e3994a3\") " Oct 30 23:42:10.524095 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:42:10.524079 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/17461b46-7ebf-49b7-afaf-12e20e3994a3-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/17461b46-7ebf-49b7-afaf-12e20e3994a3-registry-certificates\") pod \"17461b46-7ebf-49b7-afaf-12e20e3994a3\" (UID: \"17461b46-7ebf-49b7-afaf-12e20e3994a3\") " Oct 30 23:42:10.524303 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:42:10.524114 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17461b46-7ebf-49b7-afaf-12e20e3994a3-kube-api-access-xkthh\" (UniqueName: \"kubernetes.io/projected/17461b46-7ebf-49b7-afaf-12e20e3994a3-kube-api-access-xkthh\") pod \"17461b46-7ebf-49b7-afaf-12e20e3994a3\" (UID: \"17461b46-7ebf-49b7-afaf-12e20e3994a3\") " Oct 30 23:42:10.524303 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:42:10.524147 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17461b46-7ebf-49b7-afaf-12e20e3994a3-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/17461b46-7ebf-49b7-afaf-12e20e3994a3-ca-trust-extracted\") pod \"17461b46-7ebf-49b7-afaf-12e20e3994a3\" (UID: \"17461b46-7ebf-49b7-afaf-12e20e3994a3\") " Oct 30 23:42:10.524430 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:42:10.524269 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17461b46-7ebf-49b7-afaf-12e20e3994a3-trusted-ca" pod "17461b46-7ebf-49b7-afaf-12e20e3994a3" (UID: "17461b46-7ebf-49b7-afaf-12e20e3994a3"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:42:10.524494 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:42:10.524450 4927 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/17461b46-7ebf-49b7-afaf-12e20e3994a3-trusted-ca\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 23:42:10.524857 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:42:10.524811 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17461b46-7ebf-49b7-afaf-12e20e3994a3-registry-certificates" pod "17461b46-7ebf-49b7-afaf-12e20e3994a3" (UID: "17461b46-7ebf-49b7-afaf-12e20e3994a3"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:42:10.526564 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:42:10.526519 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17461b46-7ebf-49b7-afaf-12e20e3994a3-registry-tls" pod "17461b46-7ebf-49b7-afaf-12e20e3994a3" (UID: "17461b46-7ebf-49b7-afaf-12e20e3994a3"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:42:10.526678 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:42:10.526567 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17461b46-7ebf-49b7-afaf-12e20e3994a3-image-registry-private-configuration" pod "17461b46-7ebf-49b7-afaf-12e20e3994a3" (UID: "17461b46-7ebf-49b7-afaf-12e20e3994a3"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 23:42:10.526741 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:42:10.526681 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17461b46-7ebf-49b7-afaf-12e20e3994a3-installation-pull-secrets" pod "17461b46-7ebf-49b7-afaf-12e20e3994a3" (UID: "17461b46-7ebf-49b7-afaf-12e20e3994a3"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 23:42:10.526796 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:42:10.526729 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17461b46-7ebf-49b7-afaf-12e20e3994a3-bound-sa-token" pod "17461b46-7ebf-49b7-afaf-12e20e3994a3" (UID: "17461b46-7ebf-49b7-afaf-12e20e3994a3"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:42:10.526970 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:42:10.526950 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17461b46-7ebf-49b7-afaf-12e20e3994a3-kube-api-access-xkthh" pod "17461b46-7ebf-49b7-afaf-12e20e3994a3" (UID: "17461b46-7ebf-49b7-afaf-12e20e3994a3"). InnerVolumeSpecName "kube-api-access-xkthh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:42:10.531545 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:42:10.531524 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17461b46-7ebf-49b7-afaf-12e20e3994a3-ca-trust-extracted" pod "17461b46-7ebf-49b7-afaf-12e20e3994a3" (UID: "17461b46-7ebf-49b7-afaf-12e20e3994a3"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:42:10.625582 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:42:10.625539 4927 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/17461b46-7ebf-49b7-afaf-12e20e3994a3-image-registry-private-configuration\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 23:42:10.625582 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:42:10.625577 4927 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/17461b46-7ebf-49b7-afaf-12e20e3994a3-registry-tls\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 23:42:10.625582 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:42:10.625588 4927 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/17461b46-7ebf-49b7-afaf-12e20e3994a3-installation-pull-secrets\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 23:42:10.625582 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:42:10.625599 4927 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/17461b46-7ebf-49b7-afaf-12e20e3994a3-bound-sa-token\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 23:42:10.625844 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:42:10.625608 4927 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/17461b46-7ebf-49b7-afaf-12e20e3994a3-registry-certificates\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 23:42:10.625844 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:42:10.625618 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xkthh\" (UniqueName: \"kubernetes.io/projected/17461b46-7ebf-49b7-afaf-12e20e3994a3-kube-api-access-xkthh\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 23:42:10.625844 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:42:10.625627 4927 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/17461b46-7ebf-49b7-afaf-12e20e3994a3-ca-trust-extracted\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 23:42:11.208120 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:42:11.208068 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:42:11.350811 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:42:11.350745 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-dd954f877-x9h9g" Oct 30 23:42:11.350950 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:42:11.350737 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-dd954f877-x9h9g" event={"ID":"17461b46-7ebf-49b7-afaf-12e20e3994a3","Type":"ContainerDied","Data":"7a6060d10f1c9eee9db00524cfd3379928a176a560d3cadb832a71aace5e50a4"} Oct 30 23:42:11.350950 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:42:11.350890 4927 scope.go:122] "RemoveContainer" containerID="4464107bc2478344a72f7f65a202b81bebc426570b6749c04f9b8d1986cb66c7" Oct 30 23:42:11.369886 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:42:11.369843 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:42:11.585105 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:42:11.585074 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-dd954f877-x9h9g"] Oct 30 23:42:11.636528 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:42:11.636463 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-dd954f877-x9h9g"] Oct 30 23:42:13.083181 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:42:13.083146 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="17461b46-7ebf-49b7-afaf-12e20e3994a3" path="/var/lib/kubelet/pods/17461b46-7ebf-49b7-afaf-12e20e3994a3/volumes" Oct 30 23:42:16.331236 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:42:16.331195 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:42:16.494674 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:42:16.494629 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:42:17.136951 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:42:17.136917 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:42:17.327227 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:42:17.327192 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:42:17.577017 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:42:17.576981 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:42:19.677382 ip-10-0-141-180 kubenswrapper[4927]: E1030 23:42:19.677345 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sqv69" containerName="extractor" Oct 30 23:42:22.250121 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:42:22.250091 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:42:22.405690 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:42:22.405654 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:42:22.551986 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:42:22.551899 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:42:23.278660 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:42:23.278627 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:42:23.436391 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:42:23.436356 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:42:28.905121 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:42:28.905087 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:42:33.152539 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:42:33.152476 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-544b44f8d-5p6kz" podUID="58f7cf25-f5d0-4481-a927-4a745b692f7a" containerName="registry" containerID="cri-o://b66ee2127fd8d0b08162e3f6ef34c3504266c597e197a2c00d3d6f32108a93d4" gracePeriod=30 Oct 30 23:42:33.404243 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:42:33.404193 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-544b44f8d-5p6kz" Oct 30 23:42:33.435574 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:42:33.435547 4927 generic.go:356] "Generic (PLEG): container finished" podID="10774606-8e0c-460b-bd5f-4cb6d121b30b" containerID="a437bbfb58182b8229373016d264976a03b2f98ed1529a7e12eefc31a7248f16" exitCode=0 Oct 30 23:42:33.435715 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:42:33.435620 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-9mqqt" event={"ID":"10774606-8e0c-460b-bd5f-4cb6d121b30b","Type":"ContainerDied","Data":"a437bbfb58182b8229373016d264976a03b2f98ed1529a7e12eefc31a7248f16"} Oct 30 23:42:33.435958 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:42:33.435941 4927 scope.go:122] "RemoveContainer" containerID="a437bbfb58182b8229373016d264976a03b2f98ed1529a7e12eefc31a7248f16" Oct 30 23:42:33.436979 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:42:33.436956 4927 generic.go:356] "Generic (PLEG): container finished" podID="58f7cf25-f5d0-4481-a927-4a745b692f7a" containerID="b66ee2127fd8d0b08162e3f6ef34c3504266c597e197a2c00d3d6f32108a93d4" exitCode=0 Oct 30 23:42:33.437079 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:42:33.436997 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-544b44f8d-5p6kz" Oct 30 23:42:33.437079 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:42:33.437041 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-544b44f8d-5p6kz" event={"ID":"58f7cf25-f5d0-4481-a927-4a745b692f7a","Type":"ContainerDied","Data":"b66ee2127fd8d0b08162e3f6ef34c3504266c597e197a2c00d3d6f32108a93d4"} Oct 30 23:42:33.437178 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:42:33.437096 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-544b44f8d-5p6kz" event={"ID":"58f7cf25-f5d0-4481-a927-4a745b692f7a","Type":"ContainerDied","Data":"031752aff7fa5ac6deb70609bad56693e9a040632954724c65c7ac153132fae8"} Oct 30 23:42:33.437178 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:42:33.437112 4927 scope.go:122] "RemoveContainer" containerID="b66ee2127fd8d0b08162e3f6ef34c3504266c597e197a2c00d3d6f32108a93d4" Oct 30 23:42:33.440417 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:42:33.440391 4927 generic.go:356] "Generic (PLEG): container finished" podID="840aaae0-a215-4ccd-b420-d6f5723b65ae" containerID="baeb8c991d9dac5ff1f990a149a34c9e24aae4f355b3d408d8031880243578c5" exitCode=0 Oct 30 23:42:33.440533 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:42:33.440418 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-9qs84" event={"ID":"840aaae0-a215-4ccd-b420-d6f5723b65ae","Type":"ContainerDied","Data":"baeb8c991d9dac5ff1f990a149a34c9e24aae4f355b3d408d8031880243578c5"} Oct 30 23:42:33.440665 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:42:33.440657 4927 scope.go:122] "RemoveContainer" containerID="baeb8c991d9dac5ff1f990a149a34c9e24aae4f355b3d408d8031880243578c5" Oct 30 23:42:33.448171 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:42:33.448156 4927 scope.go:122] "RemoveContainer" containerID="b66ee2127fd8d0b08162e3f6ef34c3504266c597e197a2c00d3d6f32108a93d4" Oct 30 23:42:33.448449 ip-10-0-141-180 kubenswrapper[4927]: E1030 23:42:33.448421 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b66ee2127fd8d0b08162e3f6ef34c3504266c597e197a2c00d3d6f32108a93d4\": container with ID starting with b66ee2127fd8d0b08162e3f6ef34c3504266c597e197a2c00d3d6f32108a93d4 not found: ID does not exist" containerID="b66ee2127fd8d0b08162e3f6ef34c3504266c597e197a2c00d3d6f32108a93d4" Oct 30 23:42:33.448545 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:42:33.448457 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b66ee2127fd8d0b08162e3f6ef34c3504266c597e197a2c00d3d6f32108a93d4"} err="failed to get container status \"b66ee2127fd8d0b08162e3f6ef34c3504266c597e197a2c00d3d6f32108a93d4\": rpc error: code = NotFound desc = could not find container \"b66ee2127fd8d0b08162e3f6ef34c3504266c597e197a2c00d3d6f32108a93d4\": container with ID starting with b66ee2127fd8d0b08162e3f6ef34c3504266c597e197a2c00d3d6f32108a93d4 not found: ID does not exist" Oct 30 23:42:33.512773 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:42:33.510511 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/58f7cf25-f5d0-4481-a927-4a745b692f7a-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/58f7cf25-f5d0-4481-a927-4a745b692f7a-image-registry-private-configuration\") pod \"58f7cf25-f5d0-4481-a927-4a745b692f7a\" (UID: \"58f7cf25-f5d0-4481-a927-4a745b692f7a\") " Oct 30 23:42:33.512773 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:42:33.510552 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58f7cf25-f5d0-4481-a927-4a745b692f7a-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/58f7cf25-f5d0-4481-a927-4a745b692f7a-bound-sa-token\") pod \"58f7cf25-f5d0-4481-a927-4a745b692f7a\" (UID: \"58f7cf25-f5d0-4481-a927-4a745b692f7a\") " Oct 30 23:42:33.512773 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:42:33.510583 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/58f7cf25-f5d0-4481-a927-4a745b692f7a-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/58f7cf25-f5d0-4481-a927-4a745b692f7a-installation-pull-secrets\") pod \"58f7cf25-f5d0-4481-a927-4a745b692f7a\" (UID: \"58f7cf25-f5d0-4481-a927-4a745b692f7a\") " Oct 30 23:42:33.512773 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:42:33.510606 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/58f7cf25-f5d0-4481-a927-4a745b692f7a-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/58f7cf25-f5d0-4481-a927-4a745b692f7a-registry-certificates\") pod \"58f7cf25-f5d0-4481-a927-4a745b692f7a\" (UID: \"58f7cf25-f5d0-4481-a927-4a745b692f7a\") " Oct 30 23:42:33.512773 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:42:33.510627 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58f7cf25-f5d0-4481-a927-4a745b692f7a-kube-api-access-sx9pp\" (UniqueName: \"kubernetes.io/projected/58f7cf25-f5d0-4481-a927-4a745b692f7a-kube-api-access-sx9pp\") pod \"58f7cf25-f5d0-4481-a927-4a745b692f7a\" (UID: \"58f7cf25-f5d0-4481-a927-4a745b692f7a\") " Oct 30 23:42:33.512773 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:42:33.510664 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58f7cf25-f5d0-4481-a927-4a745b692f7a-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/58f7cf25-f5d0-4481-a927-4a745b692f7a-ca-trust-extracted\") pod \"58f7cf25-f5d0-4481-a927-4a745b692f7a\" (UID: \"58f7cf25-f5d0-4481-a927-4a745b692f7a\") " Oct 30 23:42:33.512773 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:42:33.510705 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58f7cf25-f5d0-4481-a927-4a745b692f7a-registry-tls\" (UniqueName: \"kubernetes.io/projected/58f7cf25-f5d0-4481-a927-4a745b692f7a-registry-tls\") pod \"58f7cf25-f5d0-4481-a927-4a745b692f7a\" (UID: \"58f7cf25-f5d0-4481-a927-4a745b692f7a\") " Oct 30 23:42:33.512773 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:42:33.510728 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/58f7cf25-f5d0-4481-a927-4a745b692f7a-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/58f7cf25-f5d0-4481-a927-4a745b692f7a-trusted-ca\") pod \"58f7cf25-f5d0-4481-a927-4a745b692f7a\" (UID: \"58f7cf25-f5d0-4481-a927-4a745b692f7a\") " Oct 30 23:42:33.512773 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:42:33.511680 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58f7cf25-f5d0-4481-a927-4a745b692f7a-registry-certificates" pod "58f7cf25-f5d0-4481-a927-4a745b692f7a" (UID: "58f7cf25-f5d0-4481-a927-4a745b692f7a"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:42:33.513714 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:42:33.513065 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58f7cf25-f5d0-4481-a927-4a745b692f7a-trusted-ca" pod "58f7cf25-f5d0-4481-a927-4a745b692f7a" (UID: "58f7cf25-f5d0-4481-a927-4a745b692f7a"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:42:33.515107 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:42:33.515070 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/58f7cf25-f5d0-4481-a927-4a745b692f7a-image-registry-private-configuration" pod "58f7cf25-f5d0-4481-a927-4a745b692f7a" (UID: "58f7cf25-f5d0-4481-a927-4a745b692f7a"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 23:42:33.515383 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:42:33.515367 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/58f7cf25-f5d0-4481-a927-4a745b692f7a-installation-pull-secrets" pod "58f7cf25-f5d0-4481-a927-4a745b692f7a" (UID: "58f7cf25-f5d0-4481-a927-4a745b692f7a"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 23:42:33.515495 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:42:33.515416 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58f7cf25-f5d0-4481-a927-4a745b692f7a-registry-tls" pod "58f7cf25-f5d0-4481-a927-4a745b692f7a" (UID: "58f7cf25-f5d0-4481-a927-4a745b692f7a"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:42:33.515591 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:42:33.515428 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58f7cf25-f5d0-4481-a927-4a745b692f7a-kube-api-access-sx9pp" pod "58f7cf25-f5d0-4481-a927-4a745b692f7a" (UID: "58f7cf25-f5d0-4481-a927-4a745b692f7a"). InnerVolumeSpecName "kube-api-access-sx9pp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:42:33.515700 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:42:33.515670 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58f7cf25-f5d0-4481-a927-4a745b692f7a-bound-sa-token" pod "58f7cf25-f5d0-4481-a927-4a745b692f7a" (UID: "58f7cf25-f5d0-4481-a927-4a745b692f7a"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:42:33.518684 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:42:33.518661 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58f7cf25-f5d0-4481-a927-4a745b692f7a-ca-trust-extracted" pod "58f7cf25-f5d0-4481-a927-4a745b692f7a" (UID: "58f7cf25-f5d0-4481-a927-4a745b692f7a"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:42:33.612225 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:42:33.612182 4927 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/58f7cf25-f5d0-4481-a927-4a745b692f7a-image-registry-private-configuration\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 23:42:33.612225 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:42:33.612222 4927 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/58f7cf25-f5d0-4481-a927-4a745b692f7a-bound-sa-token\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 23:42:33.612225 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:42:33.612232 4927 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/58f7cf25-f5d0-4481-a927-4a745b692f7a-installation-pull-secrets\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 23:42:33.612468 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:42:33.612242 4927 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/58f7cf25-f5d0-4481-a927-4a745b692f7a-registry-certificates\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 23:42:33.612468 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:42:33.612253 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sx9pp\" (UniqueName: \"kubernetes.io/projected/58f7cf25-f5d0-4481-a927-4a745b692f7a-kube-api-access-sx9pp\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 23:42:33.612468 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:42:33.612262 4927 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/58f7cf25-f5d0-4481-a927-4a745b692f7a-ca-trust-extracted\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 23:42:33.612468 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:42:33.612270 4927 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/58f7cf25-f5d0-4481-a927-4a745b692f7a-registry-tls\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 23:42:33.612468 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:42:33.612278 4927 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/58f7cf25-f5d0-4481-a927-4a745b692f7a-trusted-ca\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 23:42:34.035157 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:42:34.035077 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-544b44f8d-5p6kz"] Oct 30 23:42:34.041768 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:42:34.041741 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-544b44f8d-5p6kz"] Oct 30 23:42:34.448724 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:42:34.448687 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-9qs84" event={"ID":"840aaae0-a215-4ccd-b420-d6f5723b65ae","Type":"ContainerStarted","Data":"5a68feb6b2fd92b7422b5502fee7110ce8541bcdc060916423d538dd1eb282ff"} Oct 30 23:42:34.451071 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:42:34.451043 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-9mqqt" event={"ID":"10774606-8e0c-460b-bd5f-4cb6d121b30b","Type":"ContainerStarted","Data":"5a3ecce59c895873a4d1eeb16538eccadbe0446b0bc2ce455bc90c1284c21e8f"} Oct 30 23:42:34.486366 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:42:34.486320 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-r9crl-vqcgk"] Oct 30 23:42:34.486510 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:42:34.486491 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-insights/periodic-gathering-r9crl-vqcgk" podUID="a78879b4-333e-4f1a-88bc-e96aee0e6d6b" containerName="insights-gathering" containerID="cri-o://97361002a0136337989d4473e0d7b49dec6c5cdf9e9d6f07e7292fb2e86c3fd1" gracePeriod=30 Oct 30 23:42:34.745312 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:42:34.745286 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-r9crl-vqcgk" Oct 30 23:42:34.821294 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:42:34.821266 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a78879b4-333e-4f1a-88bc-e96aee0e6d6b-kube-api-access-sdj76\" (UniqueName: \"kubernetes.io/projected/a78879b4-333e-4f1a-88bc-e96aee0e6d6b-kube-api-access-sdj76\") pod \"a78879b4-333e-4f1a-88bc-e96aee0e6d6b\" (UID: \"a78879b4-333e-4f1a-88bc-e96aee0e6d6b\") " Oct 30 23:42:34.821462 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:42:34.821357 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a78879b4-333e-4f1a-88bc-e96aee0e6d6b-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/a78879b4-333e-4f1a-88bc-e96aee0e6d6b-archives-path\") pod \"a78879b4-333e-4f1a-88bc-e96aee0e6d6b\" (UID: \"a78879b4-333e-4f1a-88bc-e96aee0e6d6b\") " Oct 30 23:42:34.821462 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:42:34.821397 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a78879b4-333e-4f1a-88bc-e96aee0e6d6b-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a78879b4-333e-4f1a-88bc-e96aee0e6d6b-service-ca-bundle\") pod \"a78879b4-333e-4f1a-88bc-e96aee0e6d6b\" (UID: \"a78879b4-333e-4f1a-88bc-e96aee0e6d6b\") " Oct 30 23:42:34.821462 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:42:34.821454 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a78879b4-333e-4f1a-88bc-e96aee0e6d6b-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a78879b4-333e-4f1a-88bc-e96aee0e6d6b-trusted-ca-bundle\") pod \"a78879b4-333e-4f1a-88bc-e96aee0e6d6b\" (UID: \"a78879b4-333e-4f1a-88bc-e96aee0e6d6b\") " Oct 30 23:42:34.821784 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:42:34.821752 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a78879b4-333e-4f1a-88bc-e96aee0e6d6b-service-ca-bundle" pod "a78879b4-333e-4f1a-88bc-e96aee0e6d6b" (UID: "a78879b4-333e-4f1a-88bc-e96aee0e6d6b"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:42:34.821878 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:42:34.821758 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a78879b4-333e-4f1a-88bc-e96aee0e6d6b-archives-path" pod "a78879b4-333e-4f1a-88bc-e96aee0e6d6b" (UID: "a78879b4-333e-4f1a-88bc-e96aee0e6d6b"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:42:34.821878 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:42:34.821853 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a78879b4-333e-4f1a-88bc-e96aee0e6d6b-trusted-ca-bundle" pod "a78879b4-333e-4f1a-88bc-e96aee0e6d6b" (UID: "a78879b4-333e-4f1a-88bc-e96aee0e6d6b"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:42:34.823725 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:42:34.823698 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a78879b4-333e-4f1a-88bc-e96aee0e6d6b-kube-api-access-sdj76" pod "a78879b4-333e-4f1a-88bc-e96aee0e6d6b" (UID: "a78879b4-333e-4f1a-88bc-e96aee0e6d6b"). InnerVolumeSpecName "kube-api-access-sdj76". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:42:34.922158 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:42:34.922118 4927 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a78879b4-333e-4f1a-88bc-e96aee0e6d6b-trusted-ca-bundle\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 23:42:34.922158 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:42:34.922159 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sdj76\" (UniqueName: \"kubernetes.io/projected/a78879b4-333e-4f1a-88bc-e96aee0e6d6b-kube-api-access-sdj76\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 23:42:34.922396 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:42:34.922177 4927 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/a78879b4-333e-4f1a-88bc-e96aee0e6d6b-archives-path\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 23:42:34.922396 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:42:34.922190 4927 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a78879b4-333e-4f1a-88bc-e96aee0e6d6b-service-ca-bundle\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 23:42:34.937009 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:42:34.936974 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-tjj7g-mc8s9"] Oct 30 23:42:34.937316 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:42:34.937300 4927 cpu_manager.go:414] "containerMap: removing container" podUID="17461b46-7ebf-49b7-afaf-12e20e3994a3" containerName="registry" Oct 30 23:42:34.937430 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:42:34.937319 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17461b46-7ebf-49b7-afaf-12e20e3994a3" containerName="registry" Oct 30 23:42:34.937430 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:42:34.937355 4927 cpu_manager.go:414] "containerMap: removing container" podUID="58f7cf25-f5d0-4481-a927-4a745b692f7a" containerName="registry" Oct 30 23:42:34.937430 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:42:34.937362 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58f7cf25-f5d0-4481-a927-4a745b692f7a" containerName="registry" Oct 30 23:42:34.937430 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:42:34.937395 4927 cpu_manager.go:414] "containerMap: removing container" podUID="a78879b4-333e-4f1a-88bc-e96aee0e6d6b" containerName="insights-gathering" Oct 30 23:42:34.937430 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:42:34.937403 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a78879b4-333e-4f1a-88bc-e96aee0e6d6b" containerName="insights-gathering" Oct 30 23:42:34.937670 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:42:34.937483 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="a78879b4-333e-4f1a-88bc-e96aee0e6d6b" containerName="insights-gathering" Oct 30 23:42:34.937670 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:42:34.937498 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="17461b46-7ebf-49b7-afaf-12e20e3994a3" containerName="registry" Oct 30 23:42:34.937670 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:42:34.937509 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="58f7cf25-f5d0-4481-a927-4a745b692f7a" containerName="registry" Oct 30 23:42:34.940530 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:42:34.940508 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-tjj7g-mc8s9" Oct 30 23:42:34.994701 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:42:34.994648 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-tjj7g-mc8s9"] Oct 30 23:42:35.022817 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:42:35.022791 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/dbcca886-66c4-4991-9b6b-8dfe9e8fa37b-archives-path\") pod \"periodic-gathering-tjj7g-mc8s9\" (UID: \"dbcca886-66c4-4991-9b6b-8dfe9e8fa37b\") " pod="openshift-insights/periodic-gathering-tjj7g-mc8s9" Oct 30 23:42:35.022919 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:42:35.022829 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fqmbx\" (UniqueName: \"kubernetes.io/projected/dbcca886-66c4-4991-9b6b-8dfe9e8fa37b-kube-api-access-fqmbx\") pod \"periodic-gathering-tjj7g-mc8s9\" (UID: \"dbcca886-66c4-4991-9b6b-8dfe9e8fa37b\") " pod="openshift-insights/periodic-gathering-tjj7g-mc8s9" Oct 30 23:42:35.022919 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:42:35.022852 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dbcca886-66c4-4991-9b6b-8dfe9e8fa37b-trusted-ca-bundle\") pod \"periodic-gathering-tjj7g-mc8s9\" (UID: \"dbcca886-66c4-4991-9b6b-8dfe9e8fa37b\") " pod="openshift-insights/periodic-gathering-tjj7g-mc8s9" Oct 30 23:42:35.023001 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:42:35.022983 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dbcca886-66c4-4991-9b6b-8dfe9e8fa37b-service-ca-bundle\") pod \"periodic-gathering-tjj7g-mc8s9\" (UID: \"dbcca886-66c4-4991-9b6b-8dfe9e8fa37b\") " pod="openshift-insights/periodic-gathering-tjj7g-mc8s9" Oct 30 23:42:35.083139 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:42:35.083106 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="58f7cf25-f5d0-4481-a927-4a745b692f7a" path="/var/lib/kubelet/pods/58f7cf25-f5d0-4481-a927-4a745b692f7a/volumes" Oct 30 23:42:35.124212 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:42:35.124183 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/dbcca886-66c4-4991-9b6b-8dfe9e8fa37b-archives-path\") pod \"periodic-gathering-tjj7g-mc8s9\" (UID: \"dbcca886-66c4-4991-9b6b-8dfe9e8fa37b\") " pod="openshift-insights/periodic-gathering-tjj7g-mc8s9" Oct 30 23:42:35.124344 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:42:35.124217 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqmbx\" (UniqueName: \"kubernetes.io/projected/dbcca886-66c4-4991-9b6b-8dfe9e8fa37b-kube-api-access-fqmbx\") pod \"periodic-gathering-tjj7g-mc8s9\" (UID: \"dbcca886-66c4-4991-9b6b-8dfe9e8fa37b\") " pod="openshift-insights/periodic-gathering-tjj7g-mc8s9" Oct 30 23:42:35.124344 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:42:35.124237 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dbcca886-66c4-4991-9b6b-8dfe9e8fa37b-trusted-ca-bundle\") pod \"periodic-gathering-tjj7g-mc8s9\" (UID: \"dbcca886-66c4-4991-9b6b-8dfe9e8fa37b\") " pod="openshift-insights/periodic-gathering-tjj7g-mc8s9" Oct 30 23:42:35.124344 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:42:35.124269 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dbcca886-66c4-4991-9b6b-8dfe9e8fa37b-service-ca-bundle\") pod \"periodic-gathering-tjj7g-mc8s9\" (UID: \"dbcca886-66c4-4991-9b6b-8dfe9e8fa37b\") " pod="openshift-insights/periodic-gathering-tjj7g-mc8s9" Oct 30 23:42:35.124645 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:42:35.124623 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/dbcca886-66c4-4991-9b6b-8dfe9e8fa37b-archives-path\") pod \"periodic-gathering-tjj7g-mc8s9\" (UID: \"dbcca886-66c4-4991-9b6b-8dfe9e8fa37b\") " pod="openshift-insights/periodic-gathering-tjj7g-mc8s9" Oct 30 23:42:35.124867 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:42:35.124850 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dbcca886-66c4-4991-9b6b-8dfe9e8fa37b-service-ca-bundle\") pod \"periodic-gathering-tjj7g-mc8s9\" (UID: \"dbcca886-66c4-4991-9b6b-8dfe9e8fa37b\") " pod="openshift-insights/periodic-gathering-tjj7g-mc8s9" Oct 30 23:42:35.125000 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:42:35.124981 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dbcca886-66c4-4991-9b6b-8dfe9e8fa37b-trusted-ca-bundle\") pod \"periodic-gathering-tjj7g-mc8s9\" (UID: \"dbcca886-66c4-4991-9b6b-8dfe9e8fa37b\") " pod="openshift-insights/periodic-gathering-tjj7g-mc8s9" Oct 30 23:42:35.186494 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:42:35.186461 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqmbx\" (UniqueName: \"kubernetes.io/projected/dbcca886-66c4-4991-9b6b-8dfe9e8fa37b-kube-api-access-fqmbx\") pod \"periodic-gathering-tjj7g-mc8s9\" (UID: \"dbcca886-66c4-4991-9b6b-8dfe9e8fa37b\") " pod="openshift-insights/periodic-gathering-tjj7g-mc8s9" Oct 30 23:42:35.253787 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:42:35.253723 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-tjj7g-mc8s9" Oct 30 23:42:35.456067 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:42:35.456031 4927 generic.go:356] "Generic (PLEG): container finished" podID="a78879b4-333e-4f1a-88bc-e96aee0e6d6b" containerID="97361002a0136337989d4473e0d7b49dec6c5cdf9e9d6f07e7292fb2e86c3fd1" exitCode=2 Oct 30 23:42:35.456540 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:42:35.456074 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-r9crl-vqcgk" Oct 30 23:42:35.456540 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:42:35.456086 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-r9crl-vqcgk" event={"ID":"a78879b4-333e-4f1a-88bc-e96aee0e6d6b","Type":"ContainerDied","Data":"97361002a0136337989d4473e0d7b49dec6c5cdf9e9d6f07e7292fb2e86c3fd1"} Oct 30 23:42:35.456540 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:42:35.456150 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-r9crl-vqcgk" event={"ID":"a78879b4-333e-4f1a-88bc-e96aee0e6d6b","Type":"ContainerDied","Data":"bbfd66c255357d0b1ae9f531564af260a8d8bd00c26ccfe7f763b2cdb3afeaec"} Oct 30 23:42:35.456540 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:42:35.456187 4927 scope.go:122] "RemoveContainer" containerID="97361002a0136337989d4473e0d7b49dec6c5cdf9e9d6f07e7292fb2e86c3fd1" Oct 30 23:42:35.463867 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:42:35.463851 4927 scope.go:122] "RemoveContainer" containerID="97361002a0136337989d4473e0d7b49dec6c5cdf9e9d6f07e7292fb2e86c3fd1" Oct 30 23:42:35.464139 ip-10-0-141-180 kubenswrapper[4927]: E1030 23:42:35.464123 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"97361002a0136337989d4473e0d7b49dec6c5cdf9e9d6f07e7292fb2e86c3fd1\": container with ID starting with 97361002a0136337989d4473e0d7b49dec6c5cdf9e9d6f07e7292fb2e86c3fd1 not found: ID does not exist" containerID="97361002a0136337989d4473e0d7b49dec6c5cdf9e9d6f07e7292fb2e86c3fd1" Oct 30 23:42:35.464198 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:42:35.464151 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97361002a0136337989d4473e0d7b49dec6c5cdf9e9d6f07e7292fb2e86c3fd1"} err="failed to get container status \"97361002a0136337989d4473e0d7b49dec6c5cdf9e9d6f07e7292fb2e86c3fd1\": rpc error: code = NotFound desc = could not find container \"97361002a0136337989d4473e0d7b49dec6c5cdf9e9d6f07e7292fb2e86c3fd1\": container with ID starting with 97361002a0136337989d4473e0d7b49dec6c5cdf9e9d6f07e7292fb2e86c3fd1 not found: ID does not exist" Oct 30 23:42:35.622872 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:42:35.622794 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-r9crl-vqcgk"] Oct 30 23:42:35.625674 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:42:35.625648 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-tjj7g-mc8s9"] Oct 30 23:42:35.628628 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:42:35.628600 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbcca886_66c4_4991_9b6b_8dfe9e8fa37b.slice/crio-2da140d069c06ef1d3d4061138cda13333e90007c73fb7f83f8c9510bfdb952a WatchSource:0}: Error finding container 2da140d069c06ef1d3d4061138cda13333e90007c73fb7f83f8c9510bfdb952a: Status 404 returned error can't find the container with id 2da140d069c06ef1d3d4061138cda13333e90007c73fb7f83f8c9510bfdb952a Oct 30 23:42:35.673899 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:42:35.673876 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-insights/periodic-gathering-r9crl-vqcgk"] Oct 30 23:42:36.463180 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:42:36.463146 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-tjj7g-mc8s9" event={"ID":"dbcca886-66c4-4991-9b6b-8dfe9e8fa37b","Type":"ContainerStarted","Data":"dfea78d44e79b080febe0aabd34e72e0c67d822bbc7f5d197936f2f42970b91b"} Oct 30 23:42:36.463580 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:42:36.463192 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-tjj7g-mc8s9" event={"ID":"dbcca886-66c4-4991-9b6b-8dfe9e8fa37b","Type":"ContainerStarted","Data":"2da140d069c06ef1d3d4061138cda13333e90007c73fb7f83f8c9510bfdb952a"} Oct 30 23:42:36.581251 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:42:36.581194 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-tjj7g-mc8s9" podStartSLOduration=2.581181247 podStartE2EDuration="2.581181247s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:42:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:42:36.580525267 +0000 UTC m=+69.616856106" watchObservedRunningTime="2026-10-30 23:42:36.581181247 +0000 UTC m=+69.617512059" Oct 30 23:42:37.083407 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:42:37.083379 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a78879b4-333e-4f1a-88bc-e96aee0e6d6b" path="/var/lib/kubelet/pods/a78879b4-333e-4f1a-88bc-e96aee0e6d6b/volumes" Oct 30 23:42:37.444967 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:42:37.444933 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:42:37.595493 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:42:37.595452 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:42:39.475017 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:42:39.474985 4927 generic.go:356] "Generic (PLEG): container finished" podID="f464ef41-45c9-4ff4-b410-e2d9d6cdbdf5" containerID="704942de4f2ddb8ae14b522030de2121153ea50e4ac74156f93579566e481cd9" exitCode=0 Oct 30 23:42:39.475450 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:42:39.475057 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-fbgth" event={"ID":"f464ef41-45c9-4ff4-b410-e2d9d6cdbdf5","Type":"ContainerDied","Data":"704942de4f2ddb8ae14b522030de2121153ea50e4ac74156f93579566e481cd9"} Oct 30 23:42:39.475450 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:42:39.475368 4927 scope.go:122] "RemoveContainer" containerID="704942de4f2ddb8ae14b522030de2121153ea50e4ac74156f93579566e481cd9" Oct 30 23:42:40.480929 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:42:40.480892 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-fbgth" event={"ID":"f464ef41-45c9-4ff4-b410-e2d9d6cdbdf5","Type":"ContainerStarted","Data":"9a4b5f6992822a3127abce7f5aa4920ae20682939c46ef191550e8a07e4a415a"} Oct 30 23:42:43.691625 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:42:43.691591 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:42:43.887937 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:42:43.887883 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:42:45.084926 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:42:45.084892 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:42:45.234481 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:42:45.234436 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:42:47.620961 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:42:47.620926 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:42:47.767647 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:42:47.767605 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:42:48.273779 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:42:48.273740 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:42:48.420719 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:42:48.420684 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:42:49.676742 ip-10-0-141-180 kubenswrapper[4927]: E1030 23:42:49.676702 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sqv69" containerName="extractor" Oct 30 23:42:51.217932 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:42:51.217901 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:42:51.431378 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:42:51.431326 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:42:52.456370 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:42:52.456310 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:42:52.610609 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:42:52.610571 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:42:53.013756 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:42:53.013715 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:42:53.163946 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:42:53.163910 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:43:09.873780 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:43:09.873745 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:43:17.495190 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:43:17.495147 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:43:19.676970 ip-10-0-141-180 kubenswrapper[4927]: E1030 23:43:19.676931 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sqv69" containerName="extractor" Oct 30 23:43:49.677023 ip-10-0-141-180 kubenswrapper[4927]: E1030 23:43:49.676974 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sqv69" containerName="extractor" Oct 30 23:43:56.217382 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:43:56.217320 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:44:00.788850 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:44:00.788769 4927 generic.go:356] "Generic (PLEG): container finished" podID="dbcca886-66c4-4991-9b6b-8dfe9e8fa37b" containerID="dfea78d44e79b080febe0aabd34e72e0c67d822bbc7f5d197936f2f42970b91b" exitCode=0 Oct 30 23:44:00.789217 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:44:00.788841 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-tjj7g-mc8s9" event={"ID":"dbcca886-66c4-4991-9b6b-8dfe9e8fa37b","Type":"ContainerDied","Data":"dfea78d44e79b080febe0aabd34e72e0c67d822bbc7f5d197936f2f42970b91b"} Oct 30 23:44:01.933591 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:44:01.933569 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-tjj7g-mc8s9" Oct 30 23:44:01.966739 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:44:01.966712 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dbcca886-66c4-4991-9b6b-8dfe9e8fa37b-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dbcca886-66c4-4991-9b6b-8dfe9e8fa37b-trusted-ca-bundle\") pod \"dbcca886-66c4-4991-9b6b-8dfe9e8fa37b\" (UID: \"dbcca886-66c4-4991-9b6b-8dfe9e8fa37b\") " Oct 30 23:44:01.966867 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:44:01.966771 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dbcca886-66c4-4991-9b6b-8dfe9e8fa37b-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/dbcca886-66c4-4991-9b6b-8dfe9e8fa37b-archives-path\") pod \"dbcca886-66c4-4991-9b6b-8dfe9e8fa37b\" (UID: \"dbcca886-66c4-4991-9b6b-8dfe9e8fa37b\") " Oct 30 23:44:01.966867 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:44:01.966804 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dbcca886-66c4-4991-9b6b-8dfe9e8fa37b-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dbcca886-66c4-4991-9b6b-8dfe9e8fa37b-service-ca-bundle\") pod \"dbcca886-66c4-4991-9b6b-8dfe9e8fa37b\" (UID: \"dbcca886-66c4-4991-9b6b-8dfe9e8fa37b\") " Oct 30 23:44:01.966867 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:44:01.966837 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dbcca886-66c4-4991-9b6b-8dfe9e8fa37b-kube-api-access-fqmbx\" (UniqueName: \"kubernetes.io/projected/dbcca886-66c4-4991-9b6b-8dfe9e8fa37b-kube-api-access-fqmbx\") pod \"dbcca886-66c4-4991-9b6b-8dfe9e8fa37b\" (UID: \"dbcca886-66c4-4991-9b6b-8dfe9e8fa37b\") " Oct 30 23:44:01.967109 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:44:01.967083 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dbcca886-66c4-4991-9b6b-8dfe9e8fa37b-trusted-ca-bundle" pod "dbcca886-66c4-4991-9b6b-8dfe9e8fa37b" (UID: "dbcca886-66c4-4991-9b6b-8dfe9e8fa37b"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:44:01.967187 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:44:01.967132 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dbcca886-66c4-4991-9b6b-8dfe9e8fa37b-service-ca-bundle" pod "dbcca886-66c4-4991-9b6b-8dfe9e8fa37b" (UID: "dbcca886-66c4-4991-9b6b-8dfe9e8fa37b"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:44:01.967187 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:44:01.967163 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbcca886-66c4-4991-9b6b-8dfe9e8fa37b-archives-path" pod "dbcca886-66c4-4991-9b6b-8dfe9e8fa37b" (UID: "dbcca886-66c4-4991-9b6b-8dfe9e8fa37b"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:44:01.968953 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:44:01.968934 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbcca886-66c4-4991-9b6b-8dfe9e8fa37b-kube-api-access-fqmbx" pod "dbcca886-66c4-4991-9b6b-8dfe9e8fa37b" (UID: "dbcca886-66c4-4991-9b6b-8dfe9e8fa37b"). InnerVolumeSpecName "kube-api-access-fqmbx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:44:02.067943 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:44:02.067860 4927 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dbcca886-66c4-4991-9b6b-8dfe9e8fa37b-service-ca-bundle\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 23:44:02.067943 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:44:02.067898 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fqmbx\" (UniqueName: \"kubernetes.io/projected/dbcca886-66c4-4991-9b6b-8dfe9e8fa37b-kube-api-access-fqmbx\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 23:44:02.067943 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:44:02.067916 4927 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dbcca886-66c4-4991-9b6b-8dfe9e8fa37b-trusted-ca-bundle\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 23:44:02.067943 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:44:02.067933 4927 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/dbcca886-66c4-4991-9b6b-8dfe9e8fa37b-archives-path\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 23:44:02.797186 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:44:02.797156 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-tjj7g-mc8s9" Oct 30 23:44:02.797379 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:44:02.797144 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-tjj7g-mc8s9" event={"ID":"dbcca886-66c4-4991-9b6b-8dfe9e8fa37b","Type":"ContainerDied","Data":"2da140d069c06ef1d3d4061138cda13333e90007c73fb7f83f8c9510bfdb952a"} Oct 30 23:44:02.797379 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:44:02.797289 4927 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2da140d069c06ef1d3d4061138cda13333e90007c73fb7f83f8c9510bfdb952a" Oct 30 23:44:03.787296 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:44:03.787259 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:44:03.984899 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:44:03.984860 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:44:17.852669 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:44:17.852633 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:44:19.676741 ip-10-0-141-180 kubenswrapper[4927]: E1030 23:44:19.676697 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sqv69" containerName="extractor" Oct 30 23:44:31.157436 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:44:31.157395 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:44:35.000471 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:44:35.000442 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:44:35.199487 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:44:35.199450 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:44:49.675730 ip-10-0-141-180 kubenswrapper[4927]: E1030 23:44:49.675686 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sqv69" containerName="extractor" Oct 30 23:45:02.678841 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:02.678800 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:45:03.633725 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:03.633686 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:45:03.731060 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:03.731025 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448rs44"] Oct 30 23:45:03.731477 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:03.731321 4927 cpu_manager.go:414] "containerMap: removing container" podUID="dbcca886-66c4-4991-9b6b-8dfe9e8fa37b" containerName="insights-gathering" Oct 30 23:45:03.731477 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:03.731343 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dbcca886-66c4-4991-9b6b-8dfe9e8fa37b" containerName="insights-gathering" Oct 30 23:45:03.731477 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:03.731414 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="dbcca886-66c4-4991-9b6b-8dfe9e8fa37b" containerName="insights-gathering" Oct 30 23:45:03.734609 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:03.734588 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448rs44" Oct 30 23:45:03.780813 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:03.780775 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cwms65"] Oct 30 23:45:03.784129 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:03.784111 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cwms65" Oct 30 23:45:03.790696 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:03.790668 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448rs44"] Oct 30 23:45:03.831327 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:03.831289 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Oct 30 23:45:03.831327 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:03.831293 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Oct 30 23:45:03.831327 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:03.831324 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c4959aeb960c8e20c88ac2de924a35b8221494-145dcd5a-dockercfg-rppsn\"" Oct 30 23:45:03.832577 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:03.832552 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fd2ml\" (UniqueName: \"kubernetes.io/projected/bb30f2f7-5143-4b00-84c4-b691cc0b3da5-kube-api-access-fd2ml\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448rs44\" (UID: \"bb30f2f7-5143-4b00-84c4-b691cc0b3da5\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448rs44" Oct 30 23:45:03.832709 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:03.832586 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bb30f2f7-5143-4b00-84c4-b691cc0b3da5-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448rs44\" (UID: \"bb30f2f7-5143-4b00-84c4-b691cc0b3da5\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448rs44" Oct 30 23:45:03.832709 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:03.832605 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bb30f2f7-5143-4b00-84c4-b691cc0b3da5-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448rs44\" (UID: \"bb30f2f7-5143-4b00-84c4-b691cc0b3da5\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448rs44" Oct 30 23:45:03.841629 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:03.841600 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cwms65"] Oct 30 23:45:03.891430 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:03.891322 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-d4jnj\"" Oct 30 23:45:03.933670 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:03.933626 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5q26d\" (UniqueName: \"kubernetes.io/projected/604b4e17-0542-42e7-8c4b-82b0700f781d-kube-api-access-5q26d\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cwms65\" (UID: \"604b4e17-0542-42e7-8c4b-82b0700f781d\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cwms65" Oct 30 23:45:03.933847 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:03.933677 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/604b4e17-0542-42e7-8c4b-82b0700f781d-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cwms65\" (UID: \"604b4e17-0542-42e7-8c4b-82b0700f781d\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cwms65" Oct 30 23:45:03.933847 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:03.933706 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fd2ml\" (UniqueName: \"kubernetes.io/projected/bb30f2f7-5143-4b00-84c4-b691cc0b3da5-kube-api-access-fd2ml\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448rs44\" (UID: \"bb30f2f7-5143-4b00-84c4-b691cc0b3da5\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448rs44" Oct 30 23:45:03.933928 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:03.933900 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bb30f2f7-5143-4b00-84c4-b691cc0b3da5-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448rs44\" (UID: \"bb30f2f7-5143-4b00-84c4-b691cc0b3da5\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448rs44" Oct 30 23:45:03.933977 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:03.933948 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bb30f2f7-5143-4b00-84c4-b691cc0b3da5-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448rs44\" (UID: \"bb30f2f7-5143-4b00-84c4-b691cc0b3da5\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448rs44" Oct 30 23:45:03.934016 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:03.934001 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/604b4e17-0542-42e7-8c4b-82b0700f781d-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cwms65\" (UID: \"604b4e17-0542-42e7-8c4b-82b0700f781d\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cwms65" Oct 30 23:45:03.934277 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:03.934259 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bb30f2f7-5143-4b00-84c4-b691cc0b3da5-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448rs44\" (UID: \"bb30f2f7-5143-4b00-84c4-b691cc0b3da5\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448rs44" Oct 30 23:45:03.934388 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:03.934320 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bb30f2f7-5143-4b00-84c4-b691cc0b3da5-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448rs44\" (UID: \"bb30f2f7-5143-4b00-84c4-b691cc0b3da5\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448rs44" Oct 30 23:45:03.996309 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:03.996274 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fd2ml\" (UniqueName: \"kubernetes.io/projected/bb30f2f7-5143-4b00-84c4-b691cc0b3da5-kube-api-access-fd2ml\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448rs44\" (UID: \"bb30f2f7-5143-4b00-84c4-b691cc0b3da5\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448rs44" Oct 30 23:45:04.035154 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:04.035109 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/604b4e17-0542-42e7-8c4b-82b0700f781d-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cwms65\" (UID: \"604b4e17-0542-42e7-8c4b-82b0700f781d\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cwms65" Oct 30 23:45:04.035320 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:04.035163 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5q26d\" (UniqueName: \"kubernetes.io/projected/604b4e17-0542-42e7-8c4b-82b0700f781d-kube-api-access-5q26d\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cwms65\" (UID: \"604b4e17-0542-42e7-8c4b-82b0700f781d\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cwms65" Oct 30 23:45:04.035320 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:04.035192 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/604b4e17-0542-42e7-8c4b-82b0700f781d-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cwms65\" (UID: \"604b4e17-0542-42e7-8c4b-82b0700f781d\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cwms65" Oct 30 23:45:04.035531 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:04.035511 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/604b4e17-0542-42e7-8c4b-82b0700f781d-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cwms65\" (UID: \"604b4e17-0542-42e7-8c4b-82b0700f781d\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cwms65" Oct 30 23:45:04.035531 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:04.035525 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/604b4e17-0542-42e7-8c4b-82b0700f781d-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cwms65\" (UID: \"604b4e17-0542-42e7-8c4b-82b0700f781d\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cwms65" Oct 30 23:45:04.050156 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:04.050115 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448rs44" Oct 30 23:45:04.101184 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:04.098577 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5q26d\" (UniqueName: \"kubernetes.io/projected/604b4e17-0542-42e7-8c4b-82b0700f781d-kube-api-access-5q26d\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cwms65\" (UID: \"604b4e17-0542-42e7-8c4b-82b0700f781d\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cwms65" Oct 30 23:45:04.106266 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:04.101477 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cwms65" Oct 30 23:45:04.175622 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:04.175594 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:45:04.327433 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:04.327394 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:45:04.424655 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:04.424589 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448rs44"] Oct 30 23:45:04.427886 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:45:04.427858 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb30f2f7_5143_4b00_84c4_b691cc0b3da5.slice/crio-fbd76ab61ca53b52a12f3ed486f6799d6b1c884740f53899eda9739552e224c1 WatchSource:0}: Error finding container fbd76ab61ca53b52a12f3ed486f6799d6b1c884740f53899eda9739552e224c1: Status 404 returned error can't find the container with id fbd76ab61ca53b52a12f3ed486f6799d6b1c884740f53899eda9739552e224c1 Oct 30 23:45:04.529724 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:04.529575 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cwms65"] Oct 30 23:45:04.532200 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:45:04.532174 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod604b4e17_0542_42e7_8c4b_82b0700f781d.slice/crio-3381bcd8374e2c7056e0c5da98fc70f2e626804f591d71a72a45eb3b3eecaa19 WatchSource:0}: Error finding container 3381bcd8374e2c7056e0c5da98fc70f2e626804f591d71a72a45eb3b3eecaa19: Status 404 returned error can't find the container with id 3381bcd8374e2c7056e0c5da98fc70f2e626804f591d71a72a45eb3b3eecaa19 Oct 30 23:45:05.008752 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:05.008712 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cwms65" event={"ID":"604b4e17-0542-42e7-8c4b-82b0700f781d","Type":"ContainerStarted","Data":"3381bcd8374e2c7056e0c5da98fc70f2e626804f591d71a72a45eb3b3eecaa19"} Oct 30 23:45:05.009918 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:05.009882 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448rs44" event={"ID":"bb30f2f7-5143-4b00-84c4-b691cc0b3da5","Type":"ContainerStarted","Data":"fbd76ab61ca53b52a12f3ed486f6799d6b1c884740f53899eda9739552e224c1"} Oct 30 23:45:05.486643 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:05.486605 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:45:06.855268 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:06.855185 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be719c6lt"] Oct 30 23:45:06.859079 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:06.859050 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be719c6lt" Oct 30 23:45:06.913803 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:06.913776 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be719c6lt"] Oct 30 23:45:06.960048 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:06.960006 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/525fd99a-040b-43c8-bbb3-41a0f953ea4a-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be719c6lt\" (UID: \"525fd99a-040b-43c8-bbb3-41a0f953ea4a\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be719c6lt" Oct 30 23:45:06.960247 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:06.960064 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/525fd99a-040b-43c8-bbb3-41a0f953ea4a-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be719c6lt\" (UID: \"525fd99a-040b-43c8-bbb3-41a0f953ea4a\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be719c6lt" Oct 30 23:45:06.960247 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:06.960167 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5fgkd\" (UniqueName: \"kubernetes.io/projected/525fd99a-040b-43c8-bbb3-41a0f953ea4a-kube-api-access-5fgkd\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be719c6lt\" (UID: \"525fd99a-040b-43c8-bbb3-41a0f953ea4a\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be719c6lt" Oct 30 23:45:06.964107 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:06.964071 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e644421527c169d876fdb59235c927986d6bd2-9d2815fc-dockercfg-pdbb4\"" Oct 30 23:45:07.062228 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:07.061550 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/525fd99a-040b-43c8-bbb3-41a0f953ea4a-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be719c6lt\" (UID: \"525fd99a-040b-43c8-bbb3-41a0f953ea4a\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be719c6lt" Oct 30 23:45:07.062228 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:07.061613 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/525fd99a-040b-43c8-bbb3-41a0f953ea4a-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be719c6lt\" (UID: \"525fd99a-040b-43c8-bbb3-41a0f953ea4a\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be719c6lt" Oct 30 23:45:07.062228 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:07.061648 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fgkd\" (UniqueName: \"kubernetes.io/projected/525fd99a-040b-43c8-bbb3-41a0f953ea4a-kube-api-access-5fgkd\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be719c6lt\" (UID: \"525fd99a-040b-43c8-bbb3-41a0f953ea4a\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be719c6lt" Oct 30 23:45:07.062228 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:07.061953 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/525fd99a-040b-43c8-bbb3-41a0f953ea4a-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be719c6lt\" (UID: \"525fd99a-040b-43c8-bbb3-41a0f953ea4a\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be719c6lt" Oct 30 23:45:07.062228 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:07.062188 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/525fd99a-040b-43c8-bbb3-41a0f953ea4a-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be719c6lt\" (UID: \"525fd99a-040b-43c8-bbb3-41a0f953ea4a\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be719c6lt" Oct 30 23:45:07.125224 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:07.125151 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5fgkd\" (UniqueName: \"kubernetes.io/projected/525fd99a-040b-43c8-bbb3-41a0f953ea4a-kube-api-access-5fgkd\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be719c6lt\" (UID: \"525fd99a-040b-43c8-bbb3-41a0f953ea4a\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be719c6lt" Oct 30 23:45:07.178258 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:07.178223 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be719c6lt" Oct 30 23:45:07.562150 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:07.562027 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be719c6lt"] Oct 30 23:45:07.566396 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:45:07.566363 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod525fd99a_040b_43c8_bbb3_41a0f953ea4a.slice/crio-0ede853461d2cf58e89b70613aed68ce64aa5c6dc95512ad8cd023f47f9a6460 WatchSource:0}: Error finding container 0ede853461d2cf58e89b70613aed68ce64aa5c6dc95512ad8cd023f47f9a6460: Status 404 returned error can't find the container with id 0ede853461d2cf58e89b70613aed68ce64aa5c6dc95512ad8cd023f47f9a6460 Oct 30 23:45:07.912987 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:07.912951 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atc5bh"] Oct 30 23:45:07.918091 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:07.918068 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atc5bh" Oct 30 23:45:07.971964 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:07.971930 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atc5bh"] Oct 30 23:45:07.972234 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:07.972215 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"6541d93aaee335d6227a8778ab4e840bbe7f52-f98fbcf8-dockercfg-znszp\"" Oct 30 23:45:08.012695 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:08.012662 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7khvmq"] Oct 30 23:45:08.016973 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:08.016642 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7khvmq" Oct 30 23:45:08.023926 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:08.023891 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be719c6lt" event={"ID":"525fd99a-040b-43c8-bbb3-41a0f953ea4a","Type":"ContainerStarted","Data":"0ede853461d2cf58e89b70613aed68ce64aa5c6dc95512ad8cd023f47f9a6460"} Oct 30 23:45:08.071610 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:08.071579 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7khvmq"] Oct 30 23:45:08.072185 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:08.072158 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"da112bd609bff87f7bacd2ba941225d7cff15c-bd73b2fc-dockercfg-8lbpk\"" Oct 30 23:45:08.078656 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:08.078630 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2cc429ca-1667-4683-a0e2-a8d5fdd42a64-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atc5bh\" (UID: \"2cc429ca-1667-4683-a0e2-a8d5fdd42a64\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atc5bh" Oct 30 23:45:08.078865 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:08.078692 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2cc429ca-1667-4683-a0e2-a8d5fdd42a64-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atc5bh\" (UID: \"2cc429ca-1667-4683-a0e2-a8d5fdd42a64\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atc5bh" Oct 30 23:45:08.078865 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:08.078732 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8szzf\" (UniqueName: \"kubernetes.io/projected/2cc429ca-1667-4683-a0e2-a8d5fdd42a64-kube-api-access-8szzf\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atc5bh\" (UID: \"2cc429ca-1667-4683-a0e2-a8d5fdd42a64\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atc5bh" Oct 30 23:45:08.110763 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:08.110726 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpxwxl"] Oct 30 23:45:08.114539 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:08.114513 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpxwxl" Oct 30 23:45:08.169291 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:08.169219 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpxwxl"] Oct 30 23:45:08.172422 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:08.172380 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e3d011e69d9573cabb8c359e5ac709eb5ac240-be78d160-dockercfg-v2l8l\"" Oct 30 23:45:08.179858 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:08.179833 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2cc429ca-1667-4683-a0e2-a8d5fdd42a64-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atc5bh\" (UID: \"2cc429ca-1667-4683-a0e2-a8d5fdd42a64\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atc5bh" Oct 30 23:45:08.179975 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:08.179894 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pp9q6\" (UniqueName: \"kubernetes.io/projected/bf56ecae-e59d-4283-86fe-c32458951e5a-kube-api-access-pp9q6\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7khvmq\" (UID: \"bf56ecae-e59d-4283-86fe-c32458951e5a\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7khvmq" Oct 30 23:45:08.179975 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:08.179954 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2cc429ca-1667-4683-a0e2-a8d5fdd42a64-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atc5bh\" (UID: \"2cc429ca-1667-4683-a0e2-a8d5fdd42a64\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atc5bh" Oct 30 23:45:08.180148 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:08.180019 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bf56ecae-e59d-4283-86fe-c32458951e5a-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7khvmq\" (UID: \"bf56ecae-e59d-4283-86fe-c32458951e5a\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7khvmq" Oct 30 23:45:08.180148 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:08.180061 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bf56ecae-e59d-4283-86fe-c32458951e5a-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7khvmq\" (UID: \"bf56ecae-e59d-4283-86fe-c32458951e5a\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7khvmq" Oct 30 23:45:08.180148 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:08.180085 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8szzf\" (UniqueName: \"kubernetes.io/projected/2cc429ca-1667-4683-a0e2-a8d5fdd42a64-kube-api-access-8szzf\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atc5bh\" (UID: \"2cc429ca-1667-4683-a0e2-a8d5fdd42a64\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atc5bh" Oct 30 23:45:08.180700 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:08.180682 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2cc429ca-1667-4683-a0e2-a8d5fdd42a64-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atc5bh\" (UID: \"2cc429ca-1667-4683-a0e2-a8d5fdd42a64\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atc5bh" Oct 30 23:45:08.181590 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:08.181563 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2cc429ca-1667-4683-a0e2-a8d5fdd42a64-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atc5bh\" (UID: \"2cc429ca-1667-4683-a0e2-a8d5fdd42a64\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atc5bh" Oct 30 23:45:08.211388 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:08.211363 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wwjsn"] Oct 30 23:45:08.215056 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:08.215037 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wwjsn" Oct 30 23:45:08.242936 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:08.242916 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8szzf\" (UniqueName: \"kubernetes.io/projected/2cc429ca-1667-4683-a0e2-a8d5fdd42a64-kube-api-access-8szzf\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atc5bh\" (UID: \"2cc429ca-1667-4683-a0e2-a8d5fdd42a64\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atc5bh" Oct 30 23:45:08.270438 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:08.270409 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wwjsn"] Oct 30 23:45:08.274216 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:08.274194 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3-b83c8cba-dockercfg-jmp5v\"" Oct 30 23:45:08.280835 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:08.280810 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bf56ecae-e59d-4283-86fe-c32458951e5a-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7khvmq\" (UID: \"bf56ecae-e59d-4283-86fe-c32458951e5a\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7khvmq" Oct 30 23:45:08.280936 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:08.280856 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5gkdn\" (UniqueName: \"kubernetes.io/projected/2a201fda-d325-4697-98e9-f477125ce0a0-kube-api-access-5gkdn\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpxwxl\" (UID: \"2a201fda-d325-4697-98e9-f477125ce0a0\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpxwxl" Oct 30 23:45:08.280936 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:08.280910 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2a201fda-d325-4697-98e9-f477125ce0a0-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpxwxl\" (UID: \"2a201fda-d325-4697-98e9-f477125ce0a0\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpxwxl" Oct 30 23:45:08.281043 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:08.280946 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2a201fda-d325-4697-98e9-f477125ce0a0-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpxwxl\" (UID: \"2a201fda-d325-4697-98e9-f477125ce0a0\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpxwxl" Oct 30 23:45:08.281173 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:08.281156 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bf56ecae-e59d-4283-86fe-c32458951e5a-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7khvmq\" (UID: \"bf56ecae-e59d-4283-86fe-c32458951e5a\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7khvmq" Oct 30 23:45:08.281225 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:08.281161 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pp9q6\" (UniqueName: \"kubernetes.io/projected/bf56ecae-e59d-4283-86fe-c32458951e5a-kube-api-access-pp9q6\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7khvmq\" (UID: \"bf56ecae-e59d-4283-86fe-c32458951e5a\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7khvmq" Oct 30 23:45:08.281278 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:08.281228 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bf56ecae-e59d-4283-86fe-c32458951e5a-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7khvmq\" (UID: \"bf56ecae-e59d-4283-86fe-c32458951e5a\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7khvmq" Oct 30 23:45:08.281749 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:08.281727 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bf56ecae-e59d-4283-86fe-c32458951e5a-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7khvmq\" (UID: \"bf56ecae-e59d-4283-86fe-c32458951e5a\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7khvmq" Oct 30 23:45:08.344032 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:08.344004 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pp9q6\" (UniqueName: \"kubernetes.io/projected/bf56ecae-e59d-4283-86fe-c32458951e5a-kube-api-access-pp9q6\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7khvmq\" (UID: \"bf56ecae-e59d-4283-86fe-c32458951e5a\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7khvmq" Oct 30 23:45:08.381987 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:08.381951 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bhrj6\" (UniqueName: \"kubernetes.io/projected/70668fb4-8523-4bf5-94ed-0e8440598b20-kube-api-access-bhrj6\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wwjsn\" (UID: \"70668fb4-8523-4bf5-94ed-0e8440598b20\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wwjsn" Oct 30 23:45:08.382134 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:08.382032 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5gkdn\" (UniqueName: \"kubernetes.io/projected/2a201fda-d325-4697-98e9-f477125ce0a0-kube-api-access-5gkdn\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpxwxl\" (UID: \"2a201fda-d325-4697-98e9-f477125ce0a0\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpxwxl" Oct 30 23:45:08.382134 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:08.382053 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/70668fb4-8523-4bf5-94ed-0e8440598b20-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wwjsn\" (UID: \"70668fb4-8523-4bf5-94ed-0e8440598b20\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wwjsn" Oct 30 23:45:08.382221 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:08.382186 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/70668fb4-8523-4bf5-94ed-0e8440598b20-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wwjsn\" (UID: \"70668fb4-8523-4bf5-94ed-0e8440598b20\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wwjsn" Oct 30 23:45:08.382313 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:08.382291 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2a201fda-d325-4697-98e9-f477125ce0a0-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpxwxl\" (UID: \"2a201fda-d325-4697-98e9-f477125ce0a0\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpxwxl" Oct 30 23:45:08.382388 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:08.382371 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2a201fda-d325-4697-98e9-f477125ce0a0-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpxwxl\" (UID: \"2a201fda-d325-4697-98e9-f477125ce0a0\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpxwxl" Oct 30 23:45:08.382816 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:08.382786 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2a201fda-d325-4697-98e9-f477125ce0a0-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpxwxl\" (UID: \"2a201fda-d325-4697-98e9-f477125ce0a0\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpxwxl" Oct 30 23:45:08.382936 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:08.382796 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2a201fda-d325-4697-98e9-f477125ce0a0-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpxwxl\" (UID: \"2a201fda-d325-4697-98e9-f477125ce0a0\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpxwxl" Oct 30 23:45:08.447418 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:08.447322 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5gkdn\" (UniqueName: \"kubernetes.io/projected/2a201fda-d325-4697-98e9-f477125ce0a0-kube-api-access-5gkdn\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpxwxl\" (UID: \"2a201fda-d325-4697-98e9-f477125ce0a0\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpxwxl" Oct 30 23:45:08.483199 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:08.483162 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/70668fb4-8523-4bf5-94ed-0e8440598b20-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wwjsn\" (UID: \"70668fb4-8523-4bf5-94ed-0e8440598b20\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wwjsn" Oct 30 23:45:08.483397 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:08.483219 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/70668fb4-8523-4bf5-94ed-0e8440598b20-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wwjsn\" (UID: \"70668fb4-8523-4bf5-94ed-0e8440598b20\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wwjsn" Oct 30 23:45:08.483397 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:08.483279 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhrj6\" (UniqueName: \"kubernetes.io/projected/70668fb4-8523-4bf5-94ed-0e8440598b20-kube-api-access-bhrj6\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wwjsn\" (UID: \"70668fb4-8523-4bf5-94ed-0e8440598b20\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wwjsn" Oct 30 23:45:08.483642 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:08.483609 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/70668fb4-8523-4bf5-94ed-0e8440598b20-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wwjsn\" (UID: \"70668fb4-8523-4bf5-94ed-0e8440598b20\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wwjsn" Oct 30 23:45:08.483689 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:08.483621 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/70668fb4-8523-4bf5-94ed-0e8440598b20-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wwjsn\" (UID: \"70668fb4-8523-4bf5-94ed-0e8440598b20\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wwjsn" Oct 30 23:45:08.538610 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:08.538581 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atc5bh" Oct 30 23:45:08.547583 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:08.547554 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bhrj6\" (UniqueName: \"kubernetes.io/projected/70668fb4-8523-4bf5-94ed-0e8440598b20-kube-api-access-bhrj6\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wwjsn\" (UID: \"70668fb4-8523-4bf5-94ed-0e8440598b20\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wwjsn" Oct 30 23:45:08.636655 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:08.636621 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7khvmq" Oct 30 23:45:08.734063 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:08.733676 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpxwxl" Oct 30 23:45:08.833762 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:08.833542 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wwjsn" Oct 30 23:45:08.927392 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:08.925802 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atc5bh"] Oct 30 23:45:08.938929 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:45:08.938895 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2cc429ca_1667_4683_a0e2_a8d5fdd42a64.slice/crio-c3045e3e50368afb446ed657ed1ddb0f47019f537a227eedd9eaa12c5f38e7ee WatchSource:0}: Error finding container c3045e3e50368afb446ed657ed1ddb0f47019f537a227eedd9eaa12c5f38e7ee: Status 404 returned error can't find the container with id c3045e3e50368afb446ed657ed1ddb0f47019f537a227eedd9eaa12c5f38e7ee Oct 30 23:45:09.034089 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:09.033976 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7khvmq"] Oct 30 23:45:09.051549 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:09.051517 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atc5bh" event={"ID":"2cc429ca-1667-4683-a0e2-a8d5fdd42a64","Type":"ContainerStarted","Data":"c3045e3e50368afb446ed657ed1ddb0f47019f537a227eedd9eaa12c5f38e7ee"} Oct 30 23:45:09.128496 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:09.128447 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpxwxl"] Oct 30 23:45:09.131072 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:45:09.131041 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a201fda_d325_4697_98e9_f477125ce0a0.slice/crio-23272f2af931a43d8e020512513c0c404afc4b1d56d0df755a11ce55a11886ee WatchSource:0}: Error finding container 23272f2af931a43d8e020512513c0c404afc4b1d56d0df755a11ce55a11886ee: Status 404 returned error can't find the container with id 23272f2af931a43d8e020512513c0c404afc4b1d56d0df755a11ce55a11886ee Oct 30 23:45:09.233173 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:09.233140 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wwjsn"] Oct 30 23:45:09.237361 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:45:09.237312 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70668fb4_8523_4bf5_94ed_0e8440598b20.slice/crio-0eaeadb670dc743c0c4619dd5f51dd72ccb0f68324f77214ffed69cf7f9c2367 WatchSource:0}: Error finding container 0eaeadb670dc743c0c4619dd5f51dd72ccb0f68324f77214ffed69cf7f9c2367: Status 404 returned error can't find the container with id 0eaeadb670dc743c0c4619dd5f51dd72ccb0f68324f77214ffed69cf7f9c2367 Oct 30 23:45:09.664675 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:09.664635 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644p5hlp"] Oct 30 23:45:09.670724 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:09.670693 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644p5hlp" Oct 30 23:45:09.725562 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:09.725531 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644p5hlp"] Oct 30 23:45:09.797286 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:09.797020 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-72n6x\" (UniqueName: \"kubernetes.io/projected/eee7dbc4-a47b-49a7-b521-9550a9f68162-kube-api-access-72n6x\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644p5hlp\" (UID: \"eee7dbc4-a47b-49a7-b521-9550a9f68162\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644p5hlp" Oct 30 23:45:09.797286 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:09.797115 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/eee7dbc4-a47b-49a7-b521-9550a9f68162-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644p5hlp\" (UID: \"eee7dbc4-a47b-49a7-b521-9550a9f68162\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644p5hlp" Oct 30 23:45:09.797286 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:09.797173 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/eee7dbc4-a47b-49a7-b521-9550a9f68162-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644p5hlp\" (UID: \"eee7dbc4-a47b-49a7-b521-9550a9f68162\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644p5hlp" Oct 30 23:45:09.897711 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:09.897669 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/eee7dbc4-a47b-49a7-b521-9550a9f68162-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644p5hlp\" (UID: \"eee7dbc4-a47b-49a7-b521-9550a9f68162\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644p5hlp" Oct 30 23:45:09.897858 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:09.897742 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/eee7dbc4-a47b-49a7-b521-9550a9f68162-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644p5hlp\" (UID: \"eee7dbc4-a47b-49a7-b521-9550a9f68162\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644p5hlp" Oct 30 23:45:09.897858 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:09.897833 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ccm6fd"] Oct 30 23:45:09.898177 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:09.898156 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/eee7dbc4-a47b-49a7-b521-9550a9f68162-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644p5hlp\" (UID: \"eee7dbc4-a47b-49a7-b521-9550a9f68162\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644p5hlp" Oct 30 23:45:09.898402 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:09.898380 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/eee7dbc4-a47b-49a7-b521-9550a9f68162-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644p5hlp\" (UID: \"eee7dbc4-a47b-49a7-b521-9550a9f68162\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644p5hlp" Oct 30 23:45:09.900342 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:09.899966 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72n6x\" (UniqueName: \"kubernetes.io/projected/eee7dbc4-a47b-49a7-b521-9550a9f68162-kube-api-access-72n6x\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644p5hlp\" (UID: \"eee7dbc4-a47b-49a7-b521-9550a9f68162\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644p5hlp" Oct 30 23:45:09.903091 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:09.902568 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ccm6fd" Oct 30 23:45:09.955909 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:09.955750 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ccm6fd"] Oct 30 23:45:09.972400 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:09.972314 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-72n6x\" (UniqueName: \"kubernetes.io/projected/eee7dbc4-a47b-49a7-b521-9550a9f68162-kube-api-access-72n6x\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644p5hlp\" (UID: \"eee7dbc4-a47b-49a7-b521-9550a9f68162\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644p5hlp" Oct 30 23:45:09.992105 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:09.991893 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644p5hlp" Oct 30 23:45:10.000889 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:10.000840 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8ec1fa33-555e-4ce8-9884-15d05bcb9b6c-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ccm6fd\" (UID: \"8ec1fa33-555e-4ce8-9884-15d05bcb9b6c\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ccm6fd" Oct 30 23:45:10.000889 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:10.000888 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kr2f5\" (UniqueName: \"kubernetes.io/projected/8ec1fa33-555e-4ce8-9884-15d05bcb9b6c-kube-api-access-kr2f5\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ccm6fd\" (UID: \"8ec1fa33-555e-4ce8-9884-15d05bcb9b6c\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ccm6fd" Oct 30 23:45:10.001092 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:10.000935 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8ec1fa33-555e-4ce8-9884-15d05bcb9b6c-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ccm6fd\" (UID: \"8ec1fa33-555e-4ce8-9884-15d05bcb9b6c\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ccm6fd" Oct 30 23:45:10.066373 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:10.066271 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7khvmq" event={"ID":"bf56ecae-e59d-4283-86fe-c32458951e5a","Type":"ContainerStarted","Data":"a8fd9e1ec1b47ae9887fac96c834768087734877728b30069f42cdcf7b3ab2c5"} Oct 30 23:45:10.069106 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:10.068666 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wwjsn" event={"ID":"70668fb4-8523-4bf5-94ed-0e8440598b20","Type":"ContainerStarted","Data":"0eaeadb670dc743c0c4619dd5f51dd72ccb0f68324f77214ffed69cf7f9c2367"} Oct 30 23:45:10.072263 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:10.072197 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpxwxl" event={"ID":"2a201fda-d325-4697-98e9-f477125ce0a0","Type":"ContainerStarted","Data":"23272f2af931a43d8e020512513c0c404afc4b1d56d0df755a11ce55a11886ee"} Oct 30 23:45:10.103062 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:10.101845 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8ec1fa33-555e-4ce8-9884-15d05bcb9b6c-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ccm6fd\" (UID: \"8ec1fa33-555e-4ce8-9884-15d05bcb9b6c\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ccm6fd" Oct 30 23:45:10.103062 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:10.101908 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kr2f5\" (UniqueName: \"kubernetes.io/projected/8ec1fa33-555e-4ce8-9884-15d05bcb9b6c-kube-api-access-kr2f5\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ccm6fd\" (UID: \"8ec1fa33-555e-4ce8-9884-15d05bcb9b6c\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ccm6fd" Oct 30 23:45:10.103062 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:10.101969 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8ec1fa33-555e-4ce8-9884-15d05bcb9b6c-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ccm6fd\" (UID: \"8ec1fa33-555e-4ce8-9884-15d05bcb9b6c\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ccm6fd" Oct 30 23:45:10.103062 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:10.102714 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8ec1fa33-555e-4ce8-9884-15d05bcb9b6c-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ccm6fd\" (UID: \"8ec1fa33-555e-4ce8-9884-15d05bcb9b6c\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ccm6fd" Oct 30 23:45:10.103560 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:10.103516 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8ec1fa33-555e-4ce8-9884-15d05bcb9b6c-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ccm6fd\" (UID: \"8ec1fa33-555e-4ce8-9884-15d05bcb9b6c\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ccm6fd" Oct 30 23:45:10.173358 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:10.172051 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kr2f5\" (UniqueName: \"kubernetes.io/projected/8ec1fa33-555e-4ce8-9884-15d05bcb9b6c-kube-api-access-kr2f5\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ccm6fd\" (UID: \"8ec1fa33-555e-4ce8-9884-15d05bcb9b6c\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ccm6fd" Oct 30 23:45:10.225274 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:10.225191 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ccm6fd" Oct 30 23:45:10.392676 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:10.392647 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644p5hlp"] Oct 30 23:45:10.657699 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:10.657666 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ccm6fd"] Oct 30 23:45:10.669160 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:45:10.669131 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ec1fa33_555e_4ce8_9884_15d05bcb9b6c.slice/crio-e47c5221c84477abee98c021633d694e57ec31dd99c75f201b2f349f5ff5876c WatchSource:0}: Error finding container e47c5221c84477abee98c021633d694e57ec31dd99c75f201b2f349f5ff5876c: Status 404 returned error can't find the container with id e47c5221c84477abee98c021633d694e57ec31dd99c75f201b2f349f5ff5876c Oct 30 23:45:10.753283 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:10.753249 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:45:10.953437 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:10.953353 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:45:11.089123 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:11.089079 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ccm6fd" event={"ID":"8ec1fa33-555e-4ce8-9884-15d05bcb9b6c","Type":"ContainerStarted","Data":"e47c5221c84477abee98c021633d694e57ec31dd99c75f201b2f349f5ff5876c"} Oct 30 23:45:11.089747 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:11.089719 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644p5hlp" event={"ID":"eee7dbc4-a47b-49a7-b521-9550a9f68162","Type":"ContainerStarted","Data":"faee1bb0d8733f88a89815d64d22b72a4e09c285a013713274871dbc6710619a"} Oct 30 23:45:12.149735 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:12.149693 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-pnm97"] Oct 30 23:45:12.153959 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:12.153930 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-pnm97" Oct 30 23:45:12.207903 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:12.207871 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-b8qjx\"" Oct 30 23:45:12.220267 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:12.219513 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/495da7b4-1e63-46fa-ac83-6d3bb75692f8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8f6f87f59-pnm97\" (UID: \"495da7b4-1e63-46fa-ac83-6d3bb75692f8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-pnm97" Oct 30 23:45:12.220267 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:12.219567 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/495da7b4-1e63-46fa-ac83-6d3bb75692f8-workload-certs\") pod \"maas-default-gateway-openshift-default-8f6f87f59-pnm97\" (UID: \"495da7b4-1e63-46fa-ac83-6d3bb75692f8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-pnm97" Oct 30 23:45:12.220267 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:12.219611 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/495da7b4-1e63-46fa-ac83-6d3bb75692f8-credential-socket\") pod \"maas-default-gateway-openshift-default-8f6f87f59-pnm97\" (UID: \"495da7b4-1e63-46fa-ac83-6d3bb75692f8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-pnm97" Oct 30 23:45:12.220267 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:12.219643 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/495da7b4-1e63-46fa-ac83-6d3bb75692f8-istio-token\") pod \"maas-default-gateway-openshift-default-8f6f87f59-pnm97\" (UID: \"495da7b4-1e63-46fa-ac83-6d3bb75692f8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-pnm97" Oct 30 23:45:12.220267 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:12.219674 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/495da7b4-1e63-46fa-ac83-6d3bb75692f8-istio-data\") pod \"maas-default-gateway-openshift-default-8f6f87f59-pnm97\" (UID: \"495da7b4-1e63-46fa-ac83-6d3bb75692f8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-pnm97" Oct 30 23:45:12.220267 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:12.219704 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/495da7b4-1e63-46fa-ac83-6d3bb75692f8-workload-socket\") pod \"maas-default-gateway-openshift-default-8f6f87f59-pnm97\" (UID: \"495da7b4-1e63-46fa-ac83-6d3bb75692f8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-pnm97" Oct 30 23:45:12.220267 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:12.219731 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/495da7b4-1e63-46fa-ac83-6d3bb75692f8-istio-envoy\") pod \"maas-default-gateway-openshift-default-8f6f87f59-pnm97\" (UID: \"495da7b4-1e63-46fa-ac83-6d3bb75692f8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-pnm97" Oct 30 23:45:12.220267 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:12.219773 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/495da7b4-1e63-46fa-ac83-6d3bb75692f8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8f6f87f59-pnm97\" (UID: \"495da7b4-1e63-46fa-ac83-6d3bb75692f8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-pnm97" Oct 30 23:45:12.220267 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:12.219842 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhhh8\" (UniqueName: \"kubernetes.io/projected/495da7b4-1e63-46fa-ac83-6d3bb75692f8-kube-api-access-hhhh8\") pod \"maas-default-gateway-openshift-default-8f6f87f59-pnm97\" (UID: \"495da7b4-1e63-46fa-ac83-6d3bb75692f8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-pnm97" Oct 30 23:45:12.225744 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:12.225663 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Oct 30 23:45:12.254354 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:12.254275 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:45:12.260768 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:12.260684 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-pnm97"] Oct 30 23:45:12.321364 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:12.320488 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhhh8\" (UniqueName: \"kubernetes.io/projected/495da7b4-1e63-46fa-ac83-6d3bb75692f8-kube-api-access-hhhh8\") pod \"maas-default-gateway-openshift-default-8f6f87f59-pnm97\" (UID: \"495da7b4-1e63-46fa-ac83-6d3bb75692f8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-pnm97" Oct 30 23:45:12.321364 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:12.320528 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/495da7b4-1e63-46fa-ac83-6d3bb75692f8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8f6f87f59-pnm97\" (UID: \"495da7b4-1e63-46fa-ac83-6d3bb75692f8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-pnm97" Oct 30 23:45:12.321364 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:12.320552 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/495da7b4-1e63-46fa-ac83-6d3bb75692f8-workload-certs\") pod \"maas-default-gateway-openshift-default-8f6f87f59-pnm97\" (UID: \"495da7b4-1e63-46fa-ac83-6d3bb75692f8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-pnm97" Oct 30 23:45:12.321364 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:12.320583 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/495da7b4-1e63-46fa-ac83-6d3bb75692f8-credential-socket\") pod \"maas-default-gateway-openshift-default-8f6f87f59-pnm97\" (UID: \"495da7b4-1e63-46fa-ac83-6d3bb75692f8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-pnm97" Oct 30 23:45:12.321364 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:12.320598 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/495da7b4-1e63-46fa-ac83-6d3bb75692f8-istio-token\") pod \"maas-default-gateway-openshift-default-8f6f87f59-pnm97\" (UID: \"495da7b4-1e63-46fa-ac83-6d3bb75692f8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-pnm97" Oct 30 23:45:12.321364 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:12.320619 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/495da7b4-1e63-46fa-ac83-6d3bb75692f8-istio-data\") pod \"maas-default-gateway-openshift-default-8f6f87f59-pnm97\" (UID: \"495da7b4-1e63-46fa-ac83-6d3bb75692f8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-pnm97" Oct 30 23:45:12.321364 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:12.320637 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/495da7b4-1e63-46fa-ac83-6d3bb75692f8-workload-socket\") pod \"maas-default-gateway-openshift-default-8f6f87f59-pnm97\" (UID: \"495da7b4-1e63-46fa-ac83-6d3bb75692f8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-pnm97" Oct 30 23:45:12.321364 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:12.320655 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/495da7b4-1e63-46fa-ac83-6d3bb75692f8-istio-envoy\") pod \"maas-default-gateway-openshift-default-8f6f87f59-pnm97\" (UID: \"495da7b4-1e63-46fa-ac83-6d3bb75692f8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-pnm97" Oct 30 23:45:12.321364 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:12.320692 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/495da7b4-1e63-46fa-ac83-6d3bb75692f8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8f6f87f59-pnm97\" (UID: \"495da7b4-1e63-46fa-ac83-6d3bb75692f8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-pnm97" Oct 30 23:45:12.321364 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:12.321371 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/495da7b4-1e63-46fa-ac83-6d3bb75692f8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8f6f87f59-pnm97\" (UID: \"495da7b4-1e63-46fa-ac83-6d3bb75692f8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-pnm97" Oct 30 23:45:12.323969 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:12.322774 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/495da7b4-1e63-46fa-ac83-6d3bb75692f8-workload-certs\") pod \"maas-default-gateway-openshift-default-8f6f87f59-pnm97\" (UID: \"495da7b4-1e63-46fa-ac83-6d3bb75692f8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-pnm97" Oct 30 23:45:12.323969 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:12.323052 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/495da7b4-1e63-46fa-ac83-6d3bb75692f8-credential-socket\") pod \"maas-default-gateway-openshift-default-8f6f87f59-pnm97\" (UID: \"495da7b4-1e63-46fa-ac83-6d3bb75692f8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-pnm97" Oct 30 23:45:12.323969 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:12.323279 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/495da7b4-1e63-46fa-ac83-6d3bb75692f8-workload-socket\") pod \"maas-default-gateway-openshift-default-8f6f87f59-pnm97\" (UID: \"495da7b4-1e63-46fa-ac83-6d3bb75692f8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-pnm97" Oct 30 23:45:12.323969 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:12.323544 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/495da7b4-1e63-46fa-ac83-6d3bb75692f8-istio-data\") pod \"maas-default-gateway-openshift-default-8f6f87f59-pnm97\" (UID: \"495da7b4-1e63-46fa-ac83-6d3bb75692f8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-pnm97" Oct 30 23:45:12.325495 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:12.324915 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/495da7b4-1e63-46fa-ac83-6d3bb75692f8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8f6f87f59-pnm97\" (UID: \"495da7b4-1e63-46fa-ac83-6d3bb75692f8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-pnm97" Oct 30 23:45:12.336515 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:12.336490 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/495da7b4-1e63-46fa-ac83-6d3bb75692f8-istio-envoy\") pod \"maas-default-gateway-openshift-default-8f6f87f59-pnm97\" (UID: \"495da7b4-1e63-46fa-ac83-6d3bb75692f8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-pnm97" Oct 30 23:45:12.397184 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:12.397117 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhhh8\" (UniqueName: \"kubernetes.io/projected/495da7b4-1e63-46fa-ac83-6d3bb75692f8-kube-api-access-hhhh8\") pod \"maas-default-gateway-openshift-default-8f6f87f59-pnm97\" (UID: \"495da7b4-1e63-46fa-ac83-6d3bb75692f8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-pnm97" Oct 30 23:45:12.400706 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:12.400641 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/495da7b4-1e63-46fa-ac83-6d3bb75692f8-istio-token\") pod \"maas-default-gateway-openshift-default-8f6f87f59-pnm97\" (UID: \"495da7b4-1e63-46fa-ac83-6d3bb75692f8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-pnm97" Oct 30 23:45:12.445521 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:12.445485 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-7464bdcdc9-dmjmb"] Oct 30 23:45:12.451024 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:12.450997 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7464bdcdc9-dmjmb" Oct 30 23:45:12.455622 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:12.455504 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:45:12.474389 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:12.474362 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-pnm97" Oct 30 23:45:12.508292 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:12.507596 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-7464bdcdc9-dmjmb"] Oct 30 23:45:12.508292 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:12.507824 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-ai-inference-dockercfg-c5w4t\"" Oct 30 23:45:12.608855 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:12.608804 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:45:12.623878 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:12.623115 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/169707c0-e24a-489d-b56c-5f657b2a1dee-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-7464bdcdc9-dmjmb\" (UID: \"169707c0-e24a-489d-b56c-5f657b2a1dee\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7464bdcdc9-dmjmb" Oct 30 23:45:12.623878 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:12.623174 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45jwh\" (UniqueName: \"kubernetes.io/projected/169707c0-e24a-489d-b56c-5f657b2a1dee-kube-api-access-45jwh\") pod \"openshift-ai-inference-openshift-ai-inference-7464bdcdc9-dmjmb\" (UID: \"169707c0-e24a-489d-b56c-5f657b2a1dee\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7464bdcdc9-dmjmb" Oct 30 23:45:12.623878 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:12.623231 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/169707c0-e24a-489d-b56c-5f657b2a1dee-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-7464bdcdc9-dmjmb\" (UID: \"169707c0-e24a-489d-b56c-5f657b2a1dee\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7464bdcdc9-dmjmb" Oct 30 23:45:12.623878 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:12.623258 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/169707c0-e24a-489d-b56c-5f657b2a1dee-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-7464bdcdc9-dmjmb\" (UID: \"169707c0-e24a-489d-b56c-5f657b2a1dee\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7464bdcdc9-dmjmb" Oct 30 23:45:12.623878 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:12.623293 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/169707c0-e24a-489d-b56c-5f657b2a1dee-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-7464bdcdc9-dmjmb\" (UID: \"169707c0-e24a-489d-b56c-5f657b2a1dee\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7464bdcdc9-dmjmb" Oct 30 23:45:12.623878 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:12.623351 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/169707c0-e24a-489d-b56c-5f657b2a1dee-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-7464bdcdc9-dmjmb\" (UID: \"169707c0-e24a-489d-b56c-5f657b2a1dee\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7464bdcdc9-dmjmb" Oct 30 23:45:12.623878 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:12.623397 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/169707c0-e24a-489d-b56c-5f657b2a1dee-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-7464bdcdc9-dmjmb\" (UID: \"169707c0-e24a-489d-b56c-5f657b2a1dee\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7464bdcdc9-dmjmb" Oct 30 23:45:12.623878 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:12.623437 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/169707c0-e24a-489d-b56c-5f657b2a1dee-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-7464bdcdc9-dmjmb\" (UID: \"169707c0-e24a-489d-b56c-5f657b2a1dee\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7464bdcdc9-dmjmb" Oct 30 23:45:12.623878 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:12.623497 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/169707c0-e24a-489d-b56c-5f657b2a1dee-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-7464bdcdc9-dmjmb\" (UID: \"169707c0-e24a-489d-b56c-5f657b2a1dee\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7464bdcdc9-dmjmb" Oct 30 23:45:12.729253 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:12.724901 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/169707c0-e24a-489d-b56c-5f657b2a1dee-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-7464bdcdc9-dmjmb\" (UID: \"169707c0-e24a-489d-b56c-5f657b2a1dee\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7464bdcdc9-dmjmb" Oct 30 23:45:12.729253 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:12.724955 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/169707c0-e24a-489d-b56c-5f657b2a1dee-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-7464bdcdc9-dmjmb\" (UID: \"169707c0-e24a-489d-b56c-5f657b2a1dee\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7464bdcdc9-dmjmb" Oct 30 23:45:12.729253 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:12.724985 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/169707c0-e24a-489d-b56c-5f657b2a1dee-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-7464bdcdc9-dmjmb\" (UID: \"169707c0-e24a-489d-b56c-5f657b2a1dee\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7464bdcdc9-dmjmb" Oct 30 23:45:12.729253 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:12.725026 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/169707c0-e24a-489d-b56c-5f657b2a1dee-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-7464bdcdc9-dmjmb\" (UID: \"169707c0-e24a-489d-b56c-5f657b2a1dee\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7464bdcdc9-dmjmb" Oct 30 23:45:12.729253 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:12.725068 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/169707c0-e24a-489d-b56c-5f657b2a1dee-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-7464bdcdc9-dmjmb\" (UID: \"169707c0-e24a-489d-b56c-5f657b2a1dee\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7464bdcdc9-dmjmb" Oct 30 23:45:12.729253 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:12.725113 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/169707c0-e24a-489d-b56c-5f657b2a1dee-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-7464bdcdc9-dmjmb\" (UID: \"169707c0-e24a-489d-b56c-5f657b2a1dee\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7464bdcdc9-dmjmb" Oct 30 23:45:12.729253 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:12.725166 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/169707c0-e24a-489d-b56c-5f657b2a1dee-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-7464bdcdc9-dmjmb\" (UID: \"169707c0-e24a-489d-b56c-5f657b2a1dee\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7464bdcdc9-dmjmb" Oct 30 23:45:12.729253 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:12.725260 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/169707c0-e24a-489d-b56c-5f657b2a1dee-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-7464bdcdc9-dmjmb\" (UID: \"169707c0-e24a-489d-b56c-5f657b2a1dee\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7464bdcdc9-dmjmb" Oct 30 23:45:12.729253 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:12.725296 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45jwh\" (UniqueName: \"kubernetes.io/projected/169707c0-e24a-489d-b56c-5f657b2a1dee-kube-api-access-45jwh\") pod \"openshift-ai-inference-openshift-ai-inference-7464bdcdc9-dmjmb\" (UID: \"169707c0-e24a-489d-b56c-5f657b2a1dee\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7464bdcdc9-dmjmb" Oct 30 23:45:12.729253 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:12.726048 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/169707c0-e24a-489d-b56c-5f657b2a1dee-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-7464bdcdc9-dmjmb\" (UID: \"169707c0-e24a-489d-b56c-5f657b2a1dee\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7464bdcdc9-dmjmb" Oct 30 23:45:12.729253 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:12.726804 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/169707c0-e24a-489d-b56c-5f657b2a1dee-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-7464bdcdc9-dmjmb\" (UID: \"169707c0-e24a-489d-b56c-5f657b2a1dee\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7464bdcdc9-dmjmb" Oct 30 23:45:12.729253 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:12.727841 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/169707c0-e24a-489d-b56c-5f657b2a1dee-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-7464bdcdc9-dmjmb\" (UID: \"169707c0-e24a-489d-b56c-5f657b2a1dee\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7464bdcdc9-dmjmb" Oct 30 23:45:12.729253 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:12.728816 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/169707c0-e24a-489d-b56c-5f657b2a1dee-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-7464bdcdc9-dmjmb\" (UID: \"169707c0-e24a-489d-b56c-5f657b2a1dee\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7464bdcdc9-dmjmb" Oct 30 23:45:12.730654 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:12.730253 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/169707c0-e24a-489d-b56c-5f657b2a1dee-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-7464bdcdc9-dmjmb\" (UID: \"169707c0-e24a-489d-b56c-5f657b2a1dee\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7464bdcdc9-dmjmb" Oct 30 23:45:12.734958 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:12.734930 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/169707c0-e24a-489d-b56c-5f657b2a1dee-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-7464bdcdc9-dmjmb\" (UID: \"169707c0-e24a-489d-b56c-5f657b2a1dee\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7464bdcdc9-dmjmb" Oct 30 23:45:12.749989 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:12.749890 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/169707c0-e24a-489d-b56c-5f657b2a1dee-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-7464bdcdc9-dmjmb\" (UID: \"169707c0-e24a-489d-b56c-5f657b2a1dee\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7464bdcdc9-dmjmb" Oct 30 23:45:12.796568 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:12.796516 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/169707c0-e24a-489d-b56c-5f657b2a1dee-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-7464bdcdc9-dmjmb\" (UID: \"169707c0-e24a-489d-b56c-5f657b2a1dee\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7464bdcdc9-dmjmb" Oct 30 23:45:12.805074 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:12.805008 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-45jwh\" (UniqueName: \"kubernetes.io/projected/169707c0-e24a-489d-b56c-5f657b2a1dee-kube-api-access-45jwh\") pod \"openshift-ai-inference-openshift-ai-inference-7464bdcdc9-dmjmb\" (UID: \"169707c0-e24a-489d-b56c-5f657b2a1dee\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7464bdcdc9-dmjmb" Oct 30 23:45:12.997010 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:12.996876 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-pnm97"] Oct 30 23:45:13.070216 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:13.070180 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7464bdcdc9-dmjmb" Oct 30 23:45:13.123413 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:13.123369 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-pnm97" event={"ID":"495da7b4-1e63-46fa-ac83-6d3bb75692f8","Type":"ContainerStarted","Data":"633dc6e2b09285bc2bc9eceedccedacb5d5df5e2a3ada3112e76e39670b63ad9"} Oct 30 23:45:13.578443 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:13.577880 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-7464bdcdc9-dmjmb"] Oct 30 23:45:13.593865 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:45:13.593806 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod169707c0_e24a_489d_b56c_5f657b2a1dee.slice/crio-405396b28547f16084a22dd0ad3f177654c705ec7204c04d25f7aa86dff461b1 WatchSource:0}: Error finding container 405396b28547f16084a22dd0ad3f177654c705ec7204c04d25f7aa86dff461b1: Status 404 returned error can't find the container with id 405396b28547f16084a22dd0ad3f177654c705ec7204c04d25f7aa86dff461b1 Oct 30 23:45:14.152370 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:14.151508 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7464bdcdc9-dmjmb" event={"ID":"169707c0-e24a-489d-b56c-5f657b2a1dee","Type":"ContainerStarted","Data":"405396b28547f16084a22dd0ad3f177654c705ec7204c04d25f7aa86dff461b1"} Oct 30 23:45:16.402519 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:16.402463 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:45:18.219817 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:18.219755 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:45:18.387253 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:18.387136 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:45:19.700864 ip-10-0-141-180 kubenswrapper[4927]: E1030 23:45:19.700825 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sqv69" containerName="extractor" Oct 30 23:45:20.263165 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:20.263093 4927 generic.go:356] "Generic (PLEG): container finished" podID="bb30f2f7-5143-4b00-84c4-b691cc0b3da5" containerID="05bad7cf9162120695008a0b17cb3088bb380c82b5b8f9abb3579ab8a2592f00" exitCode=0 Oct 30 23:45:20.263380 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:20.263266 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448rs44" event={"ID":"bb30f2f7-5143-4b00-84c4-b691cc0b3da5","Type":"ContainerDied","Data":"05bad7cf9162120695008a0b17cb3088bb380c82b5b8f9abb3579ab8a2592f00"} Oct 30 23:45:21.281355 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:21.281306 4927 generic.go:356] "Generic (PLEG): container finished" podID="70668fb4-8523-4bf5-94ed-0e8440598b20" containerID="5cdb47a3a8237f925ce00d0584b1a49075b99fb7db79bc1645575202b94985a5" exitCode=0 Oct 30 23:45:21.281813 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:21.281448 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wwjsn" event={"ID":"70668fb4-8523-4bf5-94ed-0e8440598b20","Type":"ContainerDied","Data":"5cdb47a3a8237f925ce00d0584b1a49075b99fb7db79bc1645575202b94985a5"} Oct 30 23:45:21.284986 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:21.284958 4927 generic.go:356] "Generic (PLEG): container finished" podID="2a201fda-d325-4697-98e9-f477125ce0a0" containerID="e97648d8af05ff0640f222a5f8da4108dad81fa3527a2d1994ffa70199a73786" exitCode=0 Oct 30 23:45:21.285111 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:21.285001 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpxwxl" event={"ID":"2a201fda-d325-4697-98e9-f477125ce0a0","Type":"ContainerDied","Data":"e97648d8af05ff0640f222a5f8da4108dad81fa3527a2d1994ffa70199a73786"} Oct 30 23:45:22.294945 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:22.294172 4927 generic.go:356] "Generic (PLEG): container finished" podID="604b4e17-0542-42e7-8c4b-82b0700f781d" containerID="bfd577c9fea18a1b12ae3c814eb97ca1e4a7b6c1d1baaa4d7debed35e9ba1c6f" exitCode=0 Oct 30 23:45:22.294945 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:22.294249 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cwms65" event={"ID":"604b4e17-0542-42e7-8c4b-82b0700f781d","Type":"ContainerDied","Data":"bfd577c9fea18a1b12ae3c814eb97ca1e4a7b6c1d1baaa4d7debed35e9ba1c6f"} Oct 30 23:45:22.300506 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:22.300478 4927 generic.go:356] "Generic (PLEG): container finished" podID="2cc429ca-1667-4683-a0e2-a8d5fdd42a64" containerID="b54497176a67722c13511d1d92af04ed6d28bbff59264c2458245e65dc5c0cea" exitCode=0 Oct 30 23:45:22.300674 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:22.300578 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atc5bh" event={"ID":"2cc429ca-1667-4683-a0e2-a8d5fdd42a64","Type":"ContainerDied","Data":"b54497176a67722c13511d1d92af04ed6d28bbff59264c2458245e65dc5c0cea"} Oct 30 23:45:22.304638 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:22.304607 4927 generic.go:356] "Generic (PLEG): container finished" podID="bf56ecae-e59d-4283-86fe-c32458951e5a" containerID="3b2e7ef009e59f3b3e65383c7d02a58af65031a37a0262557cf880f2119049f9" exitCode=0 Oct 30 23:45:22.304729 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:22.304700 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7khvmq" event={"ID":"bf56ecae-e59d-4283-86fe-c32458951e5a","Type":"ContainerDied","Data":"3b2e7ef009e59f3b3e65383c7d02a58af65031a37a0262557cf880f2119049f9"} Oct 30 23:45:22.605823 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:22.605785 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 23:45:22.605958 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:22.605871 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 23:45:22.605958 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:22.605914 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 23:45:22.722872 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:22.722833 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 23:45:22.722984 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:22.722924 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 23:45:22.722984 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:22.722971 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 23:45:23.312520 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:23.312484 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7464bdcdc9-dmjmb" event={"ID":"169707c0-e24a-489d-b56c-5f657b2a1dee","Type":"ContainerStarted","Data":"59fa466593141e59f4f1dac90d1ef93bbfb3f14ef446ce358e4cdf8603c8d2ae"} Oct 30 23:45:23.314272 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:23.314158 4927 generic.go:356] "Generic (PLEG): container finished" podID="eee7dbc4-a47b-49a7-b521-9550a9f68162" containerID="954c06775c6a840ee3a2bfd86055996af1e346a7f678b5a390a6a63f43497caa" exitCode=0 Oct 30 23:45:23.314463 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:23.314409 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644p5hlp" event={"ID":"eee7dbc4-a47b-49a7-b521-9550a9f68162","Type":"ContainerDied","Data":"954c06775c6a840ee3a2bfd86055996af1e346a7f678b5a390a6a63f43497caa"} Oct 30 23:45:23.316686 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:23.316659 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-pnm97" event={"ID":"495da7b4-1e63-46fa-ac83-6d3bb75692f8","Type":"ContainerStarted","Data":"3c0baafe8176b8985b703751e8d2118f0442fe67f81363c6f1c35c195218077d"} Oct 30 23:45:23.318804 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:23.318777 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpxwxl" event={"ID":"2a201fda-d325-4697-98e9-f477125ce0a0","Type":"ContainerStarted","Data":"2ddc8095656a17d6a81696613359a6655b3f7b7fc6bdea1c731ae5189dfd7730"} Oct 30 23:45:23.320467 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:23.320440 4927 generic.go:356] "Generic (PLEG): container finished" podID="8ec1fa33-555e-4ce8-9884-15d05bcb9b6c" containerID="d2bb29377fc4f4caee6bd6c2972c6232878892100502845126c7f2dca5fcadc7" exitCode=0 Oct 30 23:45:23.320573 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:23.320509 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ccm6fd" event={"ID":"8ec1fa33-555e-4ce8-9884-15d05bcb9b6c","Type":"ContainerDied","Data":"d2bb29377fc4f4caee6bd6c2972c6232878892100502845126c7f2dca5fcadc7"} Oct 30 23:45:23.435838 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:23.435786 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7464bdcdc9-dmjmb" podStartSLOduration=2.310992117 podStartE2EDuration="11.435773182s" totalImagesPullingTime="9.124781065s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:45:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:45:23.435252984 +0000 UTC m=+236.471583809" watchObservedRunningTime="2026-10-30 23:45:23.435773182 +0000 UTC m=+236.472104004" Oct 30 23:45:23.475024 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:23.474940 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-pnm97" Oct 30 23:45:23.480250 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:23.480221 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-pnm97" Oct 30 23:45:23.561189 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:23.561137 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-pnm97" podStartSLOduration=1.967235981 podStartE2EDuration="11.561123343s" totalImagesPullingTime="9.593887362s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:45:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:45:23.558826409 +0000 UTC m=+236.595157237" watchObservedRunningTime="2026-10-30 23:45:23.561123343 +0000 UTC m=+236.597454164" Oct 30 23:45:24.071242 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:24.071159 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7464bdcdc9-dmjmb" Oct 30 23:45:24.076499 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:24.076473 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7464bdcdc9-dmjmb" Oct 30 23:45:24.327797 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:24.327725 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atc5bh" event={"ID":"2cc429ca-1667-4683-a0e2-a8d5fdd42a64","Type":"ContainerStarted","Data":"38b56d87a7c3a58cb2f1d6b41c70daf5f2e86e0310dcb0e9cadf736944c8e206"} Oct 30 23:45:24.330486 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:24.330458 4927 generic.go:356] "Generic (PLEG): container finished" podID="70668fb4-8523-4bf5-94ed-0e8440598b20" containerID="0eb5386bc1d316d4b656b0f4ee84ec03de32837cbb8b3c7e91ec4c266aef4353" exitCode=0 Oct 30 23:45:24.330612 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:24.330538 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wwjsn" event={"ID":"70668fb4-8523-4bf5-94ed-0e8440598b20","Type":"ContainerDied","Data":"0eb5386bc1d316d4b656b0f4ee84ec03de32837cbb8b3c7e91ec4c266aef4353"} Oct 30 23:45:24.333040 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:24.333019 4927 generic.go:356] "Generic (PLEG): container finished" podID="525fd99a-040b-43c8-bbb3-41a0f953ea4a" containerID="ff912333a770ac0f90761d4c3c897d58d59d986e462410c76c1555164a338904" exitCode=0 Oct 30 23:45:24.333152 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:24.333097 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be719c6lt" event={"ID":"525fd99a-040b-43c8-bbb3-41a0f953ea4a","Type":"ContainerDied","Data":"ff912333a770ac0f90761d4c3c897d58d59d986e462410c76c1555164a338904"} Oct 30 23:45:24.336294 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:24.336269 4927 generic.go:356] "Generic (PLEG): container finished" podID="2a201fda-d325-4697-98e9-f477125ce0a0" containerID="2ddc8095656a17d6a81696613359a6655b3f7b7fc6bdea1c731ae5189dfd7730" exitCode=0 Oct 30 23:45:24.336403 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:24.336382 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpxwxl" event={"ID":"2a201fda-d325-4697-98e9-f477125ce0a0","Type":"ContainerDied","Data":"2ddc8095656a17d6a81696613359a6655b3f7b7fc6bdea1c731ae5189dfd7730"} Oct 30 23:45:24.336905 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:24.336796 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-pnm97" Oct 30 23:45:24.336905 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:24.336840 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7464bdcdc9-dmjmb" Oct 30 23:45:24.338045 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:24.337864 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7464bdcdc9-dmjmb" Oct 30 23:45:24.338045 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:24.338012 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-pnm97" Oct 30 23:45:25.345024 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:25.344982 4927 generic.go:356] "Generic (PLEG): container finished" podID="eee7dbc4-a47b-49a7-b521-9550a9f68162" containerID="862fbb06df6a973fe6d93062f0266cdd1b3dc21e9665ab09912a2f3bce8838d2" exitCode=0 Oct 30 23:45:25.345415 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:25.345131 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644p5hlp" event={"ID":"eee7dbc4-a47b-49a7-b521-9550a9f68162","Type":"ContainerDied","Data":"862fbb06df6a973fe6d93062f0266cdd1b3dc21e9665ab09912a2f3bce8838d2"} Oct 30 23:45:25.360309 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:25.358495 4927 generic.go:356] "Generic (PLEG): container finished" podID="bb30f2f7-5143-4b00-84c4-b691cc0b3da5" containerID="20f222382c1a3dc5aaf8ea2399fed4e1d7843bde117d693d75f06facca248769" exitCode=0 Oct 30 23:45:25.360309 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:25.359572 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448rs44" event={"ID":"bb30f2f7-5143-4b00-84c4-b691cc0b3da5","Type":"ContainerDied","Data":"20f222382c1a3dc5aaf8ea2399fed4e1d7843bde117d693d75f06facca248769"} Oct 30 23:45:25.363821 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:25.363685 4927 generic.go:356] "Generic (PLEG): container finished" podID="2cc429ca-1667-4683-a0e2-a8d5fdd42a64" containerID="38b56d87a7c3a58cb2f1d6b41c70daf5f2e86e0310dcb0e9cadf736944c8e206" exitCode=0 Oct 30 23:45:25.363821 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:25.363793 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atc5bh" event={"ID":"2cc429ca-1667-4683-a0e2-a8d5fdd42a64","Type":"ContainerDied","Data":"38b56d87a7c3a58cb2f1d6b41c70daf5f2e86e0310dcb0e9cadf736944c8e206"} Oct 30 23:45:25.368785 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:25.368714 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7khvmq" event={"ID":"bf56ecae-e59d-4283-86fe-c32458951e5a","Type":"ContainerDied","Data":"0465fd3eb1ade17d53e659d17f129d43f80209b8bffe8fe78bca0a6de64a79d7"} Oct 30 23:45:25.369629 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:25.368293 4927 generic.go:356] "Generic (PLEG): container finished" podID="bf56ecae-e59d-4283-86fe-c32458951e5a" containerID="0465fd3eb1ade17d53e659d17f129d43f80209b8bffe8fe78bca0a6de64a79d7" exitCode=0 Oct 30 23:45:26.380522 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:26.380474 4927 generic.go:356] "Generic (PLEG): container finished" podID="525fd99a-040b-43c8-bbb3-41a0f953ea4a" containerID="bffae72f34c44adc077806746b1886d647a5fcd0b62f096af055fe9dde1d6c2c" exitCode=0 Oct 30 23:45:26.381498 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:26.381402 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be719c6lt" event={"ID":"525fd99a-040b-43c8-bbb3-41a0f953ea4a","Type":"ContainerDied","Data":"bffae72f34c44adc077806746b1886d647a5fcd0b62f096af055fe9dde1d6c2c"} Oct 30 23:45:27.394743 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:27.393720 4927 generic.go:356] "Generic (PLEG): container finished" podID="604b4e17-0542-42e7-8c4b-82b0700f781d" containerID="462f96709adfa159882a94d2da6136d3b5437f9043c8015f7364db3b655c7dee" exitCode=0 Oct 30 23:45:27.394743 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:27.393833 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cwms65" event={"ID":"604b4e17-0542-42e7-8c4b-82b0700f781d","Type":"ContainerDied","Data":"462f96709adfa159882a94d2da6136d3b5437f9043c8015f7364db3b655c7dee"} Oct 30 23:45:28.407788 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:28.407731 4927 generic.go:356] "Generic (PLEG): container finished" podID="8ec1fa33-555e-4ce8-9884-15d05bcb9b6c" containerID="4c016e51c0962e4ab885ee58deae9ea1097a61bdfb1a2d1063976770d23317e7" exitCode=0 Oct 30 23:45:28.408257 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:28.407860 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ccm6fd" event={"ID":"8ec1fa33-555e-4ce8-9884-15d05bcb9b6c","Type":"ContainerDied","Data":"4c016e51c0962e4ab885ee58deae9ea1097a61bdfb1a2d1063976770d23317e7"} Oct 30 23:45:32.995538 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:32.995489 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:45:33.187492 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:33.187305 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:45:40.567187 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:40.567097 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpxwxl" event={"ID":"2a201fda-d325-4697-98e9-f477125ce0a0","Type":"ContainerStarted","Data":"e5f098c1bc291542eb44cfcf5a68ec9fc0704c4726e7dc6cdd696a93990235ec"} Oct 30 23:45:40.572065 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:40.572039 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ccm6fd" event={"ID":"8ec1fa33-555e-4ce8-9884-15d05bcb9b6c","Type":"ContainerStarted","Data":"883c0d87f3b333c4f9ce34f944703bad1db79faeae5db69b1f348f6dcca7adbd"} Oct 30 23:45:40.689187 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:40.689124 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpxwxl" podStartSLOduration=2.534244916 podStartE2EDuration="32.689107546s" totalImagesPullingTime="30.15486263s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:45:08 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:45:40.68789863 +0000 UTC m=+253.724229480" watchObservedRunningTime="2026-10-30 23:45:40.689107546 +0000 UTC m=+253.725438370" Oct 30 23:45:40.810315 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:40.810251 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ccm6fd" podStartSLOduration=3.237937188 podStartE2EDuration="31.810229979s" totalImagesPullingTime="28.572292791s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:45:09 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:45:40.808722307 +0000 UTC m=+253.845053149" watchObservedRunningTime="2026-10-30 23:45:40.810229979 +0000 UTC m=+253.846560802" Oct 30 23:45:41.591563 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:41.591515 4927 generic.go:356] "Generic (PLEG): container finished" podID="2a201fda-d325-4697-98e9-f477125ce0a0" containerID="e5f098c1bc291542eb44cfcf5a68ec9fc0704c4726e7dc6cdd696a93990235ec" exitCode=0 Oct 30 23:45:41.592073 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:41.591613 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpxwxl" event={"ID":"2a201fda-d325-4697-98e9-f477125ce0a0","Type":"ContainerDied","Data":"e5f098c1bc291542eb44cfcf5a68ec9fc0704c4726e7dc6cdd696a93990235ec"} Oct 30 23:45:41.597666 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:41.597628 4927 generic.go:356] "Generic (PLEG): container finished" podID="8ec1fa33-555e-4ce8-9884-15d05bcb9b6c" containerID="883c0d87f3b333c4f9ce34f944703bad1db79faeae5db69b1f348f6dcca7adbd" exitCode=0 Oct 30 23:45:41.597802 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:41.597702 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ccm6fd" event={"ID":"8ec1fa33-555e-4ce8-9884-15d05bcb9b6c","Type":"ContainerDied","Data":"883c0d87f3b333c4f9ce34f944703bad1db79faeae5db69b1f348f6dcca7adbd"} Oct 30 23:45:42.618735 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:42.618073 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wwjsn" event={"ID":"70668fb4-8523-4bf5-94ed-0e8440598b20","Type":"ContainerStarted","Data":"17b4b9949b0a8a6cba447a833bab0fb26e833244fc0ace1cbdd2de1533cf3b58"} Oct 30 23:45:42.749955 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:42.749436 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wwjsn" podStartSLOduration=2.743650154 podStartE2EDuration="34.749417597s" totalImagesPullingTime="32.005767443s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:45:08 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:45:42.747316026 +0000 UTC m=+255.783646893" watchObservedRunningTime="2026-10-30 23:45:42.749417597 +0000 UTC m=+255.785748421" Oct 30 23:45:42.818063 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:42.817726 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ccm6fd" Oct 30 23:45:42.876358 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:42.876302 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpxwxl" Oct 30 23:45:42.952256 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:42.951479 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ec1fa33-555e-4ce8-9884-15d05bcb9b6c-kube-api-access-kr2f5\" (UniqueName: \"kubernetes.io/projected/8ec1fa33-555e-4ce8-9884-15d05bcb9b6c-kube-api-access-kr2f5\") pod \"8ec1fa33-555e-4ce8-9884-15d05bcb9b6c\" (UID: \"8ec1fa33-555e-4ce8-9884-15d05bcb9b6c\") " Oct 30 23:45:42.952256 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:42.951567 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a201fda-d325-4697-98e9-f477125ce0a0-bundle\" (UniqueName: \"kubernetes.io/empty-dir/2a201fda-d325-4697-98e9-f477125ce0a0-bundle\") pod \"2a201fda-d325-4697-98e9-f477125ce0a0\" (UID: \"2a201fda-d325-4697-98e9-f477125ce0a0\") " Oct 30 23:45:42.952256 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:42.951615 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a201fda-d325-4697-98e9-f477125ce0a0-util\" (UniqueName: \"kubernetes.io/empty-dir/2a201fda-d325-4697-98e9-f477125ce0a0-util\") pod \"2a201fda-d325-4697-98e9-f477125ce0a0\" (UID: \"2a201fda-d325-4697-98e9-f477125ce0a0\") " Oct 30 23:45:42.952256 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:42.951658 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ec1fa33-555e-4ce8-9884-15d05bcb9b6c-util\" (UniqueName: \"kubernetes.io/empty-dir/8ec1fa33-555e-4ce8-9884-15d05bcb9b6c-util\") pod \"8ec1fa33-555e-4ce8-9884-15d05bcb9b6c\" (UID: \"8ec1fa33-555e-4ce8-9884-15d05bcb9b6c\") " Oct 30 23:45:42.952256 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:42.951711 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ec1fa33-555e-4ce8-9884-15d05bcb9b6c-bundle\" (UniqueName: \"kubernetes.io/empty-dir/8ec1fa33-555e-4ce8-9884-15d05bcb9b6c-bundle\") pod \"8ec1fa33-555e-4ce8-9884-15d05bcb9b6c\" (UID: \"8ec1fa33-555e-4ce8-9884-15d05bcb9b6c\") " Oct 30 23:45:42.952256 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:42.951771 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a201fda-d325-4697-98e9-f477125ce0a0-kube-api-access-5gkdn\" (UniqueName: \"kubernetes.io/projected/2a201fda-d325-4697-98e9-f477125ce0a0-kube-api-access-5gkdn\") pod \"2a201fda-d325-4697-98e9-f477125ce0a0\" (UID: \"2a201fda-d325-4697-98e9-f477125ce0a0\") " Oct 30 23:45:42.957778 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:42.957741 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a201fda-d325-4697-98e9-f477125ce0a0-bundle" pod "2a201fda-d325-4697-98e9-f477125ce0a0" (UID: "2a201fda-d325-4697-98e9-f477125ce0a0"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:45:42.957920 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:42.957846 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ec1fa33-555e-4ce8-9884-15d05bcb9b6c-bundle" pod "8ec1fa33-555e-4ce8-9884-15d05bcb9b6c" (UID: "8ec1fa33-555e-4ce8-9884-15d05bcb9b6c"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:45:42.961495 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:42.961470 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a201fda-d325-4697-98e9-f477125ce0a0-util" pod "2a201fda-d325-4697-98e9-f477125ce0a0" (UID: "2a201fda-d325-4697-98e9-f477125ce0a0"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:45:42.961801 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:42.961777 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ec1fa33-555e-4ce8-9884-15d05bcb9b6c-util" pod "8ec1fa33-555e-4ce8-9884-15d05bcb9b6c" (UID: "8ec1fa33-555e-4ce8-9884-15d05bcb9b6c"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:45:42.965934 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:42.965890 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ec1fa33-555e-4ce8-9884-15d05bcb9b6c-kube-api-access-kr2f5" pod "8ec1fa33-555e-4ce8-9884-15d05bcb9b6c" (UID: "8ec1fa33-555e-4ce8-9884-15d05bcb9b6c"). InnerVolumeSpecName "kube-api-access-kr2f5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:45:42.967419 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:42.967378 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a201fda-d325-4697-98e9-f477125ce0a0-kube-api-access-5gkdn" pod "2a201fda-d325-4697-98e9-f477125ce0a0" (UID: "2a201fda-d325-4697-98e9-f477125ce0a0"). InnerVolumeSpecName "kube-api-access-5gkdn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:45:43.053828 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:43.053683 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kr2f5\" (UniqueName: \"kubernetes.io/projected/8ec1fa33-555e-4ce8-9884-15d05bcb9b6c-kube-api-access-kr2f5\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 23:45:43.053828 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:43.053717 4927 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2a201fda-d325-4697-98e9-f477125ce0a0-bundle\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 23:45:43.053828 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:43.053733 4927 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2a201fda-d325-4697-98e9-f477125ce0a0-util\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 23:45:43.053828 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:43.053746 4927 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8ec1fa33-555e-4ce8-9884-15d05bcb9b6c-util\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 23:45:43.053828 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:43.053762 4927 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8ec1fa33-555e-4ce8-9884-15d05bcb9b6c-bundle\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 23:45:43.053828 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:43.053775 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5gkdn\" (UniqueName: \"kubernetes.io/projected/2a201fda-d325-4697-98e9-f477125ce0a0-kube-api-access-5gkdn\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 23:45:43.628193 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:43.626883 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpxwxl" event={"ID":"2a201fda-d325-4697-98e9-f477125ce0a0","Type":"ContainerDied","Data":"23272f2af931a43d8e020512513c0c404afc4b1d56d0df755a11ce55a11886ee"} Oct 30 23:45:43.628193 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:43.626943 4927 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="23272f2af931a43d8e020512513c0c404afc4b1d56d0df755a11ce55a11886ee" Oct 30 23:45:43.628193 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:43.627030 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpxwxl" Oct 30 23:45:43.632356 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:43.632309 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ccm6fd" Oct 30 23:45:43.632504 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:43.632360 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ccm6fd" event={"ID":"8ec1fa33-555e-4ce8-9884-15d05bcb9b6c","Type":"ContainerDied","Data":"e47c5221c84477abee98c021633d694e57ec31dd99c75f201b2f349f5ff5876c"} Oct 30 23:45:43.632504 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:43.632411 4927 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e47c5221c84477abee98c021633d694e57ec31dd99c75f201b2f349f5ff5876c" Oct 30 23:45:43.636483 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:43.636457 4927 generic.go:356] "Generic (PLEG): container finished" podID="70668fb4-8523-4bf5-94ed-0e8440598b20" containerID="17b4b9949b0a8a6cba447a833bab0fb26e833244fc0ace1cbdd2de1533cf3b58" exitCode=0 Oct 30 23:45:43.636597 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:43.636501 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wwjsn" event={"ID":"70668fb4-8523-4bf5-94ed-0e8440598b20","Type":"ContainerDied","Data":"17b4b9949b0a8a6cba447a833bab0fb26e833244fc0ace1cbdd2de1533cf3b58"} Oct 30 23:45:44.695528 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:44.643512 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448rs44" event={"ID":"bb30f2f7-5143-4b00-84c4-b691cc0b3da5","Type":"ContainerStarted","Data":"95d1fc7ac1c146b5e3cc3fd7b92e31e71c7d909d096905266b036acb8bfca3fc"} Oct 30 23:45:44.695528 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:44.645902 4927 generic.go:356] "Generic (PLEG): container finished" podID="bf56ecae-e59d-4283-86fe-c32458951e5a" containerID="d7fb157bb6194ec8102725cbb483122090bebdecca1449dba0deb41027120098" exitCode=0 Oct 30 23:45:44.695528 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:44.645991 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7khvmq" event={"ID":"bf56ecae-e59d-4283-86fe-c32458951e5a","Type":"ContainerDied","Data":"d7fb157bb6194ec8102725cbb483122090bebdecca1449dba0deb41027120098"} Oct 30 23:45:44.695528 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:44.648362 4927 generic.go:356] "Generic (PLEG): container finished" podID="eee7dbc4-a47b-49a7-b521-9550a9f68162" containerID="0e13b42615ca3be74d6ef2f4dca75926397c3d9d8bd8a6db09f4f95f1bacf9cd" exitCode=0 Oct 30 23:45:44.695528 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:44.648438 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644p5hlp" event={"ID":"eee7dbc4-a47b-49a7-b521-9550a9f68162","Type":"ContainerDied","Data":"0e13b42615ca3be74d6ef2f4dca75926397c3d9d8bd8a6db09f4f95f1bacf9cd"} Oct 30 23:45:44.766133 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:44.766083 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448rs44" podStartSLOduration=3.724232391 podStartE2EDuration="41.7660682s" totalImagesPullingTime="38.041835809s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:45:03 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:45:44.764709545 +0000 UTC m=+257.801040367" watchObservedRunningTime="2026-10-30 23:45:44.7660682 +0000 UTC m=+257.802399023" Oct 30 23:45:44.818803 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:44.818781 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wwjsn" Oct 30 23:45:44.970518 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:44.970435 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70668fb4-8523-4bf5-94ed-0e8440598b20-util\" (UniqueName: \"kubernetes.io/empty-dir/70668fb4-8523-4bf5-94ed-0e8440598b20-util\") pod \"70668fb4-8523-4bf5-94ed-0e8440598b20\" (UID: \"70668fb4-8523-4bf5-94ed-0e8440598b20\") " Oct 30 23:45:44.970518 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:44.970468 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70668fb4-8523-4bf5-94ed-0e8440598b20-bundle\" (UniqueName: \"kubernetes.io/empty-dir/70668fb4-8523-4bf5-94ed-0e8440598b20-bundle\") pod \"70668fb4-8523-4bf5-94ed-0e8440598b20\" (UID: \"70668fb4-8523-4bf5-94ed-0e8440598b20\") " Oct 30 23:45:44.970518 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:44.970519 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70668fb4-8523-4bf5-94ed-0e8440598b20-kube-api-access-bhrj6\" (UniqueName: \"kubernetes.io/projected/70668fb4-8523-4bf5-94ed-0e8440598b20-kube-api-access-bhrj6\") pod \"70668fb4-8523-4bf5-94ed-0e8440598b20\" (UID: \"70668fb4-8523-4bf5-94ed-0e8440598b20\") " Oct 30 23:45:44.970977 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:44.970949 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70668fb4-8523-4bf5-94ed-0e8440598b20-bundle" pod "70668fb4-8523-4bf5-94ed-0e8440598b20" (UID: "70668fb4-8523-4bf5-94ed-0e8440598b20"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:45:44.972612 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:44.972589 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70668fb4-8523-4bf5-94ed-0e8440598b20-kube-api-access-bhrj6" pod "70668fb4-8523-4bf5-94ed-0e8440598b20" (UID: "70668fb4-8523-4bf5-94ed-0e8440598b20"). InnerVolumeSpecName "kube-api-access-bhrj6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:45:44.976400 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:44.976375 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70668fb4-8523-4bf5-94ed-0e8440598b20-util" pod "70668fb4-8523-4bf5-94ed-0e8440598b20" (UID: "70668fb4-8523-4bf5-94ed-0e8440598b20"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:45:45.071058 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:45.071034 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bhrj6\" (UniqueName: \"kubernetes.io/projected/70668fb4-8523-4bf5-94ed-0e8440598b20-kube-api-access-bhrj6\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 23:45:45.071058 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:45.071059 4927 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/70668fb4-8523-4bf5-94ed-0e8440598b20-util\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 23:45:45.071190 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:45.071069 4927 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/70668fb4-8523-4bf5-94ed-0e8440598b20-bundle\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 23:45:45.655247 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:45.655211 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cwms65" event={"ID":"604b4e17-0542-42e7-8c4b-82b0700f781d","Type":"ContainerStarted","Data":"613ab6fe0a4d50700851e5488c35d3c83f918ffb85ba7322f72f878ff67b7c05"} Oct 30 23:45:45.657290 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:45.657261 4927 generic.go:356] "Generic (PLEG): container finished" podID="bb30f2f7-5143-4b00-84c4-b691cc0b3da5" containerID="95d1fc7ac1c146b5e3cc3fd7b92e31e71c7d909d096905266b036acb8bfca3fc" exitCode=0 Oct 30 23:45:45.657427 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:45.657356 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448rs44" event={"ID":"bb30f2f7-5143-4b00-84c4-b691cc0b3da5","Type":"ContainerDied","Data":"95d1fc7ac1c146b5e3cc3fd7b92e31e71c7d909d096905266b036acb8bfca3fc"} Oct 30 23:45:45.659388 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:45.659360 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atc5bh" event={"ID":"2cc429ca-1667-4683-a0e2-a8d5fdd42a64","Type":"ContainerStarted","Data":"33f07fb826faaf3b6eecbf7411c78924fbf587a15ce88dce820756fb97d8fd58"} Oct 30 23:45:45.661094 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:45.661071 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wwjsn" Oct 30 23:45:45.661189 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:45.661068 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wwjsn" event={"ID":"70668fb4-8523-4bf5-94ed-0e8440598b20","Type":"ContainerDied","Data":"0eaeadb670dc743c0c4619dd5f51dd72ccb0f68324f77214ffed69cf7f9c2367"} Oct 30 23:45:45.661189 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:45.661181 4927 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0eaeadb670dc743c0c4619dd5f51dd72ccb0f68324f77214ffed69cf7f9c2367" Oct 30 23:45:45.663429 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:45.663407 4927 generic.go:356] "Generic (PLEG): container finished" podID="525fd99a-040b-43c8-bbb3-41a0f953ea4a" containerID="e7d319a871318793e80027ba61f894192f25ad454dea5fe1ba9d74aa18e0b511" exitCode=0 Oct 30 23:45:45.663519 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:45.663476 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be719c6lt" event={"ID":"525fd99a-040b-43c8-bbb3-41a0f953ea4a","Type":"ContainerDied","Data":"e7d319a871318793e80027ba61f894192f25ad454dea5fe1ba9d74aa18e0b511"} Oct 30 23:45:45.778857 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:45.778761 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cwms65" podStartSLOduration=3.385979209 podStartE2EDuration="42.778743433s" totalImagesPullingTime="39.392764224s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:45:03 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:45:45.776997799 +0000 UTC m=+258.813328620" watchObservedRunningTime="2026-10-30 23:45:45.778743433 +0000 UTC m=+258.815074252" Oct 30 23:45:45.797986 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:45.797962 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7khvmq" Oct 30 23:45:45.817842 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:45.817818 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644p5hlp" Oct 30 23:45:45.878028 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:45.877997 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eee7dbc4-a47b-49a7-b521-9550a9f68162-kube-api-access-72n6x\" (UniqueName: \"kubernetes.io/projected/eee7dbc4-a47b-49a7-b521-9550a9f68162-kube-api-access-72n6x\") pod \"eee7dbc4-a47b-49a7-b521-9550a9f68162\" (UID: \"eee7dbc4-a47b-49a7-b521-9550a9f68162\") " Oct 30 23:45:45.878206 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:45.878060 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf56ecae-e59d-4283-86fe-c32458951e5a-kube-api-access-pp9q6\" (UniqueName: \"kubernetes.io/projected/bf56ecae-e59d-4283-86fe-c32458951e5a-kube-api-access-pp9q6\") pod \"bf56ecae-e59d-4283-86fe-c32458951e5a\" (UID: \"bf56ecae-e59d-4283-86fe-c32458951e5a\") " Oct 30 23:45:45.878206 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:45.878083 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eee7dbc4-a47b-49a7-b521-9550a9f68162-bundle\" (UniqueName: \"kubernetes.io/empty-dir/eee7dbc4-a47b-49a7-b521-9550a9f68162-bundle\") pod \"eee7dbc4-a47b-49a7-b521-9550a9f68162\" (UID: \"eee7dbc4-a47b-49a7-b521-9550a9f68162\") " Oct 30 23:45:45.878206 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:45.878100 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf56ecae-e59d-4283-86fe-c32458951e5a-util\" (UniqueName: \"kubernetes.io/empty-dir/bf56ecae-e59d-4283-86fe-c32458951e5a-util\") pod \"bf56ecae-e59d-4283-86fe-c32458951e5a\" (UID: \"bf56ecae-e59d-4283-86fe-c32458951e5a\") " Oct 30 23:45:45.878206 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:45.878124 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eee7dbc4-a47b-49a7-b521-9550a9f68162-util\" (UniqueName: \"kubernetes.io/empty-dir/eee7dbc4-a47b-49a7-b521-9550a9f68162-util\") pod \"eee7dbc4-a47b-49a7-b521-9550a9f68162\" (UID: \"eee7dbc4-a47b-49a7-b521-9550a9f68162\") " Oct 30 23:45:45.878206 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:45.878148 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf56ecae-e59d-4283-86fe-c32458951e5a-bundle\" (UniqueName: \"kubernetes.io/empty-dir/bf56ecae-e59d-4283-86fe-c32458951e5a-bundle\") pod \"bf56ecae-e59d-4283-86fe-c32458951e5a\" (UID: \"bf56ecae-e59d-4283-86fe-c32458951e5a\") " Oct 30 23:45:45.878629 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:45.878602 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eee7dbc4-a47b-49a7-b521-9550a9f68162-bundle" pod "eee7dbc4-a47b-49a7-b521-9550a9f68162" (UID: "eee7dbc4-a47b-49a7-b521-9550a9f68162"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:45:45.878742 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:45.878723 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf56ecae-e59d-4283-86fe-c32458951e5a-bundle" pod "bf56ecae-e59d-4283-86fe-c32458951e5a" (UID: "bf56ecae-e59d-4283-86fe-c32458951e5a"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:45:45.880169 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:45.880149 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eee7dbc4-a47b-49a7-b521-9550a9f68162-kube-api-access-72n6x" pod "eee7dbc4-a47b-49a7-b521-9550a9f68162" (UID: "eee7dbc4-a47b-49a7-b521-9550a9f68162"). InnerVolumeSpecName "kube-api-access-72n6x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:45:45.880670 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:45.880644 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf56ecae-e59d-4283-86fe-c32458951e5a-kube-api-access-pp9q6" pod "bf56ecae-e59d-4283-86fe-c32458951e5a" (UID: "bf56ecae-e59d-4283-86fe-c32458951e5a"). InnerVolumeSpecName "kube-api-access-pp9q6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:45:45.882969 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:45.882949 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf56ecae-e59d-4283-86fe-c32458951e5a-util" pod "bf56ecae-e59d-4283-86fe-c32458951e5a" (UID: "bf56ecae-e59d-4283-86fe-c32458951e5a"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:45:45.883816 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:45.883790 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eee7dbc4-a47b-49a7-b521-9550a9f68162-util" pod "eee7dbc4-a47b-49a7-b521-9550a9f68162" (UID: "eee7dbc4-a47b-49a7-b521-9550a9f68162"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:45:45.979693 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:45.979651 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-72n6x\" (UniqueName: \"kubernetes.io/projected/eee7dbc4-a47b-49a7-b521-9550a9f68162-kube-api-access-72n6x\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 23:45:45.979693 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:45.979682 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pp9q6\" (UniqueName: \"kubernetes.io/projected/bf56ecae-e59d-4283-86fe-c32458951e5a-kube-api-access-pp9q6\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 23:45:45.979693 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:45.979692 4927 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/eee7dbc4-a47b-49a7-b521-9550a9f68162-bundle\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 23:45:45.979693 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:45.979702 4927 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bf56ecae-e59d-4283-86fe-c32458951e5a-util\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 23:45:45.979944 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:45.979710 4927 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/eee7dbc4-a47b-49a7-b521-9550a9f68162-util\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 23:45:45.979944 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:45.979718 4927 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bf56ecae-e59d-4283-86fe-c32458951e5a-bundle\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 23:45:46.093770 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:46.093673 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atc5bh" podStartSLOduration=4.905329 podStartE2EDuration="39.093657555s" totalImagesPullingTime="34.188328555s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:45:07 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:45:46.09276 +0000 UTC m=+259.129090824" watchObservedRunningTime="2026-10-30 23:45:46.093657555 +0000 UTC m=+259.129988380" Oct 30 23:45:46.189199 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:46.189164 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:45:46.669214 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:46.669187 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644p5hlp" Oct 30 23:45:46.669214 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:46.669197 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644p5hlp" event={"ID":"eee7dbc4-a47b-49a7-b521-9550a9f68162","Type":"ContainerDied","Data":"faee1bb0d8733f88a89815d64d22b72a4e09c285a013713274871dbc6710619a"} Oct 30 23:45:46.669461 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:46.669250 4927 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="faee1bb0d8733f88a89815d64d22b72a4e09c285a013713274871dbc6710619a" Oct 30 23:45:46.671469 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:46.671431 4927 generic.go:356] "Generic (PLEG): container finished" podID="604b4e17-0542-42e7-8c4b-82b0700f781d" containerID="613ab6fe0a4d50700851e5488c35d3c83f918ffb85ba7322f72f878ff67b7c05" exitCode=0 Oct 30 23:45:46.671469 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:46.671459 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cwms65" event={"ID":"604b4e17-0542-42e7-8c4b-82b0700f781d","Type":"ContainerDied","Data":"613ab6fe0a4d50700851e5488c35d3c83f918ffb85ba7322f72f878ff67b7c05"} Oct 30 23:45:46.673900 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:46.673875 4927 generic.go:356] "Generic (PLEG): container finished" podID="2cc429ca-1667-4683-a0e2-a8d5fdd42a64" containerID="33f07fb826faaf3b6eecbf7411c78924fbf587a15ce88dce820756fb97d8fd58" exitCode=0 Oct 30 23:45:46.674004 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:46.673987 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atc5bh" event={"ID":"2cc429ca-1667-4683-a0e2-a8d5fdd42a64","Type":"ContainerDied","Data":"33f07fb826faaf3b6eecbf7411c78924fbf587a15ce88dce820756fb97d8fd58"} Oct 30 23:45:46.675814 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:46.675792 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7khvmq" Oct 30 23:45:46.675909 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:46.675813 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7khvmq" event={"ID":"bf56ecae-e59d-4283-86fe-c32458951e5a","Type":"ContainerDied","Data":"a8fd9e1ec1b47ae9887fac96c834768087734877728b30069f42cdcf7b3ab2c5"} Oct 30 23:45:46.675909 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:46.675850 4927 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a8fd9e1ec1b47ae9887fac96c834768087734877728b30069f42cdcf7b3ab2c5" Oct 30 23:45:46.802647 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:46.802621 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be719c6lt" Oct 30 23:45:46.837214 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:46.837185 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448rs44" Oct 30 23:45:46.887079 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:46.887046 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/525fd99a-040b-43c8-bbb3-41a0f953ea4a-bundle\" (UniqueName: \"kubernetes.io/empty-dir/525fd99a-040b-43c8-bbb3-41a0f953ea4a-bundle\") pod \"525fd99a-040b-43c8-bbb3-41a0f953ea4a\" (UID: \"525fd99a-040b-43c8-bbb3-41a0f953ea4a\") " Oct 30 23:45:46.887079 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:46.887078 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/525fd99a-040b-43c8-bbb3-41a0f953ea4a-util\" (UniqueName: \"kubernetes.io/empty-dir/525fd99a-040b-43c8-bbb3-41a0f953ea4a-util\") pod \"525fd99a-040b-43c8-bbb3-41a0f953ea4a\" (UID: \"525fd99a-040b-43c8-bbb3-41a0f953ea4a\") " Oct 30 23:45:46.887320 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:46.887142 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/525fd99a-040b-43c8-bbb3-41a0f953ea4a-kube-api-access-5fgkd\" (UniqueName: \"kubernetes.io/projected/525fd99a-040b-43c8-bbb3-41a0f953ea4a-kube-api-access-5fgkd\") pod \"525fd99a-040b-43c8-bbb3-41a0f953ea4a\" (UID: \"525fd99a-040b-43c8-bbb3-41a0f953ea4a\") " Oct 30 23:45:46.887680 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:46.887655 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/525fd99a-040b-43c8-bbb3-41a0f953ea4a-bundle" pod "525fd99a-040b-43c8-bbb3-41a0f953ea4a" (UID: "525fd99a-040b-43c8-bbb3-41a0f953ea4a"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:45:46.889267 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:46.889247 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/525fd99a-040b-43c8-bbb3-41a0f953ea4a-kube-api-access-5fgkd" pod "525fd99a-040b-43c8-bbb3-41a0f953ea4a" (UID: "525fd99a-040b-43c8-bbb3-41a0f953ea4a"). InnerVolumeSpecName "kube-api-access-5fgkd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:45:46.892316 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:46.892294 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/525fd99a-040b-43c8-bbb3-41a0f953ea4a-util" pod "525fd99a-040b-43c8-bbb3-41a0f953ea4a" (UID: "525fd99a-040b-43c8-bbb3-41a0f953ea4a"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:45:46.968434 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:46.968402 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:45:46.988312 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:46.988280 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb30f2f7-5143-4b00-84c4-b691cc0b3da5-util\" (UniqueName: \"kubernetes.io/empty-dir/bb30f2f7-5143-4b00-84c4-b691cc0b3da5-util\") pod \"bb30f2f7-5143-4b00-84c4-b691cc0b3da5\" (UID: \"bb30f2f7-5143-4b00-84c4-b691cc0b3da5\") " Oct 30 23:45:46.988498 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:46.988359 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb30f2f7-5143-4b00-84c4-b691cc0b3da5-bundle\" (UniqueName: \"kubernetes.io/empty-dir/bb30f2f7-5143-4b00-84c4-b691cc0b3da5-bundle\") pod \"bb30f2f7-5143-4b00-84c4-b691cc0b3da5\" (UID: \"bb30f2f7-5143-4b00-84c4-b691cc0b3da5\") " Oct 30 23:45:46.988498 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:46.988390 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb30f2f7-5143-4b00-84c4-b691cc0b3da5-kube-api-access-fd2ml\" (UniqueName: \"kubernetes.io/projected/bb30f2f7-5143-4b00-84c4-b691cc0b3da5-kube-api-access-fd2ml\") pod \"bb30f2f7-5143-4b00-84c4-b691cc0b3da5\" (UID: \"bb30f2f7-5143-4b00-84c4-b691cc0b3da5\") " Oct 30 23:45:46.988633 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:46.988542 4927 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/525fd99a-040b-43c8-bbb3-41a0f953ea4a-bundle\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 23:45:46.988633 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:46.988553 4927 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/525fd99a-040b-43c8-bbb3-41a0f953ea4a-util\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 23:45:46.988633 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:46.988562 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5fgkd\" (UniqueName: \"kubernetes.io/projected/525fd99a-040b-43c8-bbb3-41a0f953ea4a-kube-api-access-5fgkd\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 23:45:46.988788 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:46.988744 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb30f2f7-5143-4b00-84c4-b691cc0b3da5-bundle" pod "bb30f2f7-5143-4b00-84c4-b691cc0b3da5" (UID: "bb30f2f7-5143-4b00-84c4-b691cc0b3da5"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:45:46.990615 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:46.990591 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb30f2f7-5143-4b00-84c4-b691cc0b3da5-kube-api-access-fd2ml" pod "bb30f2f7-5143-4b00-84c4-b691cc0b3da5" (UID: "bb30f2f7-5143-4b00-84c4-b691cc0b3da5"). InnerVolumeSpecName "kube-api-access-fd2ml". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:45:46.992830 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:46.992807 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb30f2f7-5143-4b00-84c4-b691cc0b3da5-util" pod "bb30f2f7-5143-4b00-84c4-b691cc0b3da5" (UID: "bb30f2f7-5143-4b00-84c4-b691cc0b3da5"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:45:47.089897 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:47.089817 4927 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bb30f2f7-5143-4b00-84c4-b691cc0b3da5-bundle\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 23:45:47.089897 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:47.089851 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fd2ml\" (UniqueName: \"kubernetes.io/projected/bb30f2f7-5143-4b00-84c4-b691cc0b3da5-kube-api-access-fd2ml\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 23:45:47.089897 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:47.089867 4927 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bb30f2f7-5143-4b00-84c4-b691cc0b3da5-util\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 23:45:47.172276 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:47.172234 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:45:47.681993 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:47.681963 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be719c6lt" Oct 30 23:45:47.682188 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:47.681999 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be719c6lt" event={"ID":"525fd99a-040b-43c8-bbb3-41a0f953ea4a","Type":"ContainerDied","Data":"0ede853461d2cf58e89b70613aed68ce64aa5c6dc95512ad8cd023f47f9a6460"} Oct 30 23:45:47.682188 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:47.682056 4927 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0ede853461d2cf58e89b70613aed68ce64aa5c6dc95512ad8cd023f47f9a6460" Oct 30 23:45:47.683953 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:47.683932 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448rs44" Oct 30 23:45:47.684084 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:47.683968 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448rs44" event={"ID":"bb30f2f7-5143-4b00-84c4-b691cc0b3da5","Type":"ContainerDied","Data":"fbd76ab61ca53b52a12f3ed486f6799d6b1c884740f53899eda9739552e224c1"} Oct 30 23:45:47.684084 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:47.684014 4927 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fbd76ab61ca53b52a12f3ed486f6799d6b1c884740f53899eda9739552e224c1" Oct 30 23:45:47.839786 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:47.839758 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cwms65" Oct 30 23:45:47.841561 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:47.841545 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atc5bh" Oct 30 23:45:47.998183 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:47.998145 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/604b4e17-0542-42e7-8c4b-82b0700f781d-kube-api-access-5q26d\" (UniqueName: \"kubernetes.io/projected/604b4e17-0542-42e7-8c4b-82b0700f781d-kube-api-access-5q26d\") pod \"604b4e17-0542-42e7-8c4b-82b0700f781d\" (UID: \"604b4e17-0542-42e7-8c4b-82b0700f781d\") " Oct 30 23:45:47.998390 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:47.998197 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/604b4e17-0542-42e7-8c4b-82b0700f781d-bundle\" (UniqueName: \"kubernetes.io/empty-dir/604b4e17-0542-42e7-8c4b-82b0700f781d-bundle\") pod \"604b4e17-0542-42e7-8c4b-82b0700f781d\" (UID: \"604b4e17-0542-42e7-8c4b-82b0700f781d\") " Oct 30 23:45:47.998390 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:47.998277 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2cc429ca-1667-4683-a0e2-a8d5fdd42a64-bundle\" (UniqueName: \"kubernetes.io/empty-dir/2cc429ca-1667-4683-a0e2-a8d5fdd42a64-bundle\") pod \"2cc429ca-1667-4683-a0e2-a8d5fdd42a64\" (UID: \"2cc429ca-1667-4683-a0e2-a8d5fdd42a64\") " Oct 30 23:45:47.998390 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:47.998324 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/604b4e17-0542-42e7-8c4b-82b0700f781d-util\" (UniqueName: \"kubernetes.io/empty-dir/604b4e17-0542-42e7-8c4b-82b0700f781d-util\") pod \"604b4e17-0542-42e7-8c4b-82b0700f781d\" (UID: \"604b4e17-0542-42e7-8c4b-82b0700f781d\") " Oct 30 23:45:47.998390 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:47.998376 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2cc429ca-1667-4683-a0e2-a8d5fdd42a64-kube-api-access-8szzf\" (UniqueName: \"kubernetes.io/projected/2cc429ca-1667-4683-a0e2-a8d5fdd42a64-kube-api-access-8szzf\") pod \"2cc429ca-1667-4683-a0e2-a8d5fdd42a64\" (UID: \"2cc429ca-1667-4683-a0e2-a8d5fdd42a64\") " Oct 30 23:45:47.998607 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:47.998401 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2cc429ca-1667-4683-a0e2-a8d5fdd42a64-util\" (UniqueName: \"kubernetes.io/empty-dir/2cc429ca-1667-4683-a0e2-a8d5fdd42a64-util\") pod \"2cc429ca-1667-4683-a0e2-a8d5fdd42a64\" (UID: \"2cc429ca-1667-4683-a0e2-a8d5fdd42a64\") " Oct 30 23:45:47.999286 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:47.999250 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/604b4e17-0542-42e7-8c4b-82b0700f781d-bundle" pod "604b4e17-0542-42e7-8c4b-82b0700f781d" (UID: "604b4e17-0542-42e7-8c4b-82b0700f781d"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:45:47.999440 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:47.999315 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cc429ca-1667-4683-a0e2-a8d5fdd42a64-bundle" pod "2cc429ca-1667-4683-a0e2-a8d5fdd42a64" (UID: "2cc429ca-1667-4683-a0e2-a8d5fdd42a64"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:45:48.000567 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:48.000526 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/604b4e17-0542-42e7-8c4b-82b0700f781d-kube-api-access-5q26d" pod "604b4e17-0542-42e7-8c4b-82b0700f781d" (UID: "604b4e17-0542-42e7-8c4b-82b0700f781d"). InnerVolumeSpecName "kube-api-access-5q26d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:45:48.000729 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:48.000706 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2cc429ca-1667-4683-a0e2-a8d5fdd42a64-kube-api-access-8szzf" pod "2cc429ca-1667-4683-a0e2-a8d5fdd42a64" (UID: "2cc429ca-1667-4683-a0e2-a8d5fdd42a64"). InnerVolumeSpecName "kube-api-access-8szzf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:45:48.004269 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:48.004243 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cc429ca-1667-4683-a0e2-a8d5fdd42a64-util" pod "2cc429ca-1667-4683-a0e2-a8d5fdd42a64" (UID: "2cc429ca-1667-4683-a0e2-a8d5fdd42a64"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:45:48.004700 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:48.004678 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/604b4e17-0542-42e7-8c4b-82b0700f781d-util" pod "604b4e17-0542-42e7-8c4b-82b0700f781d" (UID: "604b4e17-0542-42e7-8c4b-82b0700f781d"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:45:48.099037 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:48.098948 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5q26d\" (UniqueName: \"kubernetes.io/projected/604b4e17-0542-42e7-8c4b-82b0700f781d-kube-api-access-5q26d\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 23:45:48.099037 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:48.098976 4927 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/604b4e17-0542-42e7-8c4b-82b0700f781d-bundle\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 23:45:48.099037 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:48.098986 4927 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2cc429ca-1667-4683-a0e2-a8d5fdd42a64-bundle\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 23:45:48.099037 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:48.098994 4927 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/604b4e17-0542-42e7-8c4b-82b0700f781d-util\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 23:45:48.099037 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:48.099002 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8szzf\" (UniqueName: \"kubernetes.io/projected/2cc429ca-1667-4683-a0e2-a8d5fdd42a64-kube-api-access-8szzf\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 23:45:48.099037 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:48.099011 4927 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2cc429ca-1667-4683-a0e2-a8d5fdd42a64-util\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 23:45:48.568662 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:48.568616 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-khk74"] Oct 30 23:45:48.568993 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:48.568976 4927 cpu_manager.go:414] "containerMap: removing container" podUID="bf56ecae-e59d-4283-86fe-c32458951e5a" containerName="extract" Oct 30 23:45:48.569112 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:48.568996 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf56ecae-e59d-4283-86fe-c32458951e5a" containerName="extract" Oct 30 23:45:48.569112 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:48.569011 4927 cpu_manager.go:414] "containerMap: removing container" podUID="2cc429ca-1667-4683-a0e2-a8d5fdd42a64" containerName="extract" Oct 30 23:45:48.569112 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:48.569018 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2cc429ca-1667-4683-a0e2-a8d5fdd42a64" containerName="extract" Oct 30 23:45:48.569112 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:48.569032 4927 cpu_manager.go:414] "containerMap: removing container" podUID="2a201fda-d325-4697-98e9-f477125ce0a0" containerName="extract" Oct 30 23:45:48.569112 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:48.569039 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a201fda-d325-4697-98e9-f477125ce0a0" containerName="extract" Oct 30 23:45:48.569112 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:48.569060 4927 cpu_manager.go:414] "containerMap: removing container" podUID="eee7dbc4-a47b-49a7-b521-9550a9f68162" containerName="util" Oct 30 23:45:48.569112 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:48.569067 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eee7dbc4-a47b-49a7-b521-9550a9f68162" containerName="util" Oct 30 23:45:48.569112 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:48.569078 4927 cpu_manager.go:414] "containerMap: removing container" podUID="8ec1fa33-555e-4ce8-9884-15d05bcb9b6c" containerName="pull" Oct 30 23:45:48.569112 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:48.569085 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ec1fa33-555e-4ce8-9884-15d05bcb9b6c" containerName="pull" Oct 30 23:45:48.569112 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:48.569099 4927 cpu_manager.go:414] "containerMap: removing container" podUID="604b4e17-0542-42e7-8c4b-82b0700f781d" containerName="util" Oct 30 23:45:48.569112 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:48.569107 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="604b4e17-0542-42e7-8c4b-82b0700f781d" containerName="util" Oct 30 23:45:48.569112 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:48.569117 4927 cpu_manager.go:414] "containerMap: removing container" podUID="525fd99a-040b-43c8-bbb3-41a0f953ea4a" containerName="pull" Oct 30 23:45:48.569735 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:48.569124 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="525fd99a-040b-43c8-bbb3-41a0f953ea4a" containerName="pull" Oct 30 23:45:48.569735 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:48.569138 4927 cpu_manager.go:414] "containerMap: removing container" podUID="8ec1fa33-555e-4ce8-9884-15d05bcb9b6c" containerName="extract" Oct 30 23:45:48.569735 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:48.569144 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ec1fa33-555e-4ce8-9884-15d05bcb9b6c" containerName="extract" Oct 30 23:45:48.569735 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:48.569154 4927 cpu_manager.go:414] "containerMap: removing container" podUID="eee7dbc4-a47b-49a7-b521-9550a9f68162" containerName="extract" Oct 30 23:45:48.569735 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:48.569160 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eee7dbc4-a47b-49a7-b521-9550a9f68162" containerName="extract" Oct 30 23:45:48.569735 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:48.569171 4927 cpu_manager.go:414] "containerMap: removing container" podUID="8ec1fa33-555e-4ce8-9884-15d05bcb9b6c" containerName="util" Oct 30 23:45:48.569735 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:48.569177 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ec1fa33-555e-4ce8-9884-15d05bcb9b6c" containerName="util" Oct 30 23:45:48.569735 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:48.569188 4927 cpu_manager.go:414] "containerMap: removing container" podUID="2cc429ca-1667-4683-a0e2-a8d5fdd42a64" containerName="pull" Oct 30 23:45:48.569735 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:48.569196 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2cc429ca-1667-4683-a0e2-a8d5fdd42a64" containerName="pull" Oct 30 23:45:48.569735 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:48.569207 4927 cpu_manager.go:414] "containerMap: removing container" podUID="bb30f2f7-5143-4b00-84c4-b691cc0b3da5" containerName="extract" Oct 30 23:45:48.569735 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:48.569215 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb30f2f7-5143-4b00-84c4-b691cc0b3da5" containerName="extract" Oct 30 23:45:48.569735 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:48.569250 4927 cpu_manager.go:414] "containerMap: removing container" podUID="70668fb4-8523-4bf5-94ed-0e8440598b20" containerName="util" Oct 30 23:45:48.569735 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:48.569259 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70668fb4-8523-4bf5-94ed-0e8440598b20" containerName="util" Oct 30 23:45:48.569735 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:48.569270 4927 cpu_manager.go:414] "containerMap: removing container" podUID="bf56ecae-e59d-4283-86fe-c32458951e5a" containerName="util" Oct 30 23:45:48.569735 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:48.569277 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf56ecae-e59d-4283-86fe-c32458951e5a" containerName="util" Oct 30 23:45:48.569735 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:48.569288 4927 cpu_manager.go:414] "containerMap: removing container" podUID="2a201fda-d325-4697-98e9-f477125ce0a0" containerName="pull" Oct 30 23:45:48.569735 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:48.569297 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a201fda-d325-4697-98e9-f477125ce0a0" containerName="pull" Oct 30 23:45:48.569735 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:48.569312 4927 cpu_manager.go:414] "containerMap: removing container" podUID="2a201fda-d325-4697-98e9-f477125ce0a0" containerName="util" Oct 30 23:45:48.569735 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:48.569319 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a201fda-d325-4697-98e9-f477125ce0a0" containerName="util" Oct 30 23:45:48.569735 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:48.569344 4927 cpu_manager.go:414] "containerMap: removing container" podUID="2cc429ca-1667-4683-a0e2-a8d5fdd42a64" containerName="util" Oct 30 23:45:48.569735 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:48.569352 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2cc429ca-1667-4683-a0e2-a8d5fdd42a64" containerName="util" Oct 30 23:45:48.569735 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:48.569364 4927 cpu_manager.go:414] "containerMap: removing container" podUID="bb30f2f7-5143-4b00-84c4-b691cc0b3da5" containerName="util" Oct 30 23:45:48.569735 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:48.569370 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb30f2f7-5143-4b00-84c4-b691cc0b3da5" containerName="util" Oct 30 23:45:48.569735 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:48.569380 4927 cpu_manager.go:414] "containerMap: removing container" podUID="bf56ecae-e59d-4283-86fe-c32458951e5a" containerName="pull" Oct 30 23:45:48.569735 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:48.569387 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf56ecae-e59d-4283-86fe-c32458951e5a" containerName="pull" Oct 30 23:45:48.569735 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:48.569396 4927 cpu_manager.go:414] "containerMap: removing container" podUID="eee7dbc4-a47b-49a7-b521-9550a9f68162" containerName="pull" Oct 30 23:45:48.569735 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:48.569403 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eee7dbc4-a47b-49a7-b521-9550a9f68162" containerName="pull" Oct 30 23:45:48.569735 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:48.569412 4927 cpu_manager.go:414] "containerMap: removing container" podUID="604b4e17-0542-42e7-8c4b-82b0700f781d" containerName="pull" Oct 30 23:45:48.569735 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:48.569419 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="604b4e17-0542-42e7-8c4b-82b0700f781d" containerName="pull" Oct 30 23:45:48.569735 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:48.569436 4927 cpu_manager.go:414] "containerMap: removing container" podUID="70668fb4-8523-4bf5-94ed-0e8440598b20" containerName="pull" Oct 30 23:45:48.569735 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:48.569443 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70668fb4-8523-4bf5-94ed-0e8440598b20" containerName="pull" Oct 30 23:45:48.569735 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:48.569454 4927 cpu_manager.go:414] "containerMap: removing container" podUID="bb30f2f7-5143-4b00-84c4-b691cc0b3da5" containerName="pull" Oct 30 23:45:48.569735 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:48.569461 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb30f2f7-5143-4b00-84c4-b691cc0b3da5" containerName="pull" Oct 30 23:45:48.569735 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:48.569477 4927 cpu_manager.go:414] "containerMap: removing container" podUID="70668fb4-8523-4bf5-94ed-0e8440598b20" containerName="extract" Oct 30 23:45:48.569735 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:48.569486 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70668fb4-8523-4bf5-94ed-0e8440598b20" containerName="extract" Oct 30 23:45:48.569735 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:48.569496 4927 cpu_manager.go:414] "containerMap: removing container" podUID="525fd99a-040b-43c8-bbb3-41a0f953ea4a" containerName="extract" Oct 30 23:45:48.569735 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:48.569503 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="525fd99a-040b-43c8-bbb3-41a0f953ea4a" containerName="extract" Oct 30 23:45:48.569735 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:48.569514 4927 cpu_manager.go:414] "containerMap: removing container" podUID="525fd99a-040b-43c8-bbb3-41a0f953ea4a" containerName="util" Oct 30 23:45:48.569735 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:48.569520 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="525fd99a-040b-43c8-bbb3-41a0f953ea4a" containerName="util" Oct 30 23:45:48.569735 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:48.569533 4927 cpu_manager.go:414] "containerMap: removing container" podUID="604b4e17-0542-42e7-8c4b-82b0700f781d" containerName="extract" Oct 30 23:45:48.569735 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:48.569540 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="604b4e17-0542-42e7-8c4b-82b0700f781d" containerName="extract" Oct 30 23:45:48.569735 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:48.569631 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="70668fb4-8523-4bf5-94ed-0e8440598b20" containerName="extract" Oct 30 23:45:48.569735 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:48.569642 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="525fd99a-040b-43c8-bbb3-41a0f953ea4a" containerName="extract" Oct 30 23:45:48.569735 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:48.569653 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="604b4e17-0542-42e7-8c4b-82b0700f781d" containerName="extract" Oct 30 23:45:48.569735 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:48.569663 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf56ecae-e59d-4283-86fe-c32458951e5a" containerName="extract" Oct 30 23:45:48.569735 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:48.569672 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="2cc429ca-1667-4683-a0e2-a8d5fdd42a64" containerName="extract" Oct 30 23:45:48.569735 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:48.569682 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a201fda-d325-4697-98e9-f477125ce0a0" containerName="extract" Oct 30 23:45:48.569735 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:48.569694 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ec1fa33-555e-4ce8-9884-15d05bcb9b6c" containerName="extract" Oct 30 23:45:48.569735 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:48.569702 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="eee7dbc4-a47b-49a7-b521-9550a9f68162" containerName="extract" Oct 30 23:45:48.569735 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:48.569713 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb30f2f7-5143-4b00-84c4-b691cc0b3da5" containerName="extract" Oct 30 23:45:48.573916 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:48.573893 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-khk74" Oct 30 23:45:48.630157 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:48.630120 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Oct 30 23:45:48.630391 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:48.630167 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Oct 30 23:45:48.630391 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:48.630166 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-5vdlx\"" Oct 30 23:45:48.682963 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:48.682922 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-khk74"] Oct 30 23:45:48.690323 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:48.690302 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cwms65" Oct 30 23:45:48.690323 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:48.690315 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cwms65" event={"ID":"604b4e17-0542-42e7-8c4b-82b0700f781d","Type":"ContainerDied","Data":"3381bcd8374e2c7056e0c5da98fc70f2e626804f591d71a72a45eb3b3eecaa19"} Oct 30 23:45:48.690525 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:48.690370 4927 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3381bcd8374e2c7056e0c5da98fc70f2e626804f591d71a72a45eb3b3eecaa19" Oct 30 23:45:48.692346 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:48.692302 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atc5bh" event={"ID":"2cc429ca-1667-4683-a0e2-a8d5fdd42a64","Type":"ContainerDied","Data":"c3045e3e50368afb446ed657ed1ddb0f47019f537a227eedd9eaa12c5f38e7ee"} Oct 30 23:45:48.692428 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:48.692368 4927 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c3045e3e50368afb446ed657ed1ddb0f47019f537a227eedd9eaa12c5f38e7ee" Oct 30 23:45:48.692428 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:48.692377 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atc5bh" Oct 30 23:45:48.703826 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:48.703806 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4ba97aef-5c99-435b-8c9f-ab04baa6573b-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-khk74\" (UID: \"4ba97aef-5c99-435b-8c9f-ab04baa6573b\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-khk74" Oct 30 23:45:48.703928 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:48.703845 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42vcp\" (UniqueName: \"kubernetes.io/projected/4ba97aef-5c99-435b-8c9f-ab04baa6573b-kube-api-access-42vcp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-khk74\" (UID: \"4ba97aef-5c99-435b-8c9f-ab04baa6573b\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-khk74" Oct 30 23:45:48.804537 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:48.804494 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4ba97aef-5c99-435b-8c9f-ab04baa6573b-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-khk74\" (UID: \"4ba97aef-5c99-435b-8c9f-ab04baa6573b\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-khk74" Oct 30 23:45:48.804738 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:48.804544 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42vcp\" (UniqueName: \"kubernetes.io/projected/4ba97aef-5c99-435b-8c9f-ab04baa6573b-kube-api-access-42vcp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-khk74\" (UID: \"4ba97aef-5c99-435b-8c9f-ab04baa6573b\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-khk74" Oct 30 23:45:48.804889 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:48.804864 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4ba97aef-5c99-435b-8c9f-ab04baa6573b-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-khk74\" (UID: \"4ba97aef-5c99-435b-8c9f-ab04baa6573b\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-khk74" Oct 30 23:45:48.869294 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:48.869221 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-42vcp\" (UniqueName: \"kubernetes.io/projected/4ba97aef-5c99-435b-8c9f-ab04baa6573b-kube-api-access-42vcp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-khk74\" (UID: \"4ba97aef-5c99-435b-8c9f-ab04baa6573b\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-khk74" Oct 30 23:45:48.885003 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:48.884976 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-khk74" Oct 30 23:45:49.200757 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:49.200721 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:45:49.314185 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:49.314162 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-khk74"] Oct 30 23:45:49.316285 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:45:49.316253 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ba97aef_5c99_435b_8c9f_ab04baa6573b.slice/crio-d3bdb2701b99f6f2951fac4269b8524f5a066ed36cd1808e639114718ec4c61c WatchSource:0}: Error finding container d3bdb2701b99f6f2951fac4269b8524f5a066ed36cd1808e639114718ec4c61c: Status 404 returned error can't find the container with id d3bdb2701b99f6f2951fac4269b8524f5a066ed36cd1808e639114718ec4c61c Oct 30 23:45:49.397384 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:49.397355 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:45:49.677295 ip-10-0-141-180 kubenswrapper[4927]: E1030 23:45:49.677255 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sqv69" containerName="extractor" Oct 30 23:45:49.698033 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:49.698003 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-khk74" event={"ID":"4ba97aef-5c99-435b-8c9f-ab04baa6573b","Type":"ContainerStarted","Data":"d3bdb2701b99f6f2951fac4269b8524f5a066ed36cd1808e639114718ec4c61c"} Oct 30 23:45:49.781196 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:49.781160 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:45:49.945239 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:49.945137 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:45:52.566304 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:52.566276 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:45:52.715539 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:52.715501 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-khk74" event={"ID":"4ba97aef-5c99-435b-8c9f-ab04baa6573b","Type":"ContainerStarted","Data":"c70f835544acc526b71ef3be5b9bbf675819493c7f293ec782373e9e0aaf5905"} Oct 30 23:45:52.865326 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:52.865235 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:45:52.949376 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:52.949301 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-khk74" podStartSLOduration=1.744780236 podStartE2EDuration="4.949287735s" totalImagesPullingTime="3.204507499s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:45:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:45:52.897769714 +0000 UTC m=+265.934100539" watchObservedRunningTime="2026-10-30 23:45:52.949287735 +0000 UTC m=+265.985618556" Oct 30 23:45:53.603379 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:53.603325 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:45:53.826578 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:53.826545 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-6bfcd988bb-drg9s"] Oct 30 23:45:53.830266 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:53.830248 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bfcd988bb-drg9s" Oct 30 23:45:53.890406 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:53.890320 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Oct 30 23:45:53.890528 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:53.890411 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Oct 30 23:45:53.935521 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:53.935489 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Oct 30 23:45:53.935683 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:53.935608 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-pxfm6\"" Oct 30 23:45:53.935683 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:53.935657 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Oct 30 23:45:53.935763 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:53.935692 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Oct 30 23:45:53.950674 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:53.950641 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fhb47\" (UniqueName: \"kubernetes.io/projected/8759c4cd-4e33-454e-b4eb-9898ce5afd62-kube-api-access-fhb47\") pod \"opendatahub-operator-controller-manager-6bfcd988bb-drg9s\" (UID: \"8759c4cd-4e33-454e-b4eb-9898ce5afd62\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bfcd988bb-drg9s" Oct 30 23:45:53.950816 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:53.950688 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/8759c4cd-4e33-454e-b4eb-9898ce5afd62-manifests\") pod \"opendatahub-operator-controller-manager-6bfcd988bb-drg9s\" (UID: \"8759c4cd-4e33-454e-b4eb-9898ce5afd62\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bfcd988bb-drg9s" Oct 30 23:45:53.950816 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:53.950797 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/8759c4cd-4e33-454e-b4eb-9898ce5afd62-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6bfcd988bb-drg9s\" (UID: \"8759c4cd-4e33-454e-b4eb-9898ce5afd62\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bfcd988bb-drg9s" Oct 30 23:45:53.950921 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:53.950866 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/8759c4cd-4e33-454e-b4eb-9898ce5afd62-metrics-cert\") pod \"opendatahub-operator-controller-manager-6bfcd988bb-drg9s\" (UID: \"8759c4cd-4e33-454e-b4eb-9898ce5afd62\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bfcd988bb-drg9s" Oct 30 23:45:53.950921 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:53.950912 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8759c4cd-4e33-454e-b4eb-9898ce5afd62-webhook-cert\") pod \"opendatahub-operator-controller-manager-6bfcd988bb-drg9s\" (UID: \"8759c4cd-4e33-454e-b4eb-9898ce5afd62\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bfcd988bb-drg9s" Oct 30 23:45:53.951001 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:53.950934 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8759c4cd-4e33-454e-b4eb-9898ce5afd62-tmp\") pod \"opendatahub-operator-controller-manager-6bfcd988bb-drg9s\" (UID: \"8759c4cd-4e33-454e-b4eb-9898ce5afd62\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bfcd988bb-drg9s" Oct 30 23:45:54.051716 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:54.051679 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8759c4cd-4e33-454e-b4eb-9898ce5afd62-webhook-cert\") pod \"opendatahub-operator-controller-manager-6bfcd988bb-drg9s\" (UID: \"8759c4cd-4e33-454e-b4eb-9898ce5afd62\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bfcd988bb-drg9s" Oct 30 23:45:54.051716 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:54.051720 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8759c4cd-4e33-454e-b4eb-9898ce5afd62-tmp\") pod \"opendatahub-operator-controller-manager-6bfcd988bb-drg9s\" (UID: \"8759c4cd-4e33-454e-b4eb-9898ce5afd62\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bfcd988bb-drg9s" Oct 30 23:45:54.051954 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:54.051744 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhb47\" (UniqueName: \"kubernetes.io/projected/8759c4cd-4e33-454e-b4eb-9898ce5afd62-kube-api-access-fhb47\") pod \"opendatahub-operator-controller-manager-6bfcd988bb-drg9s\" (UID: \"8759c4cd-4e33-454e-b4eb-9898ce5afd62\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bfcd988bb-drg9s" Oct 30 23:45:54.051954 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:54.051772 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/8759c4cd-4e33-454e-b4eb-9898ce5afd62-manifests\") pod \"opendatahub-operator-controller-manager-6bfcd988bb-drg9s\" (UID: \"8759c4cd-4e33-454e-b4eb-9898ce5afd62\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bfcd988bb-drg9s" Oct 30 23:45:54.051954 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:54.051793 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/8759c4cd-4e33-454e-b4eb-9898ce5afd62-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6bfcd988bb-drg9s\" (UID: \"8759c4cd-4e33-454e-b4eb-9898ce5afd62\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bfcd988bb-drg9s" Oct 30 23:45:54.051954 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:54.051820 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/8759c4cd-4e33-454e-b4eb-9898ce5afd62-metrics-cert\") pod \"opendatahub-operator-controller-manager-6bfcd988bb-drg9s\" (UID: \"8759c4cd-4e33-454e-b4eb-9898ce5afd62\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bfcd988bb-drg9s" Oct 30 23:45:54.052286 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:54.052252 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/8759c4cd-4e33-454e-b4eb-9898ce5afd62-manifests\") pod \"opendatahub-operator-controller-manager-6bfcd988bb-drg9s\" (UID: \"8759c4cd-4e33-454e-b4eb-9898ce5afd62\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bfcd988bb-drg9s" Oct 30 23:45:54.054119 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:54.054093 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8759c4cd-4e33-454e-b4eb-9898ce5afd62-tmp\") pod \"opendatahub-operator-controller-manager-6bfcd988bb-drg9s\" (UID: \"8759c4cd-4e33-454e-b4eb-9898ce5afd62\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bfcd988bb-drg9s" Oct 30 23:45:54.054429 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:54.054407 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/8759c4cd-4e33-454e-b4eb-9898ce5afd62-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6bfcd988bb-drg9s\" (UID: \"8759c4cd-4e33-454e-b4eb-9898ce5afd62\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bfcd988bb-drg9s" Oct 30 23:45:54.054719 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:54.054697 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/8759c4cd-4e33-454e-b4eb-9898ce5afd62-metrics-cert\") pod \"opendatahub-operator-controller-manager-6bfcd988bb-drg9s\" (UID: \"8759c4cd-4e33-454e-b4eb-9898ce5afd62\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bfcd988bb-drg9s" Oct 30 23:45:54.054826 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:54.054803 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8759c4cd-4e33-454e-b4eb-9898ce5afd62-webhook-cert\") pod \"opendatahub-operator-controller-manager-6bfcd988bb-drg9s\" (UID: \"8759c4cd-4e33-454e-b4eb-9898ce5afd62\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bfcd988bb-drg9s" Oct 30 23:45:54.071835 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:54.071788 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:45:54.087589 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:54.087560 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-6bfcd988bb-drg9s"] Oct 30 23:45:54.120944 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:54.120912 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhb47\" (UniqueName: \"kubernetes.io/projected/8759c4cd-4e33-454e-b4eb-9898ce5afd62-kube-api-access-fhb47\") pod \"opendatahub-operator-controller-manager-6bfcd988bb-drg9s\" (UID: \"8759c4cd-4e33-454e-b4eb-9898ce5afd62\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bfcd988bb-drg9s" Oct 30 23:45:54.145394 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:54.145281 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bfcd988bb-drg9s" Oct 30 23:45:54.238217 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:54.238165 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:45:54.405942 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:54.405866 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:45:54.669293 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:54.669263 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-6bfcd988bb-drg9s"] Oct 30 23:45:54.672676 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:45:54.672635 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8759c4cd_4e33_454e_b4eb_9898ce5afd62.slice/crio-3e734b9c209e477afa512b57be59affcbb577802cbd231bc78904e63402b7ad0 WatchSource:0}: Error finding container 3e734b9c209e477afa512b57be59affcbb577802cbd231bc78904e63402b7ad0: Status 404 returned error can't find the container with id 3e734b9c209e477afa512b57be59affcbb577802cbd231bc78904e63402b7ad0 Oct 30 23:45:54.682570 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:54.682541 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:45:54.725356 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:54.725305 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bfcd988bb-drg9s" event={"ID":"8759c4cd-4e33-454e-b4eb-9898ce5afd62","Type":"ContainerStarted","Data":"3e734b9c209e477afa512b57be59affcbb577802cbd231bc78904e63402b7ad0"} Oct 30 23:45:54.921667 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:54.921576 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:45:56.955146 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:56.955101 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:45:57.163812 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:57.163774 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:45:57.389577 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:57.389493 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:45:57.445010 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:57.444977 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-rch45"] Oct 30 23:45:57.448746 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:57.448715 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-rch45" Oct 30 23:45:57.502622 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:57.502593 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-rch45"] Oct 30 23:45:57.551855 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:57.551812 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Oct 30 23:45:57.552022 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:57.551869 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Oct 30 23:45:57.552022 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:57.551936 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-vqvx6\"" Oct 30 23:45:57.585770 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:57.585735 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/15d076d0-ab26-4cf4-bce3-a91ed61d4a73-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-rch45\" (UID: \"15d076d0-ab26-4cf4-bce3-a91ed61d4a73\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-rch45" Oct 30 23:45:57.585958 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:57.585796 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6flgs\" (UniqueName: \"kubernetes.io/projected/15d076d0-ab26-4cf4-bce3-a91ed61d4a73-kube-api-access-6flgs\") pod \"cert-manager-cainjector-5c57bbd4fc-rch45\" (UID: \"15d076d0-ab26-4cf4-bce3-a91ed61d4a73\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-rch45" Oct 30 23:45:57.686707 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:57.686667 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6flgs\" (UniqueName: \"kubernetes.io/projected/15d076d0-ab26-4cf4-bce3-a91ed61d4a73-kube-api-access-6flgs\") pod \"cert-manager-cainjector-5c57bbd4fc-rch45\" (UID: \"15d076d0-ab26-4cf4-bce3-a91ed61d4a73\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-rch45" Oct 30 23:45:57.686909 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:57.686780 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/15d076d0-ab26-4cf4-bce3-a91ed61d4a73-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-rch45\" (UID: \"15d076d0-ab26-4cf4-bce3-a91ed61d4a73\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-rch45" Oct 30 23:45:57.752272 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:57.752244 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/15d076d0-ab26-4cf4-bce3-a91ed61d4a73-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-rch45\" (UID: \"15d076d0-ab26-4cf4-bce3-a91ed61d4a73\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-rch45" Oct 30 23:45:57.752462 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:57.752393 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6flgs\" (UniqueName: \"kubernetes.io/projected/15d076d0-ab26-4cf4-bce3-a91ed61d4a73-kube-api-access-6flgs\") pod \"cert-manager-cainjector-5c57bbd4fc-rch45\" (UID: \"15d076d0-ab26-4cf4-bce3-a91ed61d4a73\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-rch45" Oct 30 23:45:57.773182 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:57.773150 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-rch45" Oct 30 23:45:58.169497 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:58.169468 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-rch45"] Oct 30 23:45:58.171973 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:45:58.171941 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod15d076d0_ab26_4cf4_bce3_a91ed61d4a73.slice/crio-875c9a25c01675aa7a172be9c91db6fe4e50a582b860fb07b131c7c4ca6a2aa5 WatchSource:0}: Error finding container 875c9a25c01675aa7a172be9c91db6fe4e50a582b860fb07b131c7c4ca6a2aa5: Status 404 returned error can't find the container with id 875c9a25c01675aa7a172be9c91db6fe4e50a582b860fb07b131c7c4ca6a2aa5 Oct 30 23:45:58.747996 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:58.747962 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-rch45" event={"ID":"15d076d0-ab26-4cf4-bce3-a91ed61d4a73","Type":"ContainerStarted","Data":"875c9a25c01675aa7a172be9c91db6fe4e50a582b860fb07b131c7c4ca6a2aa5"} Oct 30 23:45:58.749714 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:58.749691 4927 generic.go:356] "Generic (PLEG): container finished" podID="8759c4cd-4e33-454e-b4eb-9898ce5afd62" containerID="4b208eb5c622bf2e60b94c74c2053e19cbbe1a81d3fa1c7f10febc97f97a9649" exitCode=0 Oct 30 23:45:58.749822 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:58.749773 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bfcd988bb-drg9s" event={"ID":"8759c4cd-4e33-454e-b4eb-9898ce5afd62","Type":"ContainerDied","Data":"4b208eb5c622bf2e60b94c74c2053e19cbbe1a81d3fa1c7f10febc97f97a9649"} Oct 30 23:45:59.161975 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:59.161934 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:45:59.374147 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:59.374114 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:45:59.756665 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:59.756625 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bfcd988bb-drg9s" event={"ID":"8759c4cd-4e33-454e-b4eb-9898ce5afd62","Type":"ContainerStarted","Data":"81ceda907ac4e5e8157fc1e202ae910f1399f8856c6ccbc58af2ee07e389b458"} Oct 30 23:45:59.756845 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:45:59.756824 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bfcd988bb-drg9s" Oct 30 23:46:00.021192 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:46:00.021098 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bfcd988bb-drg9s" podStartSLOduration=2.121988483 podStartE2EDuration="7.021083178s" totalImagesPullingTime="4.47920546s" totalInitContainerRuntime="419.889235ms" isStatefulPod=false podCreationTimestamp="2026-10-30 23:45:53 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:46:00.020323185 +0000 UTC m=+273.056654006" watchObservedRunningTime="2026-10-30 23:46:00.021083178 +0000 UTC m=+273.057413999" Oct 30 23:46:00.365208 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:46:00.365126 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-p7jpc"] Oct 30 23:46:00.368767 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:46:00.368738 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-p7jpc" Oct 30 23:46:00.432149 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:46:00.432098 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Oct 30 23:46:00.432654 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:46:00.432202 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Oct 30 23:46:00.432654 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:46:00.432253 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Oct 30 23:46:00.432654 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:46:00.432265 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-k9sfs\"" Oct 30 23:46:00.434810 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:46:00.434784 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-p7jpc"] Oct 30 23:46:00.511913 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:46:00.511875 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhprc\" (UniqueName: \"kubernetes.io/projected/af374ec1-e2c2-4ecc-94ea-cc92bea3b07c-kube-api-access-vhprc\") pod \"dns-operator-controller-manager-8557cbdb84-p7jpc\" (UID: \"af374ec1-e2c2-4ecc-94ea-cc92bea3b07c\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-p7jpc" Oct 30 23:46:00.612864 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:46:00.612825 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhprc\" (UniqueName: \"kubernetes.io/projected/af374ec1-e2c2-4ecc-94ea-cc92bea3b07c-kube-api-access-vhprc\") pod \"dns-operator-controller-manager-8557cbdb84-p7jpc\" (UID: \"af374ec1-e2c2-4ecc-94ea-cc92bea3b07c\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-p7jpc" Oct 30 23:46:00.678756 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:46:00.678722 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhprc\" (UniqueName: \"kubernetes.io/projected/af374ec1-e2c2-4ecc-94ea-cc92bea3b07c-kube-api-access-vhprc\") pod \"dns-operator-controller-manager-8557cbdb84-p7jpc\" (UID: \"af374ec1-e2c2-4ecc-94ea-cc92bea3b07c\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-p7jpc" Oct 30 23:46:00.684295 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:46:00.684264 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-p7jpc" Oct 30 23:46:01.085306 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:46:01.085237 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-p7jpc"] Oct 30 23:46:01.088649 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:46:01.088620 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf374ec1_e2c2_4ecc_94ea_cc92bea3b07c.slice/crio-5584424c31a62208f15fc233a8611c4fdefcc58bdfb590a29f0db3941d795e44 WatchSource:0}: Error finding container 5584424c31a62208f15fc233a8611c4fdefcc58bdfb590a29f0db3941d795e44: Status 404 returned error can't find the container with id 5584424c31a62208f15fc233a8611c4fdefcc58bdfb590a29f0db3941d795e44 Oct 30 23:46:01.380108 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:46:01.380002 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:46:01.612551 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:46:01.612512 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:46:01.776910 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:46:01.776874 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:46:01.777775 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:46:01.777750 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-p7jpc" event={"ID":"af374ec1-e2c2-4ecc-94ea-cc92bea3b07c","Type":"ContainerStarted","Data":"5584424c31a62208f15fc233a8611c4fdefcc58bdfb590a29f0db3941d795e44"} Oct 30 23:46:02.784738 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:46:02.784699 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-rch45" event={"ID":"15d076d0-ab26-4cf4-bce3-a91ed61d4a73","Type":"ContainerStarted","Data":"2856043cc7526f867349262268009d37afaafd980f9a4d82178b2220eff00403"} Oct 30 23:46:02.905736 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:46:02.905674 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-rch45" podStartSLOduration=1.959581204 podStartE2EDuration="5.905660617s" totalImagesPullingTime="3.946079413s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:45:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:46:02.904507521 +0000 UTC m=+275.940838344" watchObservedRunningTime="2026-10-30 23:46:02.905660617 +0000 UTC m=+275.941991435" Oct 30 23:46:03.653356 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:46:03.653303 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:46:03.835185 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:46:03.835135 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:46:04.352287 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:46:04.352251 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-4lpb4"] Oct 30 23:46:04.360473 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:46:04.359675 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-4lpb4" Oct 30 23:46:04.360667 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:46:04.360617 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-4lpb4"] Oct 30 23:46:04.413743 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:46:04.413711 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-kjqh5\"" Oct 30 23:46:04.449202 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:46:04.449170 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dh22v\" (UniqueName: \"kubernetes.io/projected/034bdd46-e6cc-474c-b6e7-43613ebfd7dd-kube-api-access-dh22v\") pod \"limitador-operator-controller-manager-5778bb4449-4lpb4\" (UID: \"034bdd46-e6cc-474c-b6e7-43613ebfd7dd\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-4lpb4" Oct 30 23:46:04.550087 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:46:04.550035 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dh22v\" (UniqueName: \"kubernetes.io/projected/034bdd46-e6cc-474c-b6e7-43613ebfd7dd-kube-api-access-dh22v\") pod \"limitador-operator-controller-manager-5778bb4449-4lpb4\" (UID: \"034bdd46-e6cc-474c-b6e7-43613ebfd7dd\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-4lpb4" Oct 30 23:46:04.613618 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:46:04.613543 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dh22v\" (UniqueName: \"kubernetes.io/projected/034bdd46-e6cc-474c-b6e7-43613ebfd7dd-kube-api-access-dh22v\") pod \"limitador-operator-controller-manager-5778bb4449-4lpb4\" (UID: \"034bdd46-e6cc-474c-b6e7-43613ebfd7dd\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-4lpb4" Oct 30 23:46:04.679103 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:46:04.679066 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-4lpb4" Oct 30 23:46:04.977744 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:46:04.977711 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:46:05.140628 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:46:05.140515 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-4lpb4"] Oct 30 23:46:05.151852 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:46:05.151595 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod034bdd46_e6cc_474c_b6e7_43613ebfd7dd.slice/crio-8aa8f6cd3b5b36d6bfa3923310e7f77218da7b02ce7391242bde404b357fc07b WatchSource:0}: Error finding container 8aa8f6cd3b5b36d6bfa3923310e7f77218da7b02ce7391242bde404b357fc07b: Status 404 returned error can't find the container with id 8aa8f6cd3b5b36d6bfa3923310e7f77218da7b02ce7391242bde404b357fc07b Oct 30 23:46:05.213762 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:46:05.213722 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:46:05.387231 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:46:05.387073 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:46:05.804721 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:46:05.804680 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-p7jpc" event={"ID":"af374ec1-e2c2-4ecc-94ea-cc92bea3b07c","Type":"ContainerStarted","Data":"422eab55e893706a98a5c7a65753166b3a6935db132a8c97230e1544199fdcb0"} Oct 30 23:46:05.804916 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:46:05.804800 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-p7jpc" Oct 30 23:46:05.806022 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:46:05.805997 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-4lpb4" event={"ID":"034bdd46-e6cc-474c-b6e7-43613ebfd7dd","Type":"ContainerStarted","Data":"8aa8f6cd3b5b36d6bfa3923310e7f77218da7b02ce7391242bde404b357fc07b"} Oct 30 23:46:05.930431 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:46:05.930380 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-p7jpc" podStartSLOduration=1.908535154 podStartE2EDuration="5.930365405s" totalImagesPullingTime="4.021830251s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:46:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:46:05.928649568 +0000 UTC m=+278.964980409" watchObservedRunningTime="2026-10-30 23:46:05.930365405 +0000 UTC m=+278.966696227" Oct 30 23:46:06.194439 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:46:06.194396 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:46:06.400848 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:46:06.400811 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:46:07.028616 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:46:07.028579 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:46:08.420885 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:46:08.420849 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:46:08.589745 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:46:08.589714 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:46:08.823817 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:46:08.823726 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-4lpb4" event={"ID":"034bdd46-e6cc-474c-b6e7-43613ebfd7dd","Type":"ContainerStarted","Data":"1211d8339cc51da81c32891875255bfe61ee566d23f0e9320a022f82f2e9ae54"} Oct 30 23:46:08.823965 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:46:08.823859 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-4lpb4" Oct 30 23:46:08.942371 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:46:08.942288 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-4lpb4" podStartSLOduration=1.595881598 podStartE2EDuration="4.942271493s" totalImagesPullingTime="3.346389895s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:46:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:46:08.941728823 +0000 UTC m=+281.978059644" watchObservedRunningTime="2026-10-30 23:46:08.942271493 +0000 UTC m=+281.978602316" Oct 30 23:46:10.773283 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:46:10.773253 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bfcd988bb-drg9s" Oct 30 23:46:12.791483 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:46:12.791450 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:46:13.017360 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:46:13.017312 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-fvgn8"] Oct 30 23:46:13.020231 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:46:13.020208 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-fvgn8" Oct 30 23:46:13.031311 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:46:13.031290 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-fvgn8"] Oct 30 23:46:13.085180 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:46:13.080056 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-7l4tw\"" Oct 30 23:46:13.122885 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:46:13.122836 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/89673660-8db3-4428-90cb-0ce72cf178e3-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-fvgn8\" (UID: \"89673660-8db3-4428-90cb-0ce72cf178e3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-fvgn8" Oct 30 23:46:13.123053 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:46:13.122918 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2krb\" (UniqueName: \"kubernetes.io/projected/89673660-8db3-4428-90cb-0ce72cf178e3-kube-api-access-q2krb\") pod \"kuadrant-operator-controller-manager-678d87766f-fvgn8\" (UID: \"89673660-8db3-4428-90cb-0ce72cf178e3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-fvgn8" Oct 30 23:46:13.223848 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:46:13.223803 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/89673660-8db3-4428-90cb-0ce72cf178e3-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-fvgn8\" (UID: \"89673660-8db3-4428-90cb-0ce72cf178e3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-fvgn8" Oct 30 23:46:13.224021 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:46:13.223875 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2krb\" (UniqueName: \"kubernetes.io/projected/89673660-8db3-4428-90cb-0ce72cf178e3-kube-api-access-q2krb\") pod \"kuadrant-operator-controller-manager-678d87766f-fvgn8\" (UID: \"89673660-8db3-4428-90cb-0ce72cf178e3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-fvgn8" Oct 30 23:46:13.224217 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:46:13.224196 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/89673660-8db3-4428-90cb-0ce72cf178e3-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-fvgn8\" (UID: \"89673660-8db3-4428-90cb-0ce72cf178e3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-fvgn8" Oct 30 23:46:13.295357 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:46:13.295310 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2krb\" (UniqueName: \"kubernetes.io/projected/89673660-8db3-4428-90cb-0ce72cf178e3-kube-api-access-q2krb\") pod \"kuadrant-operator-controller-manager-678d87766f-fvgn8\" (UID: \"89673660-8db3-4428-90cb-0ce72cf178e3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-fvgn8" Oct 30 23:46:13.333989 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:46:13.333958 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-fvgn8" Oct 30 23:46:13.377682 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:46:13.377646 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:46:13.786396 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:46:13.786370 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-fvgn8"] Oct 30 23:46:13.787958 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:46:13.787932 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89673660_8db3_4428_90cb_0ce72cf178e3.slice/crio-d72434e48fa34140748dda722de3b56348f76599c171dc6bba651a476ac4bb99 WatchSource:0}: Error finding container d72434e48fa34140748dda722de3b56348f76599c171dc6bba651a476ac4bb99: Status 404 returned error can't find the container with id d72434e48fa34140748dda722de3b56348f76599c171dc6bba651a476ac4bb99 Oct 30 23:46:13.848502 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:46:13.848468 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-fvgn8" event={"ID":"89673660-8db3-4428-90cb-0ce72cf178e3","Type":"ContainerStarted","Data":"d72434e48fa34140748dda722de3b56348f76599c171dc6bba651a476ac4bb99"} Oct 30 23:46:13.922939 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:46:13.922903 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:46:16.342291 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:46:16.342253 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-hrmk9"] Oct 30 23:46:16.345847 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:46:16.345825 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-hrmk9" Oct 30 23:46:16.410199 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:46:16.410160 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-n722w\"" Oct 30 23:46:16.453105 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:46:16.453069 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wrg4d\" (UniqueName: \"kubernetes.io/projected/db215b81-267b-4260-a635-12f41180c954-kube-api-access-wrg4d\") pod \"authorino-operator-7b7bb4bc77-hrmk9\" (UID: \"db215b81-267b-4260-a635-12f41180c954\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-hrmk9" Oct 30 23:46:16.463136 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:46:16.463102 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-hrmk9"] Oct 30 23:46:16.554467 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:46:16.554436 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrg4d\" (UniqueName: \"kubernetes.io/projected/db215b81-267b-4260-a635-12f41180c954-kube-api-access-wrg4d\") pod \"authorino-operator-7b7bb4bc77-hrmk9\" (UID: \"db215b81-267b-4260-a635-12f41180c954\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-hrmk9" Oct 30 23:46:16.621789 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:46:16.621714 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrg4d\" (UniqueName: \"kubernetes.io/projected/db215b81-267b-4260-a635-12f41180c954-kube-api-access-wrg4d\") pod \"authorino-operator-7b7bb4bc77-hrmk9\" (UID: \"db215b81-267b-4260-a635-12f41180c954\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-hrmk9" Oct 30 23:46:16.660724 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:46:16.660687 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-hrmk9" Oct 30 23:46:16.813775 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:46:16.813736 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-p7jpc" Oct 30 23:46:17.128052 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:46:17.128027 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-hrmk9"] Oct 30 23:46:17.130990 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:46:17.130957 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb215b81_267b_4260_a635_12f41180c954.slice/crio-528ecd0a43816316f3649d33047f040e293dc7d463668712112d2a707970f574 WatchSource:0}: Error finding container 528ecd0a43816316f3649d33047f040e293dc7d463668712112d2a707970f574: Status 404 returned error can't find the container with id 528ecd0a43816316f3649d33047f040e293dc7d463668712112d2a707970f574 Oct 30 23:46:17.872247 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:46:17.872201 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-hrmk9" event={"ID":"db215b81-267b-4260-a635-12f41180c954","Type":"ContainerStarted","Data":"528ecd0a43816316f3649d33047f040e293dc7d463668712112d2a707970f574"} Oct 30 23:46:18.260019 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:46:18.259974 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:46:18.478852 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:46:18.478805 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:46:18.698955 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:46:18.698918 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:46:19.685952 ip-10-0-141-180 kubenswrapper[4927]: E1030 23:46:19.685490 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sqv69" containerName="extractor" Oct 30 23:46:19.831203 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:46:19.831174 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-4lpb4" Oct 30 23:46:19.886665 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:46:19.886625 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-fvgn8" event={"ID":"89673660-8db3-4428-90cb-0ce72cf178e3","Type":"ContainerStarted","Data":"4284f55772c235305bf2dd1f4c18bfb6e4db105b551f1b96c4970d9d9843979d"} Oct 30 23:46:19.886833 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:46:19.886726 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-fvgn8" Oct 30 23:46:20.103142 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:46:20.103057 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-fvgn8" podStartSLOduration=2.099362984 podStartE2EDuration="8.103040585s" totalImagesPullingTime="6.003677601s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:46:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:46:20.10223915 +0000 UTC m=+293.138569991" watchObservedRunningTime="2026-10-30 23:46:20.103040585 +0000 UTC m=+293.139371409" Oct 30 23:46:20.892930 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:46:20.892880 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-hrmk9" event={"ID":"db215b81-267b-4260-a635-12f41180c954","Type":"ContainerStarted","Data":"abbca0705786c1224f7b62c56f64b9e6c283b4c8ead3c0ea0224b6d9843a8c25"} Oct 30 23:46:20.893386 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:46:20.892987 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-hrmk9" Oct 30 23:46:22.183613 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:46:22.183579 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:46:22.381402 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:46:22.381349 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:46:24.365714 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:46:24.365667 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:46:24.596533 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:46:24.596482 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:46:26.024526 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:46:26.024481 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:46:26.187993 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:46:26.187951 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:46:26.365187 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:46:26.365105 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:46:27.110203 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:46:27.110174 4927 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-pqt2c_6382dfb5-a363-47a2-bc79-0b96f7bec0c3/console-operator/3.log" Oct 30 23:46:27.110670 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:46:27.110452 4927 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-pqt2c_6382dfb5-a363-47a2-bc79-0b96f7bec0c3/console-operator/3.log" Oct 30 23:46:27.968934 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:46:27.968886 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:46:28.167190 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:46:28.167159 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:46:28.476871 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:46:28.476810 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:46:29.805107 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:46:29.805069 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:46:30.009062 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:46:30.009018 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:46:30.446132 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:46:30.446076 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-7b7bb4bc77-hrmk9" podStartSLOduration=11.4650522 podStartE2EDuration="14.446059653s" totalImagesPullingTime="2.981007453s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:46:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:46:21.015637254 +0000 UTC m=+294.051968077" watchObservedRunningTime="2026-10-30 23:46:30.446059653 +0000 UTC m=+303.482390474" Oct 30 23:46:30.446777 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:46:30.446759 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-58bf96fd7cbr2hg"] Oct 30 23:46:30.451876 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:46:30.451859 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-58bf96fd7cbr2hg" Oct 30 23:46:30.478801 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:46:30.478761 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:46:30.525622 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:46:30.525589 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-58bf96fd7cbr2hg"] Oct 30 23:46:30.558286 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:46:30.558255 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-z2j9d\"" Oct 30 23:46:30.586301 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:46:30.586253 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7cd43262-cf4d-40a6-baa7-37fdb7b34af7-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-58bf96fd7cbr2hg\" (UID: \"7cd43262-cf4d-40a6-baa7-37fdb7b34af7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-58bf96fd7cbr2hg" Oct 30 23:46:30.586567 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:46:30.586368 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7cd43262-cf4d-40a6-baa7-37fdb7b34af7-istio-token\") pod \"data-science-gateway-data-science-gateway-class-58bf96fd7cbr2hg\" (UID: \"7cd43262-cf4d-40a6-baa7-37fdb7b34af7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-58bf96fd7cbr2hg" Oct 30 23:46:30.586567 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:46:30.586402 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7cd43262-cf4d-40a6-baa7-37fdb7b34af7-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-58bf96fd7cbr2hg\" (UID: \"7cd43262-cf4d-40a6-baa7-37fdb7b34af7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-58bf96fd7cbr2hg" Oct 30 23:46:30.586567 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:46:30.586547 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cd43262-cf4d-40a6-baa7-37fdb7b34af7-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-58bf96fd7cbr2hg\" (UID: \"7cd43262-cf4d-40a6-baa7-37fdb7b34af7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-58bf96fd7cbr2hg" Oct 30 23:46:30.586749 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:46:30.586590 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cd43262-cf4d-40a6-baa7-37fdb7b34af7-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-58bf96fd7cbr2hg\" (UID: \"7cd43262-cf4d-40a6-baa7-37fdb7b34af7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-58bf96fd7cbr2hg" Oct 30 23:46:30.586749 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:46:30.586643 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7cd43262-cf4d-40a6-baa7-37fdb7b34af7-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-58bf96fd7cbr2hg\" (UID: \"7cd43262-cf4d-40a6-baa7-37fdb7b34af7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-58bf96fd7cbr2hg" Oct 30 23:46:30.586749 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:46:30.586687 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7cd43262-cf4d-40a6-baa7-37fdb7b34af7-istio-data\") pod \"data-science-gateway-data-science-gateway-class-58bf96fd7cbr2hg\" (UID: \"7cd43262-cf4d-40a6-baa7-37fdb7b34af7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-58bf96fd7cbr2hg" Oct 30 23:46:30.586749 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:46:30.586711 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7cd43262-cf4d-40a6-baa7-37fdb7b34af7-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-58bf96fd7cbr2hg\" (UID: \"7cd43262-cf4d-40a6-baa7-37fdb7b34af7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-58bf96fd7cbr2hg" Oct 30 23:46:30.586960 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:46:30.586792 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vppsb\" (UniqueName: \"kubernetes.io/projected/7cd43262-cf4d-40a6-baa7-37fdb7b34af7-kube-api-access-vppsb\") pod \"data-science-gateway-data-science-gateway-class-58bf96fd7cbr2hg\" (UID: \"7cd43262-cf4d-40a6-baa7-37fdb7b34af7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-58bf96fd7cbr2hg" Oct 30 23:46:30.688008 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:46:30.687969 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vppsb\" (UniqueName: \"kubernetes.io/projected/7cd43262-cf4d-40a6-baa7-37fdb7b34af7-kube-api-access-vppsb\") pod \"data-science-gateway-data-science-gateway-class-58bf96fd7cbr2hg\" (UID: \"7cd43262-cf4d-40a6-baa7-37fdb7b34af7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-58bf96fd7cbr2hg" Oct 30 23:46:30.688008 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:46:30.688013 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7cd43262-cf4d-40a6-baa7-37fdb7b34af7-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-58bf96fd7cbr2hg\" (UID: \"7cd43262-cf4d-40a6-baa7-37fdb7b34af7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-58bf96fd7cbr2hg" Oct 30 23:46:30.688242 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:46:30.688034 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7cd43262-cf4d-40a6-baa7-37fdb7b34af7-istio-token\") pod \"data-science-gateway-data-science-gateway-class-58bf96fd7cbr2hg\" (UID: \"7cd43262-cf4d-40a6-baa7-37fdb7b34af7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-58bf96fd7cbr2hg" Oct 30 23:46:30.688242 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:46:30.688079 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7cd43262-cf4d-40a6-baa7-37fdb7b34af7-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-58bf96fd7cbr2hg\" (UID: \"7cd43262-cf4d-40a6-baa7-37fdb7b34af7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-58bf96fd7cbr2hg" Oct 30 23:46:30.688242 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:46:30.688122 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cd43262-cf4d-40a6-baa7-37fdb7b34af7-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-58bf96fd7cbr2hg\" (UID: \"7cd43262-cf4d-40a6-baa7-37fdb7b34af7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-58bf96fd7cbr2hg" Oct 30 23:46:30.688413 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:46:30.688255 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cd43262-cf4d-40a6-baa7-37fdb7b34af7-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-58bf96fd7cbr2hg\" (UID: \"7cd43262-cf4d-40a6-baa7-37fdb7b34af7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-58bf96fd7cbr2hg" Oct 30 23:46:30.688413 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:46:30.688364 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7cd43262-cf4d-40a6-baa7-37fdb7b34af7-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-58bf96fd7cbr2hg\" (UID: \"7cd43262-cf4d-40a6-baa7-37fdb7b34af7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-58bf96fd7cbr2hg" Oct 30 23:46:30.688413 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:46:30.688402 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7cd43262-cf4d-40a6-baa7-37fdb7b34af7-istio-data\") pod \"data-science-gateway-data-science-gateway-class-58bf96fd7cbr2hg\" (UID: \"7cd43262-cf4d-40a6-baa7-37fdb7b34af7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-58bf96fd7cbr2hg" Oct 30 23:46:30.688570 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:46:30.688426 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7cd43262-cf4d-40a6-baa7-37fdb7b34af7-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-58bf96fd7cbr2hg\" (UID: \"7cd43262-cf4d-40a6-baa7-37fdb7b34af7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-58bf96fd7cbr2hg" Oct 30 23:46:30.688626 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:46:30.688563 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cd43262-cf4d-40a6-baa7-37fdb7b34af7-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-58bf96fd7cbr2hg\" (UID: \"7cd43262-cf4d-40a6-baa7-37fdb7b34af7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-58bf96fd7cbr2hg" Oct 30 23:46:30.688695 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:46:30.688663 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7cd43262-cf4d-40a6-baa7-37fdb7b34af7-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-58bf96fd7cbr2hg\" (UID: \"7cd43262-cf4d-40a6-baa7-37fdb7b34af7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-58bf96fd7cbr2hg" Oct 30 23:46:30.688812 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:46:30.688790 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7cd43262-cf4d-40a6-baa7-37fdb7b34af7-istio-data\") pod \"data-science-gateway-data-science-gateway-class-58bf96fd7cbr2hg\" (UID: \"7cd43262-cf4d-40a6-baa7-37fdb7b34af7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-58bf96fd7cbr2hg" Oct 30 23:46:30.688812 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:46:30.688798 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cd43262-cf4d-40a6-baa7-37fdb7b34af7-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-58bf96fd7cbr2hg\" (UID: \"7cd43262-cf4d-40a6-baa7-37fdb7b34af7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-58bf96fd7cbr2hg" Oct 30 23:46:30.689030 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:46:30.689010 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7cd43262-cf4d-40a6-baa7-37fdb7b34af7-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-58bf96fd7cbr2hg\" (UID: \"7cd43262-cf4d-40a6-baa7-37fdb7b34af7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-58bf96fd7cbr2hg" Oct 30 23:46:30.690654 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:46:30.690618 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7cd43262-cf4d-40a6-baa7-37fdb7b34af7-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-58bf96fd7cbr2hg\" (UID: \"7cd43262-cf4d-40a6-baa7-37fdb7b34af7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-58bf96fd7cbr2hg" Oct 30 23:46:30.690871 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:46:30.690851 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7cd43262-cf4d-40a6-baa7-37fdb7b34af7-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-58bf96fd7cbr2hg\" (UID: \"7cd43262-cf4d-40a6-baa7-37fdb7b34af7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-58bf96fd7cbr2hg" Oct 30 23:46:30.709165 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:46:30.709099 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:46:30.750718 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:46:30.750688 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7cd43262-cf4d-40a6-baa7-37fdb7b34af7-istio-token\") pod \"data-science-gateway-data-science-gateway-class-58bf96fd7cbr2hg\" (UID: \"7cd43262-cf4d-40a6-baa7-37fdb7b34af7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-58bf96fd7cbr2hg" Oct 30 23:46:30.750858 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:46:30.750780 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vppsb\" (UniqueName: \"kubernetes.io/projected/7cd43262-cf4d-40a6-baa7-37fdb7b34af7-kube-api-access-vppsb\") pod \"data-science-gateway-data-science-gateway-class-58bf96fd7cbr2hg\" (UID: \"7cd43262-cf4d-40a6-baa7-37fdb7b34af7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-58bf96fd7cbr2hg" Oct 30 23:46:30.767627 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:46:30.767595 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-58bf96fd7cbr2hg" Oct 30 23:46:30.858649 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:46:30.858610 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:46:30.895287 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:46:30.895255 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-fvgn8" Oct 30 23:46:31.200255 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:46:31.200216 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-58bf96fd7cbr2hg"] Oct 30 23:46:31.200707 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:46:31.200681 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7cd43262_cf4d_40a6_baa7_37fdb7b34af7.slice/crio-c5ed75f5fc7979fe40d6e46032bb11348f16fac9b12e438904d69b650d5d2554 WatchSource:0}: Error finding container c5ed75f5fc7979fe40d6e46032bb11348f16fac9b12e438904d69b650d5d2554: Status 404 returned error can't find the container with id c5ed75f5fc7979fe40d6e46032bb11348f16fac9b12e438904d69b650d5d2554 Oct 30 23:46:31.202892 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:46:31.202857 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 23:46:31.202996 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:46:31.202935 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 23:46:31.203056 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:46:31.203014 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 23:46:31.900487 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:46:31.900454 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-hrmk9" Oct 30 23:46:31.957208 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:46:31.957170 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-58bf96fd7cbr2hg" event={"ID":"7cd43262-cf4d-40a6-baa7-37fdb7b34af7","Type":"ContainerStarted","Data":"c9ecb0d83b4b3f5cb70e879cacd80ca92f5afe7672aca719f9c26a0a23c7d8c9"} Oct 30 23:46:31.957396 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:46:31.957247 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-58bf96fd7cbr2hg" event={"ID":"7cd43262-cf4d-40a6-baa7-37fdb7b34af7","Type":"ContainerStarted","Data":"c5ed75f5fc7979fe40d6e46032bb11348f16fac9b12e438904d69b650d5d2554"} Oct 30 23:46:32.232286 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:46:32.232225 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-58bf96fd7cbr2hg" podStartSLOduration=2.232210244 podStartE2EDuration="2.232210244s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:46:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:46:32.230856585 +0000 UTC m=+305.267187427" watchObservedRunningTime="2026-10-30 23:46:32.232210244 +0000 UTC m=+305.268541066" Oct 30 23:46:32.768243 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:46:32.768201 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-58bf96fd7cbr2hg" Oct 30 23:46:32.773121 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:46:32.773092 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-58bf96fd7cbr2hg" Oct 30 23:46:32.962777 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:46:32.962750 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-58bf96fd7cbr2hg" Oct 30 23:46:32.963813 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:46:32.963793 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-58bf96fd7cbr2hg" Oct 30 23:46:33.069452 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:46:33.069370 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:46:33.438628 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:46:33.438597 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:46:33.646656 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:46:33.646618 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:46:34.650698 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:46:34.650661 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-57f7f8659f-2kjlm"] Oct 30 23:46:34.655641 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:46:34.655615 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-2kjlm" Oct 30 23:46:34.706982 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:46:34.706948 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-2kjlm"] Oct 30 23:46:34.723715 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:46:34.723682 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l254m\" (UniqueName: \"kubernetes.io/projected/66074f55-7f40-4461-8003-3f8dc69d54e0-kube-api-access-l254m\") pod \"authorino-57f7f8659f-2kjlm\" (UID: \"66074f55-7f40-4461-8003-3f8dc69d54e0\") " pod="kuadrant-system/authorino-57f7f8659f-2kjlm" Oct 30 23:46:34.755910 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:46:34.755876 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-dn9sl\"" Oct 30 23:46:34.825183 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:46:34.825143 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l254m\" (UniqueName: \"kubernetes.io/projected/66074f55-7f40-4461-8003-3f8dc69d54e0-kube-api-access-l254m\") pod \"authorino-57f7f8659f-2kjlm\" (UID: \"66074f55-7f40-4461-8003-3f8dc69d54e0\") " pod="kuadrant-system/authorino-57f7f8659f-2kjlm" Oct 30 23:46:34.827050 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:46:34.826999 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:46:34.888248 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:46:34.888216 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l254m\" (UniqueName: \"kubernetes.io/projected/66074f55-7f40-4461-8003-3f8dc69d54e0-kube-api-access-l254m\") pod \"authorino-57f7f8659f-2kjlm\" (UID: \"66074f55-7f40-4461-8003-3f8dc69d54e0\") " pod="kuadrant-system/authorino-57f7f8659f-2kjlm" Oct 30 23:46:34.967241 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:46:34.967207 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-2kjlm" Oct 30 23:46:34.988594 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:46:34.988561 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:46:35.341725 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:46:35.341701 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-2kjlm"] Oct 30 23:46:35.343616 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:46:35.343589 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod66074f55_7f40_4461_8003_3f8dc69d54e0.slice/crio-23a26db55d992c790d471b3fd5a9e75c852dd5d84bf6b4545c381e81c5874adc WatchSource:0}: Error finding container 23a26db55d992c790d471b3fd5a9e75c852dd5d84bf6b4545c381e81c5874adc: Status 404 returned error can't find the container with id 23a26db55d992c790d471b3fd5a9e75c852dd5d84bf6b4545c381e81c5874adc Oct 30 23:46:35.345013 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:46:35.344990 4927 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 23:46:35.979630 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:46:35.979593 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-2kjlm" event={"ID":"66074f55-7f40-4461-8003-3f8dc69d54e0","Type":"ContainerStarted","Data":"23a26db55d992c790d471b3fd5a9e75c852dd5d84bf6b4545c381e81c5874adc"} Oct 30 23:46:37.515178 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:46:37.515092 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:46:37.725543 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:46:37.725503 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:46:37.882069 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:46:37.881988 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:46:38.999141 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:46:38.999104 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-2kjlm" event={"ID":"66074f55-7f40-4461-8003-3f8dc69d54e0","Type":"ContainerStarted","Data":"d406d8d2ff301650be9dbb60ae657efd5ddbf92284ee887b36ad30e198de9d2f"} Oct 30 23:46:39.146748 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:46:39.146687 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-57f7f8659f-2kjlm" podStartSLOduration=2.020320059 podStartE2EDuration="5.146672779s" totalImagesPullingTime="3.12635272s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:46:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:46:39.145663901 +0000 UTC m=+312.181994720" watchObservedRunningTime="2026-10-30 23:46:39.146672779 +0000 UTC m=+312.183003600" Oct 30 23:46:40.780353 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:46:40.780307 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:46:40.949282 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:46:40.949249 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:46:41.156477 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:46:41.156391 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-2kjlm"] Oct 30 23:46:41.156640 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:46:41.156568 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-57f7f8659f-2kjlm" podUID="66074f55-7f40-4461-8003-3f8dc69d54e0" containerName="authorino" containerID="cri-o://d406d8d2ff301650be9dbb60ae657efd5ddbf92284ee887b36ad30e198de9d2f" gracePeriod=30 Oct 30 23:46:41.408563 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:46:41.408540 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-2kjlm" Oct 30 23:46:41.484687 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:46:41.484649 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66074f55-7f40-4461-8003-3f8dc69d54e0-kube-api-access-l254m\" (UniqueName: \"kubernetes.io/projected/66074f55-7f40-4461-8003-3f8dc69d54e0-kube-api-access-l254m\") pod \"66074f55-7f40-4461-8003-3f8dc69d54e0\" (UID: \"66074f55-7f40-4461-8003-3f8dc69d54e0\") " Oct 30 23:46:41.486971 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:46:41.486938 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66074f55-7f40-4461-8003-3f8dc69d54e0-kube-api-access-l254m" pod "66074f55-7f40-4461-8003-3f8dc69d54e0" (UID: "66074f55-7f40-4461-8003-3f8dc69d54e0"). InnerVolumeSpecName "kube-api-access-l254m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:46:41.585673 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:46:41.585629 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l254m\" (UniqueName: \"kubernetes.io/projected/66074f55-7f40-4461-8003-3f8dc69d54e0-kube-api-access-l254m\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 23:46:42.021291 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:46:42.021250 4927 generic.go:356] "Generic (PLEG): container finished" podID="66074f55-7f40-4461-8003-3f8dc69d54e0" containerID="d406d8d2ff301650be9dbb60ae657efd5ddbf92284ee887b36ad30e198de9d2f" exitCode=0 Oct 30 23:46:42.021801 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:46:42.021308 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-2kjlm" Oct 30 23:46:42.021801 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:46:42.021321 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-2kjlm" event={"ID":"66074f55-7f40-4461-8003-3f8dc69d54e0","Type":"ContainerDied","Data":"d406d8d2ff301650be9dbb60ae657efd5ddbf92284ee887b36ad30e198de9d2f"} Oct 30 23:46:42.021801 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:46:42.021423 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-2kjlm" event={"ID":"66074f55-7f40-4461-8003-3f8dc69d54e0","Type":"ContainerDied","Data":"23a26db55d992c790d471b3fd5a9e75c852dd5d84bf6b4545c381e81c5874adc"} Oct 30 23:46:42.021801 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:46:42.021470 4927 scope.go:122] "RemoveContainer" containerID="d406d8d2ff301650be9dbb60ae657efd5ddbf92284ee887b36ad30e198de9d2f" Oct 30 23:46:42.035263 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:46:42.035237 4927 scope.go:122] "RemoveContainer" containerID="d406d8d2ff301650be9dbb60ae657efd5ddbf92284ee887b36ad30e198de9d2f" Oct 30 23:46:42.035645 ip-10-0-141-180 kubenswrapper[4927]: E1030 23:46:42.035625 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d406d8d2ff301650be9dbb60ae657efd5ddbf92284ee887b36ad30e198de9d2f\": container with ID starting with d406d8d2ff301650be9dbb60ae657efd5ddbf92284ee887b36ad30e198de9d2f not found: ID does not exist" containerID="d406d8d2ff301650be9dbb60ae657efd5ddbf92284ee887b36ad30e198de9d2f" Oct 30 23:46:42.035689 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:46:42.035654 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d406d8d2ff301650be9dbb60ae657efd5ddbf92284ee887b36ad30e198de9d2f"} err="failed to get container status \"d406d8d2ff301650be9dbb60ae657efd5ddbf92284ee887b36ad30e198de9d2f\": rpc error: code = NotFound desc = could not find container \"d406d8d2ff301650be9dbb60ae657efd5ddbf92284ee887b36ad30e198de9d2f\": container with ID starting with d406d8d2ff301650be9dbb60ae657efd5ddbf92284ee887b36ad30e198de9d2f not found: ID does not exist" Oct 30 23:46:42.191266 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:46:42.191225 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-2kjlm"] Oct 30 23:46:42.198502 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:46:42.198478 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-2kjlm"] Oct 30 23:46:42.659283 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:46:42.659241 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-xlnnv"] Oct 30 23:46:42.659715 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:46:42.659697 4927 cpu_manager.go:414] "containerMap: removing container" podUID="66074f55-7f40-4461-8003-3f8dc69d54e0" containerName="authorino" Oct 30 23:46:42.659823 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:46:42.659717 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66074f55-7f40-4461-8003-3f8dc69d54e0" containerName="authorino" Oct 30 23:46:42.659878 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:46:42.659842 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="66074f55-7f40-4461-8003-3f8dc69d54e0" containerName="authorino" Oct 30 23:46:42.663934 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:46:42.663909 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-xlnnv" Oct 30 23:46:42.718814 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:46:42.718778 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Oct 30 23:46:42.769603 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:46:42.769569 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Oct 30 23:46:42.769603 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:46:42.769570 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-module-controller-manager-dockercfg-srcdz\"" Oct 30 23:46:42.797447 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:46:42.797407 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qgkwr\" (UniqueName: \"kubernetes.io/projected/e47a3fb2-7c06-48fd-a056-bd4cba964438-kube-api-access-qgkwr\") pod \"kserve-module-controller-manager-7566cf5fc8-xlnnv\" (UID: \"e47a3fb2-7c06-48fd-a056-bd4cba964438\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-xlnnv" Oct 30 23:46:42.797615 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:46:42.797569 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e47a3fb2-7c06-48fd-a056-bd4cba964438-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-xlnnv\" (UID: \"e47a3fb2-7c06-48fd-a056-bd4cba964438\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-xlnnv" Oct 30 23:46:42.797672 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:46:42.797631 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e47a3fb2-7c06-48fd-a056-bd4cba964438-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-xlnnv\" (UID: \"e47a3fb2-7c06-48fd-a056-bd4cba964438\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-xlnnv" Oct 30 23:46:42.797734 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:46:42.797717 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/e47a3fb2-7c06-48fd-a056-bd4cba964438-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-xlnnv\" (UID: \"e47a3fb2-7c06-48fd-a056-bd4cba964438\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-xlnnv" Oct 30 23:46:42.873248 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:46:42.873211 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"odh-trusted-ca-bundle\"" Oct 30 23:46:42.875386 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:46:42.875360 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-xlnnv"] Oct 30 23:46:42.898623 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:46:42.898588 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e47a3fb2-7c06-48fd-a056-bd4cba964438-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-xlnnv\" (UID: \"e47a3fb2-7c06-48fd-a056-bd4cba964438\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-xlnnv" Oct 30 23:46:42.898793 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:46:42.898640 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e47a3fb2-7c06-48fd-a056-bd4cba964438-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-xlnnv\" (UID: \"e47a3fb2-7c06-48fd-a056-bd4cba964438\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-xlnnv" Oct 30 23:46:42.898793 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:46:42.898666 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/e47a3fb2-7c06-48fd-a056-bd4cba964438-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-xlnnv\" (UID: \"e47a3fb2-7c06-48fd-a056-bd4cba964438\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-xlnnv" Oct 30 23:46:42.898793 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:46:42.898696 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qgkwr\" (UniqueName: \"kubernetes.io/projected/e47a3fb2-7c06-48fd-a056-bd4cba964438-kube-api-access-qgkwr\") pod \"kserve-module-controller-manager-7566cf5fc8-xlnnv\" (UID: \"e47a3fb2-7c06-48fd-a056-bd4cba964438\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-xlnnv" Oct 30 23:46:42.899093 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:46:42.899074 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/e47a3fb2-7c06-48fd-a056-bd4cba964438-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-xlnnv\" (UID: \"e47a3fb2-7c06-48fd-a056-bd4cba964438\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-xlnnv" Oct 30 23:46:42.899195 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:46:42.899172 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e47a3fb2-7c06-48fd-a056-bd4cba964438-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-xlnnv\" (UID: \"e47a3fb2-7c06-48fd-a056-bd4cba964438\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-xlnnv" Oct 30 23:46:42.899919 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:46:42.899895 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e47a3fb2-7c06-48fd-a056-bd4cba964438-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-xlnnv\" (UID: \"e47a3fb2-7c06-48fd-a056-bd4cba964438\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-xlnnv" Oct 30 23:46:42.961566 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:46:42.961492 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qgkwr\" (UniqueName: \"kubernetes.io/projected/e47a3fb2-7c06-48fd-a056-bd4cba964438-kube-api-access-qgkwr\") pod \"kserve-module-controller-manager-7566cf5fc8-xlnnv\" (UID: \"e47a3fb2-7c06-48fd-a056-bd4cba964438\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-xlnnv" Oct 30 23:46:42.977319 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:46:42.977282 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-xlnnv" Oct 30 23:46:43.085001 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:46:43.084965 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="66074f55-7f40-4461-8003-3f8dc69d54e0" path="/var/lib/kubelet/pods/66074f55-7f40-4461-8003-3f8dc69d54e0/volumes" Oct 30 23:46:43.410100 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:46:43.410072 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-xlnnv"] Oct 30 23:46:43.411995 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:46:43.411971 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode47a3fb2_7c06_48fd_a056_bd4cba964438.slice/crio-40b79c9bb84a503fe9b10c46ba38617f958ecf9590e01cccdcbedac78ea88d94 WatchSource:0}: Error finding container 40b79c9bb84a503fe9b10c46ba38617f958ecf9590e01cccdcbedac78ea88d94: Status 404 returned error can't find the container with id 40b79c9bb84a503fe9b10c46ba38617f958ecf9590e01cccdcbedac78ea88d94 Oct 30 23:46:44.035878 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:46:44.035841 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-xlnnv" event={"ID":"e47a3fb2-7c06-48fd-a056-bd4cba964438","Type":"ContainerStarted","Data":"40b79c9bb84a503fe9b10c46ba38617f958ecf9590e01cccdcbedac78ea88d94"} Oct 30 23:46:45.304877 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:46:45.304790 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:46:45.467809 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:46:45.467745 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:46:45.620026 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:46:45.619939 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:46:45.779006 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:46:45.778971 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:46:47.055783 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:46:47.055688 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-xlnnv" event={"ID":"e47a3fb2-7c06-48fd-a056-bd4cba964438","Type":"ContainerStarted","Data":"4eea4f6ad07e5f0549df830e1386cd9be525c1d701f129458b02f7aa381bd90b"} Oct 30 23:46:47.056178 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:46:47.055868 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-xlnnv" Oct 30 23:46:47.229032 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:46:47.228974 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-xlnnv" podStartSLOduration=1.960505338 podStartE2EDuration="5.228959031s" totalImagesPullingTime="3.268453693s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:46:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:46:47.228088057 +0000 UTC m=+320.264418892" watchObservedRunningTime="2026-10-30 23:46:47.228959031 +0000 UTC m=+320.265289853" Oct 30 23:46:49.677858 ip-10-0-141-180 kubenswrapper[4927]: E1030 23:46:49.677818 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sqv69" containerName="extractor" Oct 30 23:46:51.131846 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:46:51.131812 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:46:51.288061 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:46:51.288023 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:46:51.486007 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:46:51.485971 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:46:54.165597 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:46:54.165562 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:46:54.314458 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:46:54.314427 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:46:55.504661 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:46:55.504622 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:46:55.705951 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:46:55.705917 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:46:56.214409 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:46:56.214376 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-mtsp7"] Oct 30 23:46:56.217978 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:46:56.217955 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-mtsp7" Oct 30 23:46:56.220470 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:46:56.220449 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p96dz\" (UniqueName: \"kubernetes.io/projected/8ddd2816-2a86-4118-bdda-d1f50dc650ee-kube-api-access-p96dz\") pod \"postgres-7cdbf6dc44-mtsp7\" (UID: \"8ddd2816-2a86-4118-bdda-d1f50dc650ee\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-mtsp7" Oct 30 23:46:56.220568 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:46:56.220510 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/8ddd2816-2a86-4118-bdda-d1f50dc650ee-data\") pod \"postgres-7cdbf6dc44-mtsp7\" (UID: \"8ddd2816-2a86-4118-bdda-d1f50dc650ee\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-mtsp7" Oct 30 23:46:56.273715 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:46:56.273688 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Oct 30 23:46:56.275302 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:46:56.275279 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-mtsp7"] Oct 30 23:46:56.313372 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:46:56.313318 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:46:56.321531 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:46:56.321510 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/8ddd2816-2a86-4118-bdda-d1f50dc650ee-data\") pod \"postgres-7cdbf6dc44-mtsp7\" (UID: \"8ddd2816-2a86-4118-bdda-d1f50dc650ee\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-mtsp7" Oct 30 23:46:56.321645 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:46:56.321561 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p96dz\" (UniqueName: \"kubernetes.io/projected/8ddd2816-2a86-4118-bdda-d1f50dc650ee-kube-api-access-p96dz\") pod \"postgres-7cdbf6dc44-mtsp7\" (UID: \"8ddd2816-2a86-4118-bdda-d1f50dc650ee\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-mtsp7" Oct 30 23:46:56.321909 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:46:56.321889 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/8ddd2816-2a86-4118-bdda-d1f50dc650ee-data\") pod \"postgres-7cdbf6dc44-mtsp7\" (UID: \"8ddd2816-2a86-4118-bdda-d1f50dc650ee\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-mtsp7" Oct 30 23:46:56.323618 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:46:56.323599 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-vtgdc\"" Oct 30 23:46:56.323734 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:46:56.323648 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 23:46:56.323779 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:46:56.323750 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 23:46:56.388242 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:46:56.388207 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p96dz\" (UniqueName: \"kubernetes.io/projected/8ddd2816-2a86-4118-bdda-d1f50dc650ee-kube-api-access-p96dz\") pod \"postgres-7cdbf6dc44-mtsp7\" (UID: \"8ddd2816-2a86-4118-bdda-d1f50dc650ee\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-mtsp7" Oct 30 23:46:56.530457 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:46:56.530378 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-mtsp7" Oct 30 23:46:56.905959 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:46:56.905934 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-mtsp7"] Oct 30 23:46:56.907068 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:46:56.907042 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ddd2816_2a86_4118_bdda_d1f50dc650ee.slice/crio-9f9a45059bb7f32694748a17a2460eec237b54b7d16c6938304636d5548419fb WatchSource:0}: Error finding container 9f9a45059bb7f32694748a17a2460eec237b54b7d16c6938304636d5548419fb: Status 404 returned error can't find the container with id 9f9a45059bb7f32694748a17a2460eec237b54b7d16c6938304636d5548419fb Oct 30 23:46:57.105926 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:46:57.105892 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-mtsp7" event={"ID":"8ddd2816-2a86-4118-bdda-d1f50dc650ee","Type":"ContainerStarted","Data":"9f9a45059bb7f32694748a17a2460eec237b54b7d16c6938304636d5548419fb"} Oct 30 23:47:05.155384 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:47:05.155274 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-mtsp7" event={"ID":"8ddd2816-2a86-4118-bdda-d1f50dc650ee","Type":"ContainerStarted","Data":"f3bd5b9c8e9030ccb40c0ccd79e4287d2a7e805561d7e4f5be2d2c50665455d8"} Oct 30 23:47:05.155769 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:47:05.155403 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-mtsp7" Oct 30 23:47:05.273897 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:47:05.273842 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-mtsp7" podStartSLOduration=1.510561213 podStartE2EDuration="9.273830342s" totalImagesPullingTime="7.763269129s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:46:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:47:05.273463523 +0000 UTC m=+338.309794384" watchObservedRunningTime="2026-10-30 23:47:05.273830342 +0000 UTC m=+338.310161163" Oct 30 23:47:05.651151 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:47:05.651112 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:47:05.888787 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:47:05.888748 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:47:08.061971 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:47:08.061937 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-xlnnv" Oct 30 23:47:11.191065 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:47:11.191033 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-mtsp7" Oct 30 23:47:18.389066 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:47:18.389025 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:47:18.553264 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:47:18.553225 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:47:18.707003 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:47:18.706961 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:47:18.863176 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:47:18.863133 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:47:19.016519 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:47:19.016420 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:47:19.677746 ip-10-0-141-180 kubenswrapper[4927]: E1030 23:47:19.677709 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sqv69" containerName="extractor" Oct 30 23:47:23.118178 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:47:23.118141 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:47:23.317029 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:47:23.316969 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:47:26.358088 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:47:26.358046 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:47:26.562861 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:47:26.562827 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:47:44.844367 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:47:44.844314 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8t7ww"] Oct 30 23:47:44.851480 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:47:44.851455 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8t7ww" Oct 30 23:47:44.904843 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:47:44.904808 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Oct 30 23:47:44.905096 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:47:44.905076 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Oct 30 23:47:44.905256 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:47:44.905237 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Oct 30 23:47:44.956198 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:47:44.956167 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-zbdd5\"" Oct 30 23:47:44.958588 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:47:44.958560 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8t7ww"] Oct 30 23:47:44.958716 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:47:44.958668 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/939eeff5-dd27-4317-89a0-0355418b01c1-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8t7ww\" (UID: \"939eeff5-dd27-4317-89a0-0355418b01c1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8t7ww" Oct 30 23:47:44.958716 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:47:44.958699 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/939eeff5-dd27-4317-89a0-0355418b01c1-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8t7ww\" (UID: \"939eeff5-dd27-4317-89a0-0355418b01c1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8t7ww" Oct 30 23:47:44.958813 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:47:44.958722 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/939eeff5-dd27-4317-89a0-0355418b01c1-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8t7ww\" (UID: \"939eeff5-dd27-4317-89a0-0355418b01c1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8t7ww" Oct 30 23:47:44.958855 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:47:44.958843 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/939eeff5-dd27-4317-89a0-0355418b01c1-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8t7ww\" (UID: \"939eeff5-dd27-4317-89a0-0355418b01c1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8t7ww" Oct 30 23:47:44.958898 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:47:44.958870 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/939eeff5-dd27-4317-89a0-0355418b01c1-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8t7ww\" (UID: \"939eeff5-dd27-4317-89a0-0355418b01c1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8t7ww" Oct 30 23:47:44.958950 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:47:44.958897 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/939eeff5-dd27-4317-89a0-0355418b01c1-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8t7ww\" (UID: \"939eeff5-dd27-4317-89a0-0355418b01c1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8t7ww" Oct 30 23:47:44.959023 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:47:44.959003 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dzqg4\" (UniqueName: \"kubernetes.io/projected/939eeff5-dd27-4317-89a0-0355418b01c1-kube-api-access-dzqg4\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8t7ww\" (UID: \"939eeff5-dd27-4317-89a0-0355418b01c1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8t7ww" Oct 30 23:47:45.060467 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:47:45.060433 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/939eeff5-dd27-4317-89a0-0355418b01c1-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8t7ww\" (UID: \"939eeff5-dd27-4317-89a0-0355418b01c1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8t7ww" Oct 30 23:47:45.060657 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:47:45.060474 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/939eeff5-dd27-4317-89a0-0355418b01c1-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8t7ww\" (UID: \"939eeff5-dd27-4317-89a0-0355418b01c1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8t7ww" Oct 30 23:47:45.060657 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:47:45.060499 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/939eeff5-dd27-4317-89a0-0355418b01c1-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8t7ww\" (UID: \"939eeff5-dd27-4317-89a0-0355418b01c1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8t7ww" Oct 30 23:47:45.060657 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:47:45.060529 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/939eeff5-dd27-4317-89a0-0355418b01c1-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8t7ww\" (UID: \"939eeff5-dd27-4317-89a0-0355418b01c1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8t7ww" Oct 30 23:47:45.060657 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:47:45.060550 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/939eeff5-dd27-4317-89a0-0355418b01c1-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8t7ww\" (UID: \"939eeff5-dd27-4317-89a0-0355418b01c1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8t7ww" Oct 30 23:47:45.060657 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:47:45.060572 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/939eeff5-dd27-4317-89a0-0355418b01c1-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8t7ww\" (UID: \"939eeff5-dd27-4317-89a0-0355418b01c1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8t7ww" Oct 30 23:47:45.060657 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:47:45.060592 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzqg4\" (UniqueName: \"kubernetes.io/projected/939eeff5-dd27-4317-89a0-0355418b01c1-kube-api-access-dzqg4\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8t7ww\" (UID: \"939eeff5-dd27-4317-89a0-0355418b01c1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8t7ww" Oct 30 23:47:45.061518 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:47:45.061483 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/939eeff5-dd27-4317-89a0-0355418b01c1-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8t7ww\" (UID: \"939eeff5-dd27-4317-89a0-0355418b01c1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8t7ww" Oct 30 23:47:45.062963 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:47:45.062935 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/939eeff5-dd27-4317-89a0-0355418b01c1-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8t7ww\" (UID: \"939eeff5-dd27-4317-89a0-0355418b01c1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8t7ww" Oct 30 23:47:45.063055 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:47:45.062965 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/939eeff5-dd27-4317-89a0-0355418b01c1-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8t7ww\" (UID: \"939eeff5-dd27-4317-89a0-0355418b01c1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8t7ww" Oct 30 23:47:45.063055 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:47:45.063038 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/939eeff5-dd27-4317-89a0-0355418b01c1-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8t7ww\" (UID: \"939eeff5-dd27-4317-89a0-0355418b01c1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8t7ww" Oct 30 23:47:45.063127 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:47:45.063099 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/939eeff5-dd27-4317-89a0-0355418b01c1-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8t7ww\" (UID: \"939eeff5-dd27-4317-89a0-0355418b01c1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8t7ww" Oct 30 23:47:45.123313 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:47:45.123230 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dzqg4\" (UniqueName: \"kubernetes.io/projected/939eeff5-dd27-4317-89a0-0355418b01c1-kube-api-access-dzqg4\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8t7ww\" (UID: \"939eeff5-dd27-4317-89a0-0355418b01c1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8t7ww" Oct 30 23:47:45.124179 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:47:45.124157 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/939eeff5-dd27-4317-89a0-0355418b01c1-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8t7ww\" (UID: \"939eeff5-dd27-4317-89a0-0355418b01c1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8t7ww" Oct 30 23:47:45.164318 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:47:45.164288 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8t7ww" Oct 30 23:47:45.595377 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:47:45.595343 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8t7ww"] Oct 30 23:47:45.596867 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:47:45.596843 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod939eeff5_dd27_4317_89a0_0355418b01c1.slice/crio-7a988c9bc29b18dc6a24363cb9eec7e87dc14e502f6e65567afb1f847d6dd335 WatchSource:0}: Error finding container 7a988c9bc29b18dc6a24363cb9eec7e87dc14e502f6e65567afb1f847d6dd335: Status 404 returned error can't find the container with id 7a988c9bc29b18dc6a24363cb9eec7e87dc14e502f6e65567afb1f847d6dd335 Oct 30 23:47:46.384633 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:47:46.384594 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8t7ww" event={"ID":"939eeff5-dd27-4317-89a0-0355418b01c1","Type":"ContainerStarted","Data":"7a988c9bc29b18dc6a24363cb9eec7e87dc14e502f6e65567afb1f847d6dd335"} Oct 30 23:47:49.252630 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:47:49.252576 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:47:49.406304 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:47:49.406266 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:47:49.561658 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:47:49.561570 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:47:49.684444 ip-10-0-141-180 kubenswrapper[4927]: E1030 23:47:49.684406 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sqv69" containerName="extractor" Oct 30 23:47:49.777005 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:47:49.776859 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:47:49.927580 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:47:49.927543 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:47:50.088277 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:47:50.088238 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:47:50.264297 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:47:50.264217 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:47:50.514942 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:47:50.514857 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 23:47:50.514942 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:47:50.514926 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 23:47:51.419973 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:47:51.419936 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8t7ww" event={"ID":"939eeff5-dd27-4317-89a0-0355418b01c1","Type":"ContainerStarted","Data":"ca973d3bfe144df72d4578f219a666b85df0e30999211af79043431b7bcc2d39"} Oct 30 23:47:51.420453 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:47:51.420087 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8t7ww" Oct 30 23:47:51.421873 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:47:51.421850 4927 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-8t7ww container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 23:47:51.421977 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:47:51.421893 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8t7ww" podUID="939eeff5-dd27-4317-89a0-0355418b01c1" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 23:47:51.454372 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:47:51.454321 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:47:52.425749 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:47:52.425720 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8t7ww" Oct 30 23:47:52.600353 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:47:52.600290 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8t7ww" podStartSLOduration=3.685095934 podStartE2EDuration="8.600276058s" totalImagesPullingTime="4.915180124s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:47:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:47:51.630969962 +0000 UTC m=+384.667300783" watchObservedRunningTime="2026-10-30 23:47:52.600276058 +0000 UTC m=+385.636606880" Oct 30 23:47:53.379805 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:47:53.379752 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:47:53.537042 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:47:53.537007 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:47:55.570496 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:47:55.570447 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:47:55.751104 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:47:55.751071 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:47:55.923325 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:47:55.923289 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:47:56.081242 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:47:56.081208 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:47:56.547234 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:47:56.547201 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:47:56.703116 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:47:56.703076 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:47:56.940200 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:47:56.940158 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:47:57.092134 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:47:57.092099 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:47:57.264779 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:47:57.264656 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:47:57.491305 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:47:57.491265 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:47:57.639578 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:47:57.639494 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:48:07.748219 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:48:07.748099 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:48:08.034357 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:48:08.034238 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:48:10.379506 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:48:10.379457 4927 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-pnm97 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 23:48:10.380018 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:48:10.379529 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-pnm97" podUID="495da7b4-1e63-46fa-ac83-6d3bb75692f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:48:12.530718 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:48:12.530686 4927 generic.go:356] "Generic (PLEG): container finished" podID="495da7b4-1e63-46fa-ac83-6d3bb75692f8" containerID="3c0baafe8176b8985b703751e8d2118f0442fe67f81363c6f1c35c195218077d" exitCode=0 Oct 30 23:48:12.531157 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:48:12.530758 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-pnm97" event={"ID":"495da7b4-1e63-46fa-ac83-6d3bb75692f8","Type":"ContainerDied","Data":"3c0baafe8176b8985b703751e8d2118f0442fe67f81363c6f1c35c195218077d"} Oct 30 23:48:12.531157 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:48:12.531106 4927 scope.go:122] "RemoveContainer" containerID="3c0baafe8176b8985b703751e8d2118f0442fe67f81363c6f1c35c195218077d" Oct 30 23:48:12.532174 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:48:12.532147 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 23:48:12.532230 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:48:12.532216 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 23:48:12.532265 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:48:12.532252 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 23:48:13.540654 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:48:13.540615 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-pnm97" event={"ID":"495da7b4-1e63-46fa-ac83-6d3bb75692f8","Type":"ContainerStarted","Data":"0ac36170caeaed36c8f33fe01cbef7a150c4d67109850ec0c78261b4d55968cf"} Oct 30 23:48:14.474932 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:48:14.474897 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-pnm97" Oct 30 23:48:14.475108 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:48:14.475049 4927 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-pnm97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" start-of-body= Oct 30 23:48:14.475157 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:48:14.475104 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-pnm97" podUID="495da7b4-1e63-46fa-ac83-6d3bb75692f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" Oct 30 23:48:15.475125 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:48:15.475086 4927 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-pnm97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" start-of-body= Oct 30 23:48:15.475587 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:48:15.475143 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-pnm97" podUID="495da7b4-1e63-46fa-ac83-6d3bb75692f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" Oct 30 23:48:16.475791 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:48:16.475757 4927 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-pnm97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" start-of-body= Oct 30 23:48:16.476226 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:48:16.475814 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-pnm97" podUID="495da7b4-1e63-46fa-ac83-6d3bb75692f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" Oct 30 23:48:17.475712 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:48:17.475670 4927 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-pnm97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" start-of-body= Oct 30 23:48:17.475897 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:48:17.475728 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-pnm97" podUID="495da7b4-1e63-46fa-ac83-6d3bb75692f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" Oct 30 23:48:18.475413 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:48:18.475379 4927 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-pnm97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" start-of-body= Oct 30 23:48:18.475591 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:48:18.475442 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-pnm97" podUID="495da7b4-1e63-46fa-ac83-6d3bb75692f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" Oct 30 23:48:19.280624 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:48:19.280588 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:48:19.478604 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:48:19.478575 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-pnm97" Oct 30 23:48:19.478931 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:48:19.478909 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-pnm97" Oct 30 23:48:19.479466 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:48:19.479451 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-pnm97" Oct 30 23:48:19.677060 ip-10-0-141-180 kubenswrapper[4927]: E1030 23:48:19.677016 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sqv69" containerName="extractor" Oct 30 23:48:24.736463 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:48:24.736423 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:48:24.893181 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:48:24.893144 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:48:48.788042 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:48:48.788009 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:48:49.677320 ip-10-0-141-180 kubenswrapper[4927]: E1030 23:48:49.677283 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sqv69" containerName="extractor" Oct 30 23:49:19.612321 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:49:19.612281 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:49:19.677599 ip-10-0-141-180 kubenswrapper[4927]: E1030 23:49:19.677562 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sqv69" containerName="extractor" Oct 30 23:49:28.026911 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:49:28.026871 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:49:28.227859 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:49:28.227819 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:49:46.858828 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:49:46.858735 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:49:47.056200 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:49:47.056158 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:49:49.677562 ip-10-0-141-180 kubenswrapper[4927]: E1030 23:49:49.677526 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sqv69" containerName="extractor" Oct 30 23:50:05.572917 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:50:05.572884 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:50:07.612689 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:50:07.612649 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:50:09.653765 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:50:09.653725 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:50:13.816318 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:50:13.816282 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:50:18.744735 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:50:18.744697 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:50:19.678635 ip-10-0-141-180 kubenswrapper[4927]: E1030 23:50:19.678596 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sqv69" containerName="extractor" Oct 30 23:50:19.959685 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:50:19.959652 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:50:25.185685 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:50:25.185647 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:50:28.548598 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:50:28.548565 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:50:28.803850 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:50:28.803768 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:50:32.109978 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:50:32.109940 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-8s8j7"] Oct 30 23:50:32.113574 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:50:32.113555 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-8s8j7" Oct 30 23:50:32.172057 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:50:32.172020 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-8s8j7"] Oct 30 23:50:32.217356 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:50:32.217305 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:50:32.218236 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:50:32.218176 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Oct 30 23:50:32.218418 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:50:32.218399 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Oct 30 23:50:32.218519 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:50:32.218503 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Oct 30 23:50:32.218591 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:50:32.218578 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-hg7dl\"" Oct 30 23:50:32.240695 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:50:32.240669 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7nnm\" (UniqueName: \"kubernetes.io/projected/2d879158-1e38-47e0-8f8e-9f91e5d213ca-kube-api-access-s7nnm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-8s8j7\" (UID: \"2d879158-1e38-47e0-8f8e-9f91e5d213ca\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-8s8j7" Oct 30 23:50:32.240794 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:50:32.240730 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d879158-1e38-47e0-8f8e-9f91e5d213ca-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-8s8j7\" (UID: \"2d879158-1e38-47e0-8f8e-9f91e5d213ca\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-8s8j7" Oct 30 23:50:32.240835 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:50:32.240791 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d879158-1e38-47e0-8f8e-9f91e5d213ca-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-8s8j7\" (UID: \"2d879158-1e38-47e0-8f8e-9f91e5d213ca\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-8s8j7" Oct 30 23:50:32.240872 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:50:32.240844 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d879158-1e38-47e0-8f8e-9f91e5d213ca-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-8s8j7\" (UID: \"2d879158-1e38-47e0-8f8e-9f91e5d213ca\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-8s8j7" Oct 30 23:50:32.240872 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:50:32.240865 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2d879158-1e38-47e0-8f8e-9f91e5d213ca-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-8s8j7\" (UID: \"2d879158-1e38-47e0-8f8e-9f91e5d213ca\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-8s8j7" Oct 30 23:50:32.240993 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:50:32.240974 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d879158-1e38-47e0-8f8e-9f91e5d213ca-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-8s8j7\" (UID: \"2d879158-1e38-47e0-8f8e-9f91e5d213ca\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-8s8j7" Oct 30 23:50:32.341777 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:50:32.341744 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d879158-1e38-47e0-8f8e-9f91e5d213ca-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-8s8j7\" (UID: \"2d879158-1e38-47e0-8f8e-9f91e5d213ca\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-8s8j7" Oct 30 23:50:32.341777 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:50:32.341779 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d879158-1e38-47e0-8f8e-9f91e5d213ca-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-8s8j7\" (UID: \"2d879158-1e38-47e0-8f8e-9f91e5d213ca\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-8s8j7" Oct 30 23:50:32.342030 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:50:32.341821 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d879158-1e38-47e0-8f8e-9f91e5d213ca-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-8s8j7\" (UID: \"2d879158-1e38-47e0-8f8e-9f91e5d213ca\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-8s8j7" Oct 30 23:50:32.342030 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:50:32.341846 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2d879158-1e38-47e0-8f8e-9f91e5d213ca-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-8s8j7\" (UID: \"2d879158-1e38-47e0-8f8e-9f91e5d213ca\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-8s8j7" Oct 30 23:50:32.342030 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:50:32.341893 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d879158-1e38-47e0-8f8e-9f91e5d213ca-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-8s8j7\" (UID: \"2d879158-1e38-47e0-8f8e-9f91e5d213ca\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-8s8j7" Oct 30 23:50:32.342030 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:50:32.341933 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7nnm\" (UniqueName: \"kubernetes.io/projected/2d879158-1e38-47e0-8f8e-9f91e5d213ca-kube-api-access-s7nnm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-8s8j7\" (UID: \"2d879158-1e38-47e0-8f8e-9f91e5d213ca\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-8s8j7" Oct 30 23:50:32.342405 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:50:32.342365 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d879158-1e38-47e0-8f8e-9f91e5d213ca-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-8s8j7\" (UID: \"2d879158-1e38-47e0-8f8e-9f91e5d213ca\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-8s8j7" Oct 30 23:50:32.342405 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:50:32.342380 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2d879158-1e38-47e0-8f8e-9f91e5d213ca-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-8s8j7\" (UID: \"2d879158-1e38-47e0-8f8e-9f91e5d213ca\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-8s8j7" Oct 30 23:50:32.342574 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:50:32.342429 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d879158-1e38-47e0-8f8e-9f91e5d213ca-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-8s8j7\" (UID: \"2d879158-1e38-47e0-8f8e-9f91e5d213ca\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-8s8j7" Oct 30 23:50:32.344173 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:50:32.344153 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d879158-1e38-47e0-8f8e-9f91e5d213ca-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-8s8j7\" (UID: \"2d879158-1e38-47e0-8f8e-9f91e5d213ca\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-8s8j7" Oct 30 23:50:32.344431 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:50:32.344412 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d879158-1e38-47e0-8f8e-9f91e5d213ca-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-8s8j7\" (UID: \"2d879158-1e38-47e0-8f8e-9f91e5d213ca\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-8s8j7" Oct 30 23:50:32.408407 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:50:32.408370 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7nnm\" (UniqueName: \"kubernetes.io/projected/2d879158-1e38-47e0-8f8e-9f91e5d213ca-kube-api-access-s7nnm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-8s8j7\" (UID: \"2d879158-1e38-47e0-8f8e-9f91e5d213ca\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-8s8j7" Oct 30 23:50:32.426101 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:50:32.426073 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-8s8j7" Oct 30 23:50:32.804034 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:50:32.804003 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-8s8j7"] Oct 30 23:50:32.805705 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:50:32.805677 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d879158_1e38_47e0_8f8e_9f91e5d213ca.slice/crio-6f61beb0c2ae23b46a8a8bc9979588e376ff31a80829b99599a497df9504e68c WatchSource:0}: Error finding container 6f61beb0c2ae23b46a8a8bc9979588e376ff31a80829b99599a497df9504e68c: Status 404 returned error can't find the container with id 6f61beb0c2ae23b46a8a8bc9979588e376ff31a80829b99599a497df9504e68c Oct 30 23:50:33.263584 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:50:33.263551 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-8s8j7" event={"ID":"2d879158-1e38-47e0-8f8e-9f91e5d213ca","Type":"ContainerStarted","Data":"6f61beb0c2ae23b46a8a8bc9979588e376ff31a80829b99599a497df9504e68c"} Oct 30 23:50:35.274066 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:50:35.274026 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-8s8j7" event={"ID":"2d879158-1e38-47e0-8f8e-9f91e5d213ca","Type":"ContainerStarted","Data":"2df26c5f0297fe6661a398bef2831415faa2572c449e987bd72ad995a48b5e23"} Oct 30 23:50:35.394934 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:50:35.394884 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-8s8j7" podStartSLOduration=1.932490648 podStartE2EDuration="3.394870721s" totalImagesPullingTime="1.462380073s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:50:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:50:35.39378899 +0000 UTC m=+548.430119812" watchObservedRunningTime="2026-10-30 23:50:35.394870721 +0000 UTC m=+548.431201544" Oct 30 23:50:35.426882 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:50:35.426849 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-8s8j7" Oct 30 23:50:36.282453 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:50:36.282422 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:50:42.426995 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:50:42.426961 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-8s8j7" Oct 30 23:50:42.439064 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:50:42.439045 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-8s8j7" Oct 30 23:50:43.332738 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:50:43.332708 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-8s8j7" Oct 30 23:50:49.677669 ip-10-0-141-180 kubenswrapper[4927]: E1030 23:50:49.677632 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sqv69" containerName="extractor" Oct 30 23:50:53.808242 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:50:53.808200 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-754bf99fd5-9tr2s"] Oct 30 23:50:53.814532 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:50:53.814507 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-754bf99fd5-9tr2s" Oct 30 23:50:53.865478 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:50:53.865442 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-754bf99fd5-9tr2s"] Oct 30 23:50:53.914240 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:50:53.914210 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-dn9sl\"" Oct 30 23:50:53.919617 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:50:53.919599 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Oct 30 23:50:53.928022 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:50:53.927996 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zllrw\" (UniqueName: \"kubernetes.io/projected/cd6cd72d-4a9f-43fd-bd87-480fea8a8773-kube-api-access-zllrw\") pod \"authorino-754bf99fd5-9tr2s\" (UID: \"cd6cd72d-4a9f-43fd-bd87-480fea8a8773\") " pod="kuadrant-system/authorino-754bf99fd5-9tr2s" Oct 30 23:50:53.928112 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:50:53.928072 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/cd6cd72d-4a9f-43fd-bd87-480fea8a8773-tls-cert\") pod \"authorino-754bf99fd5-9tr2s\" (UID: \"cd6cd72d-4a9f-43fd-bd87-480fea8a8773\") " pod="kuadrant-system/authorino-754bf99fd5-9tr2s" Oct 30 23:50:54.028679 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:50:54.028637 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/cd6cd72d-4a9f-43fd-bd87-480fea8a8773-tls-cert\") pod \"authorino-754bf99fd5-9tr2s\" (UID: \"cd6cd72d-4a9f-43fd-bd87-480fea8a8773\") " pod="kuadrant-system/authorino-754bf99fd5-9tr2s" Oct 30 23:50:54.028870 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:50:54.028710 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zllrw\" (UniqueName: \"kubernetes.io/projected/cd6cd72d-4a9f-43fd-bd87-480fea8a8773-kube-api-access-zllrw\") pod \"authorino-754bf99fd5-9tr2s\" (UID: \"cd6cd72d-4a9f-43fd-bd87-480fea8a8773\") " pod="kuadrant-system/authorino-754bf99fd5-9tr2s" Oct 30 23:50:54.030996 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:50:54.030971 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/cd6cd72d-4a9f-43fd-bd87-480fea8a8773-tls-cert\") pod \"authorino-754bf99fd5-9tr2s\" (UID: \"cd6cd72d-4a9f-43fd-bd87-480fea8a8773\") " pod="kuadrant-system/authorino-754bf99fd5-9tr2s" Oct 30 23:50:54.090967 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:50:54.090891 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zllrw\" (UniqueName: \"kubernetes.io/projected/cd6cd72d-4a9f-43fd-bd87-480fea8a8773-kube-api-access-zllrw\") pod \"authorino-754bf99fd5-9tr2s\" (UID: \"cd6cd72d-4a9f-43fd-bd87-480fea8a8773\") " pod="kuadrant-system/authorino-754bf99fd5-9tr2s" Oct 30 23:50:54.127888 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:50:54.127855 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-754bf99fd5-9tr2s" Oct 30 23:50:54.130775 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:50:54.130743 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:50:54.291978 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:50:54.291937 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:50:54.507797 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:50:54.507773 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-754bf99fd5-9tr2s"] Oct 30 23:50:54.509689 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:50:54.509662 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcd6cd72d_4a9f_43fd_bd87_480fea8a8773.slice/crio-b4319925c2c1193c736487c4ab33e30468ed8418c1523fb8a43f58918021371e WatchSource:0}: Error finding container b4319925c2c1193c736487c4ab33e30468ed8418c1523fb8a43f58918021371e: Status 404 returned error can't find the container with id b4319925c2c1193c736487c4ab33e30468ed8418c1523fb8a43f58918021371e Oct 30 23:50:55.385939 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:50:55.385893 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-754bf99fd5-9tr2s" event={"ID":"cd6cd72d-4a9f-43fd-bd87-480fea8a8773","Type":"ContainerStarted","Data":"b4319925c2c1193c736487c4ab33e30468ed8418c1523fb8a43f58918021371e"} Oct 30 23:50:57.398940 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:50:57.398890 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-754bf99fd5-9tr2s" event={"ID":"cd6cd72d-4a9f-43fd-bd87-480fea8a8773","Type":"ContainerStarted","Data":"43e594eabc9e2c42f01f4c99235dfeb6c8a824df1ce2390aad7434c857925f87"} Oct 30 23:50:57.524084 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:50:57.524028 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-754bf99fd5-9tr2s" podStartSLOduration=2.571287844 podStartE2EDuration="4.52401538s" totalImagesPullingTime="1.952727536s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:50:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:50:57.522628592 +0000 UTC m=+570.558959415" watchObservedRunningTime="2026-10-30 23:50:57.52401538 +0000 UTC m=+570.560346200" Oct 30 23:51:00.210130 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:51:00.210093 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:51:01.021509 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:51:01.021473 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:51:01.196522 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:51:01.196476 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:51:01.434789 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:51:01.434746 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-njw5h"] Oct 30 23:51:01.438594 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:51:01.438570 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-njw5h" Oct 30 23:51:01.494973 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:51:01.494903 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-njw5h"] Oct 30 23:51:01.501603 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:51:01.501564 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/712b9e4e-651e-403a-959f-f534d0f2a7b9-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-njw5h\" (UID: \"712b9e4e-651e-403a-959f-f534d0f2a7b9\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-njw5h" Oct 30 23:51:01.501772 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:51:01.501622 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6znj\" (UniqueName: \"kubernetes.io/projected/712b9e4e-651e-403a-959f-f534d0f2a7b9-kube-api-access-j6znj\") pod \"payload-pre-processing-7b98bb678c-njw5h\" (UID: \"712b9e4e-651e-403a-959f-f534d0f2a7b9\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-njw5h" Oct 30 23:51:01.542184 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:51:01.542140 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Oct 30 23:51:01.543134 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:51:01.543108 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-sckbv\"" Oct 30 23:51:01.579131 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:51:01.579091 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-7bb9489676-2kpl5"] Oct 30 23:51:01.582779 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:51:01.582760 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-7bb9489676-2kpl5" Oct 30 23:51:01.602625 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:51:01.602590 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/712b9e4e-651e-403a-959f-f534d0f2a7b9-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-njw5h\" (UID: \"712b9e4e-651e-403a-959f-f534d0f2a7b9\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-njw5h" Oct 30 23:51:01.602625 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:51:01.602621 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6znj\" (UniqueName: \"kubernetes.io/projected/712b9e4e-651e-403a-959f-f534d0f2a7b9-kube-api-access-j6znj\") pod \"payload-pre-processing-7b98bb678c-njw5h\" (UID: \"712b9e4e-651e-403a-959f-f534d0f2a7b9\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-njw5h" Oct 30 23:51:01.603211 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:51:01.603186 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/712b9e4e-651e-403a-959f-f534d0f2a7b9-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-njw5h\" (UID: \"712b9e4e-651e-403a-959f-f534d0f2a7b9\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-njw5h" Oct 30 23:51:01.638987 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:51:01.638957 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-7bb9489676-2kpl5"] Oct 30 23:51:01.664726 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:51:01.664695 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6znj\" (UniqueName: \"kubernetes.io/projected/712b9e4e-651e-403a-959f-f534d0f2a7b9-kube-api-access-j6znj\") pod \"payload-pre-processing-7b98bb678c-njw5h\" (UID: \"712b9e4e-651e-403a-959f-f534d0f2a7b9\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-njw5h" Oct 30 23:51:01.704064 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:51:01.703986 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lk54w\" (UniqueName: \"kubernetes.io/projected/4130bb96-d382-4b5e-b310-dcef9bf001de-kube-api-access-lk54w\") pod \"payload-processing-7bb9489676-2kpl5\" (UID: \"4130bb96-d382-4b5e-b310-dcef9bf001de\") " pod="openshift-ingress/payload-processing-7bb9489676-2kpl5" Oct 30 23:51:01.704064 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:51:01.704029 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4130bb96-d382-4b5e-b310-dcef9bf001de-plugins-config-volume\") pod \"payload-processing-7bb9489676-2kpl5\" (UID: \"4130bb96-d382-4b5e-b310-dcef9bf001de\") " pod="openshift-ingress/payload-processing-7bb9489676-2kpl5" Oct 30 23:51:01.752029 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:51:01.751994 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-njw5h" Oct 30 23:51:01.805405 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:51:01.805368 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lk54w\" (UniqueName: \"kubernetes.io/projected/4130bb96-d382-4b5e-b310-dcef9bf001de-kube-api-access-lk54w\") pod \"payload-processing-7bb9489676-2kpl5\" (UID: \"4130bb96-d382-4b5e-b310-dcef9bf001de\") " pod="openshift-ingress/payload-processing-7bb9489676-2kpl5" Oct 30 23:51:01.805595 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:51:01.805428 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4130bb96-d382-4b5e-b310-dcef9bf001de-plugins-config-volume\") pod \"payload-processing-7bb9489676-2kpl5\" (UID: \"4130bb96-d382-4b5e-b310-dcef9bf001de\") " pod="openshift-ingress/payload-processing-7bb9489676-2kpl5" Oct 30 23:51:01.806149 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:51:01.806124 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4130bb96-d382-4b5e-b310-dcef9bf001de-plugins-config-volume\") pod \"payload-processing-7bb9489676-2kpl5\" (UID: \"4130bb96-d382-4b5e-b310-dcef9bf001de\") " pod="openshift-ingress/payload-processing-7bb9489676-2kpl5" Oct 30 23:51:01.868510 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:51:01.868477 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lk54w\" (UniqueName: \"kubernetes.io/projected/4130bb96-d382-4b5e-b310-dcef9bf001de-kube-api-access-lk54w\") pod \"payload-processing-7bb9489676-2kpl5\" (UID: \"4130bb96-d382-4b5e-b310-dcef9bf001de\") " pod="openshift-ingress/payload-processing-7bb9489676-2kpl5" Oct 30 23:51:01.895717 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:51:01.895685 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-7bb9489676-2kpl5" Oct 30 23:51:02.142054 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:51:02.142022 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-njw5h"] Oct 30 23:51:02.143694 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:51:02.143666 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod712b9e4e_651e_403a_959f_f534d0f2a7b9.slice/crio-0b7e5d2f2ed8da1fd997a8b89ecf0dbf8e6ca1683cb2ec0fad0056035dd978ae WatchSource:0}: Error finding container 0b7e5d2f2ed8da1fd997a8b89ecf0dbf8e6ca1683cb2ec0fad0056035dd978ae: Status 404 returned error can't find the container with id 0b7e5d2f2ed8da1fd997a8b89ecf0dbf8e6ca1683cb2ec0fad0056035dd978ae Oct 30 23:51:02.272355 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:51:02.272152 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-7bb9489676-2kpl5"] Oct 30 23:51:02.304280 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:51:02.303897 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4130bb96_d382_4b5e_b310_dcef9bf001de.slice/crio-773bb121739bcc6e393f755e644a76336604670e4f6c8e1be5ff6aefe5598769 WatchSource:0}: Error finding container 773bb121739bcc6e393f755e644a76336604670e4f6c8e1be5ff6aefe5598769: Status 404 returned error can't find the container with id 773bb121739bcc6e393f755e644a76336604670e4f6c8e1be5ff6aefe5598769 Oct 30 23:51:02.428491 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:51:02.428454 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-7bb9489676-2kpl5" event={"ID":"4130bb96-d382-4b5e-b310-dcef9bf001de","Type":"ContainerStarted","Data":"773bb121739bcc6e393f755e644a76336604670e4f6c8e1be5ff6aefe5598769"} Oct 30 23:51:02.429987 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:51:02.429955 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-njw5h" event={"ID":"712b9e4e-651e-403a-959f-f534d0f2a7b9","Type":"ContainerStarted","Data":"0b7e5d2f2ed8da1fd997a8b89ecf0dbf8e6ca1683cb2ec0fad0056035dd978ae"} Oct 30 23:51:05.454243 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:51:05.454208 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-7bb9489676-2kpl5" event={"ID":"4130bb96-d382-4b5e-b310-dcef9bf001de","Type":"ContainerStarted","Data":"1d262506e3c53daa85282ea088412ba250a6fd0ac818d480e6593703688c7eb0"} Oct 30 23:51:05.454702 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:51:05.454312 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-7bb9489676-2kpl5" Oct 30 23:51:05.455969 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:51:05.455902 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-njw5h" event={"ID":"712b9e4e-651e-403a-959f-f534d0f2a7b9","Type":"ContainerStarted","Data":"2888d64e0e760b0bf63e377bc7ea0193160cda4d507cb6d903b3728acfa4974f"} Oct 30 23:51:05.455969 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:51:05.455952 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-njw5h" Oct 30 23:51:05.573379 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:51:05.573302 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-7bb9489676-2kpl5" podStartSLOduration=1.6467088909999998 podStartE2EDuration="4.573290273s" totalImagesPullingTime="2.926581382s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:51:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:51:05.571136402 +0000 UTC m=+578.607467225" watchObservedRunningTime="2026-10-30 23:51:05.573290273 +0000 UTC m=+578.609621094" Oct 30 23:51:05.690876 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:51:05.690812 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7b98bb678c-njw5h" podStartSLOduration=1.630017573 podStartE2EDuration="4.690793422s" totalImagesPullingTime="3.060775849s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:51:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:51:05.689505662 +0000 UTC m=+578.725836485" watchObservedRunningTime="2026-10-30 23:51:05.690793422 +0000 UTC m=+578.727124244" Oct 30 23:51:15.003637 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:51:15.003559 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-lwhxc"] Oct 30 23:51:15.006861 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:51:15.006844 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-lwhxc" Oct 30 23:51:15.104562 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:51:15.104529 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-lwhxc"] Oct 30 23:51:15.136129 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:51:15.136098 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8c6381b-c2ea-4a6c-8929-a7dd5fb29514-credential-socket\") pod \"maas-default-gateway-openshift-default-8f6f87f59-lwhxc\" (UID: \"e8c6381b-c2ea-4a6c-8929-a7dd5fb29514\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-lwhxc" Oct 30 23:51:15.136267 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:51:15.136142 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8c6381b-c2ea-4a6c-8929-a7dd5fb29514-istio-data\") pod \"maas-default-gateway-openshift-default-8f6f87f59-lwhxc\" (UID: \"e8c6381b-c2ea-4a6c-8929-a7dd5fb29514\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-lwhxc" Oct 30 23:51:15.136267 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:51:15.136161 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cbwl9\" (UniqueName: \"kubernetes.io/projected/e8c6381b-c2ea-4a6c-8929-a7dd5fb29514-kube-api-access-cbwl9\") pod \"maas-default-gateway-openshift-default-8f6f87f59-lwhxc\" (UID: \"e8c6381b-c2ea-4a6c-8929-a7dd5fb29514\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-lwhxc" Oct 30 23:51:15.136382 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:51:15.136283 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8c6381b-c2ea-4a6c-8929-a7dd5fb29514-workload-certs\") pod \"maas-default-gateway-openshift-default-8f6f87f59-lwhxc\" (UID: \"e8c6381b-c2ea-4a6c-8929-a7dd5fb29514\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-lwhxc" Oct 30 23:51:15.136382 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:51:15.136311 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8c6381b-c2ea-4a6c-8929-a7dd5fb29514-istio-token\") pod \"maas-default-gateway-openshift-default-8f6f87f59-lwhxc\" (UID: \"e8c6381b-c2ea-4a6c-8929-a7dd5fb29514\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-lwhxc" Oct 30 23:51:15.136382 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:51:15.136351 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8c6381b-c2ea-4a6c-8929-a7dd5fb29514-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8f6f87f59-lwhxc\" (UID: \"e8c6381b-c2ea-4a6c-8929-a7dd5fb29514\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-lwhxc" Oct 30 23:51:15.136382 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:51:15.136376 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8c6381b-c2ea-4a6c-8929-a7dd5fb29514-workload-socket\") pod \"maas-default-gateway-openshift-default-8f6f87f59-lwhxc\" (UID: \"e8c6381b-c2ea-4a6c-8929-a7dd5fb29514\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-lwhxc" Oct 30 23:51:15.136530 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:51:15.136405 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8c6381b-c2ea-4a6c-8929-a7dd5fb29514-istio-envoy\") pod \"maas-default-gateway-openshift-default-8f6f87f59-lwhxc\" (UID: \"e8c6381b-c2ea-4a6c-8929-a7dd5fb29514\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-lwhxc" Oct 30 23:51:15.136530 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:51:15.136429 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8c6381b-c2ea-4a6c-8929-a7dd5fb29514-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8f6f87f59-lwhxc\" (UID: \"e8c6381b-c2ea-4a6c-8929-a7dd5fb29514\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-lwhxc" Oct 30 23:51:15.237035 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:51:15.236997 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8c6381b-c2ea-4a6c-8929-a7dd5fb29514-credential-socket\") pod \"maas-default-gateway-openshift-default-8f6f87f59-lwhxc\" (UID: \"e8c6381b-c2ea-4a6c-8929-a7dd5fb29514\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-lwhxc" Oct 30 23:51:15.237198 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:51:15.237132 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8c6381b-c2ea-4a6c-8929-a7dd5fb29514-istio-data\") pod \"maas-default-gateway-openshift-default-8f6f87f59-lwhxc\" (UID: \"e8c6381b-c2ea-4a6c-8929-a7dd5fb29514\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-lwhxc" Oct 30 23:51:15.237198 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:51:15.237152 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbwl9\" (UniqueName: \"kubernetes.io/projected/e8c6381b-c2ea-4a6c-8929-a7dd5fb29514-kube-api-access-cbwl9\") pod \"maas-default-gateway-openshift-default-8f6f87f59-lwhxc\" (UID: \"e8c6381b-c2ea-4a6c-8929-a7dd5fb29514\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-lwhxc" Oct 30 23:51:15.237198 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:51:15.237190 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8c6381b-c2ea-4a6c-8929-a7dd5fb29514-workload-certs\") pod \"maas-default-gateway-openshift-default-8f6f87f59-lwhxc\" (UID: \"e8c6381b-c2ea-4a6c-8929-a7dd5fb29514\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-lwhxc" Oct 30 23:51:15.237422 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:51:15.237215 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8c6381b-c2ea-4a6c-8929-a7dd5fb29514-istio-token\") pod \"maas-default-gateway-openshift-default-8f6f87f59-lwhxc\" (UID: \"e8c6381b-c2ea-4a6c-8929-a7dd5fb29514\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-lwhxc" Oct 30 23:51:15.237422 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:51:15.237254 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8c6381b-c2ea-4a6c-8929-a7dd5fb29514-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8f6f87f59-lwhxc\" (UID: \"e8c6381b-c2ea-4a6c-8929-a7dd5fb29514\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-lwhxc" Oct 30 23:51:15.237422 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:51:15.237280 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8c6381b-c2ea-4a6c-8929-a7dd5fb29514-workload-socket\") pod \"maas-default-gateway-openshift-default-8f6f87f59-lwhxc\" (UID: \"e8c6381b-c2ea-4a6c-8929-a7dd5fb29514\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-lwhxc" Oct 30 23:51:15.237422 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:51:15.237315 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8c6381b-c2ea-4a6c-8929-a7dd5fb29514-istio-envoy\") pod \"maas-default-gateway-openshift-default-8f6f87f59-lwhxc\" (UID: \"e8c6381b-c2ea-4a6c-8929-a7dd5fb29514\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-lwhxc" Oct 30 23:51:15.237422 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:51:15.237363 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8c6381b-c2ea-4a6c-8929-a7dd5fb29514-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8f6f87f59-lwhxc\" (UID: \"e8c6381b-c2ea-4a6c-8929-a7dd5fb29514\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-lwhxc" Oct 30 23:51:15.237754 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:51:15.237596 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8c6381b-c2ea-4a6c-8929-a7dd5fb29514-istio-data\") pod \"maas-default-gateway-openshift-default-8f6f87f59-lwhxc\" (UID: \"e8c6381b-c2ea-4a6c-8929-a7dd5fb29514\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-lwhxc" Oct 30 23:51:15.237754 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:51:15.237615 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8c6381b-c2ea-4a6c-8929-a7dd5fb29514-workload-certs\") pod \"maas-default-gateway-openshift-default-8f6f87f59-lwhxc\" (UID: \"e8c6381b-c2ea-4a6c-8929-a7dd5fb29514\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-lwhxc" Oct 30 23:51:15.237754 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:51:15.237676 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8c6381b-c2ea-4a6c-8929-a7dd5fb29514-credential-socket\") pod \"maas-default-gateway-openshift-default-8f6f87f59-lwhxc\" (UID: \"e8c6381b-c2ea-4a6c-8929-a7dd5fb29514\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-lwhxc" Oct 30 23:51:15.237911 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:51:15.237759 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8c6381b-c2ea-4a6c-8929-a7dd5fb29514-workload-socket\") pod \"maas-default-gateway-openshift-default-8f6f87f59-lwhxc\" (UID: \"e8c6381b-c2ea-4a6c-8929-a7dd5fb29514\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-lwhxc" Oct 30 23:51:15.238224 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:51:15.238204 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8c6381b-c2ea-4a6c-8929-a7dd5fb29514-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8f6f87f59-lwhxc\" (UID: \"e8c6381b-c2ea-4a6c-8929-a7dd5fb29514\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-lwhxc" Oct 30 23:51:15.239784 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:51:15.239758 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8c6381b-c2ea-4a6c-8929-a7dd5fb29514-istio-envoy\") pod \"maas-default-gateway-openshift-default-8f6f87f59-lwhxc\" (UID: \"e8c6381b-c2ea-4a6c-8929-a7dd5fb29514\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-lwhxc" Oct 30 23:51:15.240021 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:51:15.239998 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8c6381b-c2ea-4a6c-8929-a7dd5fb29514-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8f6f87f59-lwhxc\" (UID: \"e8c6381b-c2ea-4a6c-8929-a7dd5fb29514\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-lwhxc" Oct 30 23:51:15.300730 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:51:15.300654 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8c6381b-c2ea-4a6c-8929-a7dd5fb29514-istio-token\") pod \"maas-default-gateway-openshift-default-8f6f87f59-lwhxc\" (UID: \"e8c6381b-c2ea-4a6c-8929-a7dd5fb29514\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-lwhxc" Oct 30 23:51:15.300858 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:51:15.300770 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbwl9\" (UniqueName: \"kubernetes.io/projected/e8c6381b-c2ea-4a6c-8929-a7dd5fb29514-kube-api-access-cbwl9\") pod \"maas-default-gateway-openshift-default-8f6f87f59-lwhxc\" (UID: \"e8c6381b-c2ea-4a6c-8929-a7dd5fb29514\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-lwhxc" Oct 30 23:51:15.319893 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:51:15.319868 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-lwhxc" Oct 30 23:51:15.749009 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:51:15.748982 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-lwhxc"] Oct 30 23:51:15.750301 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:51:15.750270 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8c6381b_c2ea_4a6c_8929_a7dd5fb29514.slice/crio-87f5969e481b8061de62b4bb167f1a8f7f60e7e3b7d49ee410a78a77f54574df WatchSource:0}: Error finding container 87f5969e481b8061de62b4bb167f1a8f7f60e7e3b7d49ee410a78a77f54574df: Status 404 returned error can't find the container with id 87f5969e481b8061de62b4bb167f1a8f7f60e7e3b7d49ee410a78a77f54574df Oct 30 23:51:15.752498 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:51:15.752454 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 23:51:15.752597 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:51:15.752535 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 23:51:15.752597 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:51:15.752583 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 23:51:16.464834 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:51:16.464804 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-7bb9489676-2kpl5" Oct 30 23:51:16.465348 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:51:16.464875 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-njw5h" Oct 30 23:51:16.538282 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:51:16.538245 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-lwhxc" event={"ID":"e8c6381b-c2ea-4a6c-8929-a7dd5fb29514","Type":"ContainerStarted","Data":"c5762b14869d1c58ae84ad0f8e3529ad7453e6139a45ad6e7edcf1a220922218"} Oct 30 23:51:16.538471 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:51:16.538300 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-lwhxc" event={"ID":"e8c6381b-c2ea-4a6c-8929-a7dd5fb29514","Type":"ContainerStarted","Data":"87f5969e481b8061de62b4bb167f1a8f7f60e7e3b7d49ee410a78a77f54574df"} Oct 30 23:51:16.856779 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:51:16.856682 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-lwhxc" podStartSLOduration=2.856667179 podStartE2EDuration="2.856667179s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:51:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:51:16.85371831 +0000 UTC m=+589.890049132" watchObservedRunningTime="2026-10-30 23:51:16.856667179 +0000 UTC m=+589.892998000" Oct 30 23:51:17.332381 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:51:17.326741 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-lwhxc" Oct 30 23:51:17.332381 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:51:17.328980 4927 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-lwhxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Oct 30 23:51:17.332381 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:51:17.329033 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-lwhxc" podUID="e8c6381b-c2ea-4a6c-8929-a7dd5fb29514" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Oct 30 23:51:18.332717 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:51:18.332683 4927 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-lwhxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Oct 30 23:51:18.333368 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:51:18.333302 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-lwhxc" podUID="e8c6381b-c2ea-4a6c-8929-a7dd5fb29514" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Oct 30 23:51:19.332793 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:51:19.332747 4927 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-lwhxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Oct 30 23:51:19.333225 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:51:19.332852 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-lwhxc" podUID="e8c6381b-c2ea-4a6c-8929-a7dd5fb29514" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Oct 30 23:51:19.677442 ip-10-0-141-180 kubenswrapper[4927]: E1030 23:51:19.677406 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sqv69" containerName="extractor" Oct 30 23:51:20.301150 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:51:20.301105 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:51:20.323356 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:51:20.321638 4927 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-lwhxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Oct 30 23:51:20.323356 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:51:20.321706 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-lwhxc" podUID="e8c6381b-c2ea-4a6c-8929-a7dd5fb29514" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Oct 30 23:51:21.323356 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:51:21.320432 4927 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-lwhxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Oct 30 23:51:21.323356 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:51:21.320497 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-lwhxc" podUID="e8c6381b-c2ea-4a6c-8929-a7dd5fb29514" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Oct 30 23:51:22.320569 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:51:22.320526 4927 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-lwhxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Oct 30 23:51:22.320793 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:51:22.320578 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-lwhxc" podUID="e8c6381b-c2ea-4a6c-8929-a7dd5fb29514" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Oct 30 23:51:23.944945 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:51:23.944903 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-lwhxc" Oct 30 23:51:23.945513 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:51:23.945279 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-lwhxc" Oct 30 23:51:23.946067 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:51:23.946050 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-lwhxc" Oct 30 23:51:24.819665 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:51:24.819625 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:51:25.309460 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:51:25.309428 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:51:26.920317 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:51:26.920284 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:51:27.149978 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:51:27.149947 4927 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-pqt2c_6382dfb5-a363-47a2-bc79-0b96f7bec0c3/console-operator/3.log" Oct 30 23:51:27.152151 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:51:27.152131 4927 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-pqt2c_6382dfb5-a363-47a2-bc79-0b96f7bec0c3/console-operator/3.log" Oct 30 23:51:27.173409 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:51:27.173290 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:51:28.560657 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:51:28.560621 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:51:28.733087 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:51:28.733044 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:51:49.677512 ip-10-0-141-180 kubenswrapper[4927]: E1030 23:51:49.677476 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sqv69" containerName="extractor" Oct 30 23:51:50.926199 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:51:50.926161 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:51:51.124968 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:51:51.124933 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:51:53.713590 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:51:53.713548 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:51:53.918971 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:51:53.918935 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:51:54.380517 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:51:54.380484 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:51:54.623249 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:51:54.623215 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-jqc5p"] Oct 30 23:51:54.627879 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:51:54.627852 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-jqc5p" Oct 30 23:51:54.686550 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:51:54.686502 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a23bdae1-1e2a-4e26-8e2b-43d7b50a800e-workload-certs\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-jqc5p\" (UID: \"a23bdae1-1e2a-4e26-8e2b-43d7b50a800e\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-jqc5p" Oct 30 23:51:54.686803 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:51:54.686786 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a23bdae1-1e2a-4e26-8e2b-43d7b50a800e-istio-podinfo\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-jqc5p\" (UID: \"a23bdae1-1e2a-4e26-8e2b-43d7b50a800e\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-jqc5p" Oct 30 23:51:54.686974 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:51:54.686958 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a23bdae1-1e2a-4e26-8e2b-43d7b50a800e-credential-socket\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-jqc5p\" (UID: \"a23bdae1-1e2a-4e26-8e2b-43d7b50a800e\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-jqc5p" Oct 30 23:51:54.687096 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:51:54.687075 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86d72\" (UniqueName: \"kubernetes.io/projected/a23bdae1-1e2a-4e26-8e2b-43d7b50a800e-kube-api-access-86d72\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-jqc5p\" (UID: \"a23bdae1-1e2a-4e26-8e2b-43d7b50a800e\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-jqc5p" Oct 30 23:51:54.687219 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:51:54.687196 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a23bdae1-1e2a-4e26-8e2b-43d7b50a800e-workload-socket\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-jqc5p\" (UID: \"a23bdae1-1e2a-4e26-8e2b-43d7b50a800e\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-jqc5p" Oct 30 23:51:54.687371 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:51:54.687225 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a23bdae1-1e2a-4e26-8e2b-43d7b50a800e-istio-data\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-jqc5p\" (UID: \"a23bdae1-1e2a-4e26-8e2b-43d7b50a800e\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-jqc5p" Oct 30 23:51:54.687371 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:51:54.687278 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a23bdae1-1e2a-4e26-8e2b-43d7b50a800e-istio-envoy\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-jqc5p\" (UID: \"a23bdae1-1e2a-4e26-8e2b-43d7b50a800e\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-jqc5p" Oct 30 23:51:54.687371 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:51:54.687306 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a23bdae1-1e2a-4e26-8e2b-43d7b50a800e-istio-token\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-jqc5p\" (UID: \"a23bdae1-1e2a-4e26-8e2b-43d7b50a800e\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-jqc5p" Oct 30 23:51:54.687576 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:51:54.687424 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a23bdae1-1e2a-4e26-8e2b-43d7b50a800e-istiod-ca-cert\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-jqc5p\" (UID: \"a23bdae1-1e2a-4e26-8e2b-43d7b50a800e\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-jqc5p" Oct 30 23:51:54.698783 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:51:54.698752 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-jqc5p"] Oct 30 23:51:54.729007 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:51:54.728972 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:51:54.758736 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:51:54.758704 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-986d37-openshift-default-dockercfg-rm9vh\"" Oct 30 23:51:54.788562 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:51:54.788528 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a23bdae1-1e2a-4e26-8e2b-43d7b50a800e-istio-envoy\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-jqc5p\" (UID: \"a23bdae1-1e2a-4e26-8e2b-43d7b50a800e\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-jqc5p" Oct 30 23:51:54.788765 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:51:54.788576 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a23bdae1-1e2a-4e26-8e2b-43d7b50a800e-istio-token\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-jqc5p\" (UID: \"a23bdae1-1e2a-4e26-8e2b-43d7b50a800e\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-jqc5p" Oct 30 23:51:54.788765 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:51:54.788604 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a23bdae1-1e2a-4e26-8e2b-43d7b50a800e-istiod-ca-cert\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-jqc5p\" (UID: \"a23bdae1-1e2a-4e26-8e2b-43d7b50a800e\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-jqc5p" Oct 30 23:51:54.788765 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:51:54.788648 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a23bdae1-1e2a-4e26-8e2b-43d7b50a800e-workload-certs\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-jqc5p\" (UID: \"a23bdae1-1e2a-4e26-8e2b-43d7b50a800e\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-jqc5p" Oct 30 23:51:54.788765 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:51:54.788682 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a23bdae1-1e2a-4e26-8e2b-43d7b50a800e-istio-podinfo\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-jqc5p\" (UID: \"a23bdae1-1e2a-4e26-8e2b-43d7b50a800e\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-jqc5p" Oct 30 23:51:54.789015 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:51:54.788771 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a23bdae1-1e2a-4e26-8e2b-43d7b50a800e-credential-socket\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-jqc5p\" (UID: \"a23bdae1-1e2a-4e26-8e2b-43d7b50a800e\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-jqc5p" Oct 30 23:51:54.789015 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:51:54.788810 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86d72\" (UniqueName: \"kubernetes.io/projected/a23bdae1-1e2a-4e26-8e2b-43d7b50a800e-kube-api-access-86d72\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-jqc5p\" (UID: \"a23bdae1-1e2a-4e26-8e2b-43d7b50a800e\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-jqc5p" Oct 30 23:51:54.789015 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:51:54.788851 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a23bdae1-1e2a-4e26-8e2b-43d7b50a800e-workload-socket\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-jqc5p\" (UID: \"a23bdae1-1e2a-4e26-8e2b-43d7b50a800e\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-jqc5p" Oct 30 23:51:54.789015 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:51:54.788885 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a23bdae1-1e2a-4e26-8e2b-43d7b50a800e-istio-data\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-jqc5p\" (UID: \"a23bdae1-1e2a-4e26-8e2b-43d7b50a800e\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-jqc5p" Oct 30 23:51:54.789469 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:51:54.789442 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a23bdae1-1e2a-4e26-8e2b-43d7b50a800e-istio-data\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-jqc5p\" (UID: \"a23bdae1-1e2a-4e26-8e2b-43d7b50a800e\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-jqc5p" Oct 30 23:51:54.789950 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:51:54.789907 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a23bdae1-1e2a-4e26-8e2b-43d7b50a800e-workload-socket\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-jqc5p\" (UID: \"a23bdae1-1e2a-4e26-8e2b-43d7b50a800e\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-jqc5p" Oct 30 23:51:54.790070 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:51:54.789959 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a23bdae1-1e2a-4e26-8e2b-43d7b50a800e-credential-socket\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-jqc5p\" (UID: \"a23bdae1-1e2a-4e26-8e2b-43d7b50a800e\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-jqc5p" Oct 30 23:51:54.790143 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:51:54.790094 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a23bdae1-1e2a-4e26-8e2b-43d7b50a800e-istiod-ca-cert\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-jqc5p\" (UID: \"a23bdae1-1e2a-4e26-8e2b-43d7b50a800e\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-jqc5p" Oct 30 23:51:54.790422 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:51:54.790401 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a23bdae1-1e2a-4e26-8e2b-43d7b50a800e-workload-certs\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-jqc5p\" (UID: \"a23bdae1-1e2a-4e26-8e2b-43d7b50a800e\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-jqc5p" Oct 30 23:51:54.791803 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:51:54.791786 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a23bdae1-1e2a-4e26-8e2b-43d7b50a800e-istio-podinfo\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-jqc5p\" (UID: \"a23bdae1-1e2a-4e26-8e2b-43d7b50a800e\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-jqc5p" Oct 30 23:51:54.791938 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:51:54.791922 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a23bdae1-1e2a-4e26-8e2b-43d7b50a800e-istio-envoy\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-jqc5p\" (UID: \"a23bdae1-1e2a-4e26-8e2b-43d7b50a800e\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-jqc5p" Oct 30 23:51:54.852993 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:51:54.852958 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a23bdae1-1e2a-4e26-8e2b-43d7b50a800e-istio-token\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-jqc5p\" (UID: \"a23bdae1-1e2a-4e26-8e2b-43d7b50a800e\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-jqc5p" Oct 30 23:51:54.853138 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:51:54.852964 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-86d72\" (UniqueName: \"kubernetes.io/projected/a23bdae1-1e2a-4e26-8e2b-43d7b50a800e-kube-api-access-86d72\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-jqc5p\" (UID: \"a23bdae1-1e2a-4e26-8e2b-43d7b50a800e\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-jqc5p" Oct 30 23:51:54.931034 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:51:54.931002 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:51:54.942028 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:51:54.942001 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-jqc5p" Oct 30 23:51:55.085806 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:51:55.085738 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:51:55.239118 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:51:55.239027 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:51:55.278032 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:51:55.277995 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-94tkp"] Oct 30 23:51:55.282737 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:51:55.282714 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-94tkp" Oct 30 23:51:55.336849 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:51:55.336811 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-35a187-openshift-default-dockercfg-4wqw9\"" Oct 30 23:51:55.342529 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:51:55.342495 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-94tkp"] Oct 30 23:51:55.383686 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:51:55.383658 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-jqc5p"] Oct 30 23:51:55.385090 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:51:55.385063 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda23bdae1_1e2a_4e26_8e2b_43d7b50a800e.slice/crio-6926825dfbee21fd260113e9c0aa1c7f14d821b4d30af33e5ee86d8050ffc4f0 WatchSource:0}: Error finding container 6926825dfbee21fd260113e9c0aa1c7f14d821b4d30af33e5ee86d8050ffc4f0: Status 404 returned error can't find the container with id 6926825dfbee21fd260113e9c0aa1c7f14d821b4d30af33e5ee86d8050ffc4f0 Oct 30 23:51:55.387198 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:51:55.387170 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 23:51:55.387254 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:51:55.387229 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 23:51:55.387293 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:51:55.387269 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 23:51:55.390115 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:51:55.390085 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:51:55.401164 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:51:55.401143 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f348596-394c-4bfb-ba79-4788da8fc6de-istio-token\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-94tkp\" (UID: \"9f348596-394c-4bfb-ba79-4788da8fc6de\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-94tkp" Oct 30 23:51:55.401223 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:51:55.401191 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f348596-394c-4bfb-ba79-4788da8fc6de-istio-data\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-94tkp\" (UID: \"9f348596-394c-4bfb-ba79-4788da8fc6de\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-94tkp" Oct 30 23:51:55.401300 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:51:55.401288 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f348596-394c-4bfb-ba79-4788da8fc6de-istio-podinfo\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-94tkp\" (UID: \"9f348596-394c-4bfb-ba79-4788da8fc6de\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-94tkp" Oct 30 23:51:55.401356 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:51:55.401320 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f348596-394c-4bfb-ba79-4788da8fc6de-workload-socket\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-94tkp\" (UID: \"9f348596-394c-4bfb-ba79-4788da8fc6de\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-94tkp" Oct 30 23:51:55.401402 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:51:55.401385 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f348596-394c-4bfb-ba79-4788da8fc6de-credential-socket\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-94tkp\" (UID: \"9f348596-394c-4bfb-ba79-4788da8fc6de\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-94tkp" Oct 30 23:51:55.401504 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:51:55.401487 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f348596-394c-4bfb-ba79-4788da8fc6de-istio-envoy\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-94tkp\" (UID: \"9f348596-394c-4bfb-ba79-4788da8fc6de\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-94tkp" Oct 30 23:51:55.401557 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:51:55.401542 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ml2l5\" (UniqueName: \"kubernetes.io/projected/9f348596-394c-4bfb-ba79-4788da8fc6de-kube-api-access-ml2l5\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-94tkp\" (UID: \"9f348596-394c-4bfb-ba79-4788da8fc6de\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-94tkp" Oct 30 23:51:55.401599 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:51:55.401574 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f348596-394c-4bfb-ba79-4788da8fc6de-istiod-ca-cert\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-94tkp\" (UID: \"9f348596-394c-4bfb-ba79-4788da8fc6de\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-94tkp" Oct 30 23:51:55.401678 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:51:55.401665 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f348596-394c-4bfb-ba79-4788da8fc6de-workload-certs\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-94tkp\" (UID: \"9f348596-394c-4bfb-ba79-4788da8fc6de\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-94tkp" Oct 30 23:51:55.502247 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:51:55.502165 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f348596-394c-4bfb-ba79-4788da8fc6de-istiod-ca-cert\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-94tkp\" (UID: \"9f348596-394c-4bfb-ba79-4788da8fc6de\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-94tkp" Oct 30 23:51:55.502247 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:51:55.502216 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f348596-394c-4bfb-ba79-4788da8fc6de-workload-certs\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-94tkp\" (UID: \"9f348596-394c-4bfb-ba79-4788da8fc6de\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-94tkp" Oct 30 23:51:55.502511 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:51:55.502257 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f348596-394c-4bfb-ba79-4788da8fc6de-istio-token\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-94tkp\" (UID: \"9f348596-394c-4bfb-ba79-4788da8fc6de\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-94tkp" Oct 30 23:51:55.502511 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:51:55.502305 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f348596-394c-4bfb-ba79-4788da8fc6de-istio-data\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-94tkp\" (UID: \"9f348596-394c-4bfb-ba79-4788da8fc6de\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-94tkp" Oct 30 23:51:55.502511 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:51:55.502357 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f348596-394c-4bfb-ba79-4788da8fc6de-istio-podinfo\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-94tkp\" (UID: \"9f348596-394c-4bfb-ba79-4788da8fc6de\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-94tkp" Oct 30 23:51:55.502511 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:51:55.502391 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f348596-394c-4bfb-ba79-4788da8fc6de-workload-socket\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-94tkp\" (UID: \"9f348596-394c-4bfb-ba79-4788da8fc6de\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-94tkp" Oct 30 23:51:55.502511 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:51:55.502417 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f348596-394c-4bfb-ba79-4788da8fc6de-credential-socket\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-94tkp\" (UID: \"9f348596-394c-4bfb-ba79-4788da8fc6de\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-94tkp" Oct 30 23:51:55.502511 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:51:55.502448 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f348596-394c-4bfb-ba79-4788da8fc6de-istio-envoy\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-94tkp\" (UID: \"9f348596-394c-4bfb-ba79-4788da8fc6de\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-94tkp" Oct 30 23:51:55.502809 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:51:55.502517 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ml2l5\" (UniqueName: \"kubernetes.io/projected/9f348596-394c-4bfb-ba79-4788da8fc6de-kube-api-access-ml2l5\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-94tkp\" (UID: \"9f348596-394c-4bfb-ba79-4788da8fc6de\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-94tkp" Oct 30 23:51:55.502809 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:51:55.502699 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f348596-394c-4bfb-ba79-4788da8fc6de-workload-certs\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-94tkp\" (UID: \"9f348596-394c-4bfb-ba79-4788da8fc6de\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-94tkp" Oct 30 23:51:55.502980 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:51:55.502952 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f348596-394c-4bfb-ba79-4788da8fc6de-istio-data\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-94tkp\" (UID: \"9f348596-394c-4bfb-ba79-4788da8fc6de\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-94tkp" Oct 30 23:51:55.503206 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:51:55.503185 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f348596-394c-4bfb-ba79-4788da8fc6de-credential-socket\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-94tkp\" (UID: \"9f348596-394c-4bfb-ba79-4788da8fc6de\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-94tkp" Oct 30 23:51:55.503279 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:51:55.503261 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f348596-394c-4bfb-ba79-4788da8fc6de-workload-socket\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-94tkp\" (UID: \"9f348596-394c-4bfb-ba79-4788da8fc6de\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-94tkp" Oct 30 23:51:55.503394 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:51:55.503377 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f348596-394c-4bfb-ba79-4788da8fc6de-istiod-ca-cert\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-94tkp\" (UID: \"9f348596-394c-4bfb-ba79-4788da8fc6de\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-94tkp" Oct 30 23:51:55.505156 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:51:55.505129 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f348596-394c-4bfb-ba79-4788da8fc6de-istio-podinfo\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-94tkp\" (UID: \"9f348596-394c-4bfb-ba79-4788da8fc6de\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-94tkp" Oct 30 23:51:55.505309 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:51:55.505286 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f348596-394c-4bfb-ba79-4788da8fc6de-istio-envoy\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-94tkp\" (UID: \"9f348596-394c-4bfb-ba79-4788da8fc6de\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-94tkp" Oct 30 23:51:55.564391 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:51:55.564359 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f348596-394c-4bfb-ba79-4788da8fc6de-istio-token\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-94tkp\" (UID: \"9f348596-394c-4bfb-ba79-4788da8fc6de\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-94tkp" Oct 30 23:51:55.565709 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:51:55.565690 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ml2l5\" (UniqueName: \"kubernetes.io/projected/9f348596-394c-4bfb-ba79-4788da8fc6de-kube-api-access-ml2l5\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-94tkp\" (UID: \"9f348596-394c-4bfb-ba79-4788da8fc6de\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-94tkp" Oct 30 23:51:55.584566 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:51:55.584533 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:51:55.596422 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:51:55.596400 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-94tkp" Oct 30 23:51:55.737870 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:51:55.737804 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:51:55.760295 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:51:55.760191 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-jqc5p" event={"ID":"a23bdae1-1e2a-4e26-8e2b-43d7b50a800e","Type":"ContainerStarted","Data":"66fe27523fed3875d511f1effa6f7cdd362683d0e88c130474cb5f5fb93ff4fd"} Oct 30 23:51:55.760295 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:51:55.760255 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-jqc5p" event={"ID":"a23bdae1-1e2a-4e26-8e2b-43d7b50a800e","Type":"ContainerStarted","Data":"6926825dfbee21fd260113e9c0aa1c7f14d821b4d30af33e5ee86d8050ffc4f0"} Oct 30 23:51:55.911472 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:51:55.911421 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-jqc5p" podStartSLOduration=1.9114059829999999 podStartE2EDuration="1.911405983s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:51:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:51:55.909462456 +0000 UTC m=+628.945793290" watchObservedRunningTime="2026-10-30 23:51:55.911405983 +0000 UTC m=+628.947736822" Oct 30 23:51:55.942678 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:51:55.942650 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-jqc5p" Oct 30 23:51:55.966665 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:51:55.966629 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:51:56.033576 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:51:56.033545 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-94tkp"] Oct 30 23:51:56.034803 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:51:56.034771 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f348596_394c_4bfb_ba79_4788da8fc6de.slice/crio-f83de21595e5c3c41d8b1dc997798e97442e8bb3e5a0cb3c45cbfc86d21c97ac WatchSource:0}: Error finding container f83de21595e5c3c41d8b1dc997798e97442e8bb3e5a0cb3c45cbfc86d21c97ac: Status 404 returned error can't find the container with id f83de21595e5c3c41d8b1dc997798e97442e8bb3e5a0cb3c45cbfc86d21c97ac Oct 30 23:51:56.037648 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:51:56.037616 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 23:51:56.037756 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:51:56.037689 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 23:51:56.037756 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:51:56.037725 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 23:51:56.149363 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:51:56.148261 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:51:56.327389 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:51:56.327285 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-678tp"] Oct 30 23:51:56.331468 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:51:56.331430 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-678tp" Oct 30 23:51:56.341409 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:51:56.341361 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:51:56.391552 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:51:56.391517 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-678tp"] Oct 30 23:51:56.412481 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:51:56.412446 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8844df03-4076-4f22-94bd-49a4514c21ff-istio-token\") pod \"e2e-worker-w1-5c3374-openshift-default-d6ff699c7-678tp\" (UID: \"8844df03-4076-4f22-94bd-49a4514c21ff\") " pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-678tp" Oct 30 23:51:56.412655 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:51:56.412495 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8844df03-4076-4f22-94bd-49a4514c21ff-workload-certs\") pod \"e2e-worker-w1-5c3374-openshift-default-d6ff699c7-678tp\" (UID: \"8844df03-4076-4f22-94bd-49a4514c21ff\") " pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-678tp" Oct 30 23:51:56.412655 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:51:56.412551 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5lvp\" (UniqueName: \"kubernetes.io/projected/8844df03-4076-4f22-94bd-49a4514c21ff-kube-api-access-z5lvp\") pod \"e2e-worker-w1-5c3374-openshift-default-d6ff699c7-678tp\" (UID: \"8844df03-4076-4f22-94bd-49a4514c21ff\") " pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-678tp" Oct 30 23:51:56.412769 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:51:56.412658 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8844df03-4076-4f22-94bd-49a4514c21ff-workload-socket\") pod \"e2e-worker-w1-5c3374-openshift-default-d6ff699c7-678tp\" (UID: \"8844df03-4076-4f22-94bd-49a4514c21ff\") " pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-678tp" Oct 30 23:51:56.412821 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:51:56.412764 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8844df03-4076-4f22-94bd-49a4514c21ff-istio-data\") pod \"e2e-worker-w1-5c3374-openshift-default-d6ff699c7-678tp\" (UID: \"8844df03-4076-4f22-94bd-49a4514c21ff\") " pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-678tp" Oct 30 23:51:56.412917 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:51:56.412890 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8844df03-4076-4f22-94bd-49a4514c21ff-credential-socket\") pod \"e2e-worker-w1-5c3374-openshift-default-d6ff699c7-678tp\" (UID: \"8844df03-4076-4f22-94bd-49a4514c21ff\") " pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-678tp" Oct 30 23:51:56.413030 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:51:56.412997 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8844df03-4076-4f22-94bd-49a4514c21ff-istio-envoy\") pod \"e2e-worker-w1-5c3374-openshift-default-d6ff699c7-678tp\" (UID: \"8844df03-4076-4f22-94bd-49a4514c21ff\") " pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-678tp" Oct 30 23:51:56.413099 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:51:56.413045 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8844df03-4076-4f22-94bd-49a4514c21ff-istio-podinfo\") pod \"e2e-worker-w1-5c3374-openshift-default-d6ff699c7-678tp\" (UID: \"8844df03-4076-4f22-94bd-49a4514c21ff\") " pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-678tp" Oct 30 23:51:56.413139 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:51:56.413099 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8844df03-4076-4f22-94bd-49a4514c21ff-istiod-ca-cert\") pod \"e2e-worker-w1-5c3374-openshift-default-d6ff699c7-678tp\" (UID: \"8844df03-4076-4f22-94bd-49a4514c21ff\") " pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-678tp" Oct 30 23:51:56.438913 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:51:56.438888 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-5c3374-openshift-default-dockercfg-jbr44\"" Oct 30 23:51:56.502351 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:51:56.502297 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:51:56.514743 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:51:56.514696 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8844df03-4076-4f22-94bd-49a4514c21ff-istio-envoy\") pod \"e2e-worker-w1-5c3374-openshift-default-d6ff699c7-678tp\" (UID: \"8844df03-4076-4f22-94bd-49a4514c21ff\") " pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-678tp" Oct 30 23:51:56.514743 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:51:56.514745 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8844df03-4076-4f22-94bd-49a4514c21ff-istio-podinfo\") pod \"e2e-worker-w1-5c3374-openshift-default-d6ff699c7-678tp\" (UID: \"8844df03-4076-4f22-94bd-49a4514c21ff\") " pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-678tp" Oct 30 23:51:56.515002 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:51:56.514773 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8844df03-4076-4f22-94bd-49a4514c21ff-istiod-ca-cert\") pod \"e2e-worker-w1-5c3374-openshift-default-d6ff699c7-678tp\" (UID: \"8844df03-4076-4f22-94bd-49a4514c21ff\") " pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-678tp" Oct 30 23:51:56.515002 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:51:56.514825 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8844df03-4076-4f22-94bd-49a4514c21ff-istio-token\") pod \"e2e-worker-w1-5c3374-openshift-default-d6ff699c7-678tp\" (UID: \"8844df03-4076-4f22-94bd-49a4514c21ff\") " pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-678tp" Oct 30 23:51:56.515002 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:51:56.514853 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8844df03-4076-4f22-94bd-49a4514c21ff-workload-certs\") pod \"e2e-worker-w1-5c3374-openshift-default-d6ff699c7-678tp\" (UID: \"8844df03-4076-4f22-94bd-49a4514c21ff\") " pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-678tp" Oct 30 23:51:56.515002 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:51:56.514905 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5lvp\" (UniqueName: \"kubernetes.io/projected/8844df03-4076-4f22-94bd-49a4514c21ff-kube-api-access-z5lvp\") pod \"e2e-worker-w1-5c3374-openshift-default-d6ff699c7-678tp\" (UID: \"8844df03-4076-4f22-94bd-49a4514c21ff\") " pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-678tp" Oct 30 23:51:56.515002 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:51:56.514934 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8844df03-4076-4f22-94bd-49a4514c21ff-workload-socket\") pod \"e2e-worker-w1-5c3374-openshift-default-d6ff699c7-678tp\" (UID: \"8844df03-4076-4f22-94bd-49a4514c21ff\") " pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-678tp" Oct 30 23:51:56.515002 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:51:56.514997 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8844df03-4076-4f22-94bd-49a4514c21ff-istio-data\") pod \"e2e-worker-w1-5c3374-openshift-default-d6ff699c7-678tp\" (UID: \"8844df03-4076-4f22-94bd-49a4514c21ff\") " pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-678tp" Oct 30 23:51:56.515372 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:51:56.515055 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8844df03-4076-4f22-94bd-49a4514c21ff-credential-socket\") pod \"e2e-worker-w1-5c3374-openshift-default-d6ff699c7-678tp\" (UID: \"8844df03-4076-4f22-94bd-49a4514c21ff\") " pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-678tp" Oct 30 23:51:56.515567 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:51:56.515541 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8844df03-4076-4f22-94bd-49a4514c21ff-credential-socket\") pod \"e2e-worker-w1-5c3374-openshift-default-d6ff699c7-678tp\" (UID: \"8844df03-4076-4f22-94bd-49a4514c21ff\") " pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-678tp" Oct 30 23:51:56.517937 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:51:56.516322 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8844df03-4076-4f22-94bd-49a4514c21ff-workload-socket\") pod \"e2e-worker-w1-5c3374-openshift-default-d6ff699c7-678tp\" (UID: \"8844df03-4076-4f22-94bd-49a4514c21ff\") " pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-678tp" Oct 30 23:51:56.517937 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:51:56.516933 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8844df03-4076-4f22-94bd-49a4514c21ff-istio-data\") pod \"e2e-worker-w1-5c3374-openshift-default-d6ff699c7-678tp\" (UID: \"8844df03-4076-4f22-94bd-49a4514c21ff\") " pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-678tp" Oct 30 23:51:56.517937 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:51:56.517577 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8844df03-4076-4f22-94bd-49a4514c21ff-istiod-ca-cert\") pod \"e2e-worker-w1-5c3374-openshift-default-d6ff699c7-678tp\" (UID: \"8844df03-4076-4f22-94bd-49a4514c21ff\") " pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-678tp" Oct 30 23:51:56.518466 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:51:56.518443 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8844df03-4076-4f22-94bd-49a4514c21ff-workload-certs\") pod \"e2e-worker-w1-5c3374-openshift-default-d6ff699c7-678tp\" (UID: \"8844df03-4076-4f22-94bd-49a4514c21ff\") " pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-678tp" Oct 30 23:51:56.522091 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:51:56.522068 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8844df03-4076-4f22-94bd-49a4514c21ff-istio-envoy\") pod \"e2e-worker-w1-5c3374-openshift-default-d6ff699c7-678tp\" (UID: \"8844df03-4076-4f22-94bd-49a4514c21ff\") " pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-678tp" Oct 30 23:51:56.524058 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:51:56.523047 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8844df03-4076-4f22-94bd-49a4514c21ff-istio-podinfo\") pod \"e2e-worker-w1-5c3374-openshift-default-d6ff699c7-678tp\" (UID: \"8844df03-4076-4f22-94bd-49a4514c21ff\") " pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-678tp" Oct 30 23:51:56.578675 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:51:56.578609 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5lvp\" (UniqueName: \"kubernetes.io/projected/8844df03-4076-4f22-94bd-49a4514c21ff-kube-api-access-z5lvp\") pod \"e2e-worker-w1-5c3374-openshift-default-d6ff699c7-678tp\" (UID: \"8844df03-4076-4f22-94bd-49a4514c21ff\") " pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-678tp" Oct 30 23:51:56.579755 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:51:56.579738 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8844df03-4076-4f22-94bd-49a4514c21ff-istio-token\") pod \"e2e-worker-w1-5c3374-openshift-default-d6ff699c7-678tp\" (UID: \"8844df03-4076-4f22-94bd-49a4514c21ff\") " pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-678tp" Oct 30 23:51:56.650203 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:51:56.650167 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-678tp" Oct 30 23:51:56.664244 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:51:56.664213 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:51:56.768721 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:51:56.768684 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-94tkp" event={"ID":"9f348596-394c-4bfb-ba79-4788da8fc6de","Type":"ContainerStarted","Data":"65e5c2072ab717203bf04484e90c092ed6756580261f380bc47fb601fcba23d1"} Oct 30 23:51:56.769156 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:51:56.768743 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-94tkp" event={"ID":"9f348596-394c-4bfb-ba79-4788da8fc6de","Type":"ContainerStarted","Data":"f83de21595e5c3c41d8b1dc997798e97442e8bb3e5a0cb3c45cbfc86d21c97ac"} Oct 30 23:51:56.898420 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:51:56.898287 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-94tkp" podStartSLOduration=1.898273949 podStartE2EDuration="1.898273949s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:51:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:51:56.897762618 +0000 UTC m=+629.934093465" watchObservedRunningTime="2026-10-30 23:51:56.898273949 +0000 UTC m=+629.934604770" Oct 30 23:51:56.947673 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:51:56.947642 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-jqc5p" Oct 30 23:51:57.085785 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:51:57.085756 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8844df03_4076_4f22_94bd_49a4514c21ff.slice/crio-970f4bdd6ccf5cc6f2ec3c3b9f89f3eefb48a2f1211f35a2592e860ba0ad7ce1 WatchSource:0}: Error finding container 970f4bdd6ccf5cc6f2ec3c3b9f89f3eefb48a2f1211f35a2592e860ba0ad7ce1: Status 404 returned error can't find the container with id 970f4bdd6ccf5cc6f2ec3c3b9f89f3eefb48a2f1211f35a2592e860ba0ad7ce1 Oct 30 23:51:57.085785 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:51:57.085768 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-678tp"] Oct 30 23:51:57.087992 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:51:57.087961 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 23:51:57.088078 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:51:57.088020 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 23:51:57.088078 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:51:57.088048 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 23:51:57.596801 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:51:57.596765 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-94tkp" Oct 30 23:51:57.602387 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:51:57.602360 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-94tkp" Oct 30 23:51:57.777405 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:51:57.777367 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-678tp" event={"ID":"8844df03-4076-4f22-94bd-49a4514c21ff","Type":"ContainerStarted","Data":"2eb5077930c6b1e1cddac4a30c1a767126179b165ee2d731d11d0be2e41468fe"} Oct 30 23:51:57.777876 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:51:57.777438 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-678tp" event={"ID":"8844df03-4076-4f22-94bd-49a4514c21ff","Type":"ContainerStarted","Data":"970f4bdd6ccf5cc6f2ec3c3b9f89f3eefb48a2f1211f35a2592e860ba0ad7ce1"} Oct 30 23:51:57.777876 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:51:57.777756 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-jqc5p" Oct 30 23:51:57.777876 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:51:57.777802 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-94tkp" Oct 30 23:51:57.778915 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:51:57.778889 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-jqc5p" Oct 30 23:51:57.779025 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:51:57.778944 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-94tkp" Oct 30 23:51:57.909469 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:51:57.909426 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-678tp" podStartSLOduration=1.909413257 podStartE2EDuration="1.909413257s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:51:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:51:57.9066496 +0000 UTC m=+630.942980423" watchObservedRunningTime="2026-10-30 23:51:57.909413257 +0000 UTC m=+630.945744080" Oct 30 23:51:58.650633 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:51:58.650597 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-678tp" Oct 30 23:51:58.655906 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:51:58.655881 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-678tp" Oct 30 23:51:58.783417 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:51:58.783384 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-678tp" Oct 30 23:51:58.784273 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:51:58.784249 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-678tp" Oct 30 23:51:59.007724 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:51:59.007688 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:51:59.178131 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:51:59.178098 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:51:59.334832 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:51:59.334742 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:51:59.764034 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:51:59.763995 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:51:59.964838 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:51:59.964807 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:52:00.114089 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:00.113999 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:52:00.386706 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:00.386626 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:52:00.615667 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:00.615635 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:52:00.886126 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:00.886095 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:52:01.262436 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:01.262385 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:52:01.437314 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:01.437278 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:52:01.589113 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:01.589024 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:52:01.854583 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:01.854503 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:52:02.012836 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:02.012799 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:52:02.164837 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:02.164809 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:52:03.451046 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:03.451007 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:52:03.657300 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:03.657256 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:52:03.810051 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:03.809959 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:52:03.990089 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:03.990053 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:52:04.231574 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:04.231541 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:52:04.406507 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:04.406469 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-986d37-5d7d64658c-h2tgp"] Oct 30 23:52:04.411374 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:04.411352 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-986d37-5d7d64658c-h2tgp" Oct 30 23:52:04.466836 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:04.466810 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-986d37-5d7d64658c-h2tgp"] Oct 30 23:52:04.526450 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:04.520697 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-25hhv\" (UniqueName: \"kubernetes.io/projected/fc5ca633-21a6-4853-88d1-cccb1e17ad4e-kube-api-access-25hhv\") pod \"payload-pre-processing-e2e-worker-w3-986d37-5d7d64658c-h2tgp\" (UID: \"fc5ca633-21a6-4853-88d1-cccb1e17ad4e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-986d37-5d7d64658c-h2tgp" Oct 30 23:52:04.526450 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:04.520794 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc5ca633-21a6-4853-88d1-cccb1e17ad4e-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-986d37-5d7d64658c-h2tgp\" (UID: \"fc5ca633-21a6-4853-88d1-cccb1e17ad4e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-986d37-5d7d64658c-h2tgp" Oct 30 23:52:04.526450 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:04.521815 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-986d37\"" Oct 30 23:52:04.526450 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:04.521960 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-986d37-dockercfg-w4vht\"" Oct 30 23:52:04.576039 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:04.576004 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-986d37-695476d5cd-99x6j"] Oct 30 23:52:04.579860 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:04.579837 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-986d37-695476d5cd-99x6j" Oct 30 23:52:04.622873 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:04.622835 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc5ca633-21a6-4853-88d1-cccb1e17ad4e-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-986d37-5d7d64658c-h2tgp\" (UID: \"fc5ca633-21a6-4853-88d1-cccb1e17ad4e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-986d37-5d7d64658c-h2tgp" Oct 30 23:52:04.623045 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:04.622980 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25hhv\" (UniqueName: \"kubernetes.io/projected/fc5ca633-21a6-4853-88d1-cccb1e17ad4e-kube-api-access-25hhv\") pod \"payload-pre-processing-e2e-worker-w3-986d37-5d7d64658c-h2tgp\" (UID: \"fc5ca633-21a6-4853-88d1-cccb1e17ad4e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-986d37-5d7d64658c-h2tgp" Oct 30 23:52:04.623917 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:04.623894 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc5ca633-21a6-4853-88d1-cccb1e17ad4e-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-986d37-5d7d64658c-h2tgp\" (UID: \"fc5ca633-21a6-4853-88d1-cccb1e17ad4e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-986d37-5d7d64658c-h2tgp" Oct 30 23:52:04.629515 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:04.629486 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-986d37-695476d5cd-99x6j"] Oct 30 23:52:04.685145 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:04.685113 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-25hhv\" (UniqueName: \"kubernetes.io/projected/fc5ca633-21a6-4853-88d1-cccb1e17ad4e-kube-api-access-25hhv\") pod \"payload-pre-processing-e2e-worker-w3-986d37-5d7d64658c-h2tgp\" (UID: \"fc5ca633-21a6-4853-88d1-cccb1e17ad4e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-986d37-5d7d64658c-h2tgp" Oct 30 23:52:04.724430 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:04.724385 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wwn86\" (UniqueName: \"kubernetes.io/projected/915e2f48-c25a-4bcf-a1d1-edcc5245dfb8-kube-api-access-wwn86\") pod \"payload-processing-e2e-worker-w3-986d37-695476d5cd-99x6j\" (UID: \"915e2f48-c25a-4bcf-a1d1-edcc5245dfb8\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-986d37-695476d5cd-99x6j" Oct 30 23:52:04.724631 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:04.724600 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/915e2f48-c25a-4bcf-a1d1-edcc5245dfb8-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-986d37-695476d5cd-99x6j\" (UID: \"915e2f48-c25a-4bcf-a1d1-edcc5245dfb8\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-986d37-695476d5cd-99x6j" Oct 30 23:52:04.725097 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:04.725074 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-986d37-5d7d64658c-h2tgp" Oct 30 23:52:04.825533 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:04.825503 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/915e2f48-c25a-4bcf-a1d1-edcc5245dfb8-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-986d37-695476d5cd-99x6j\" (UID: \"915e2f48-c25a-4bcf-a1d1-edcc5245dfb8\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-986d37-695476d5cd-99x6j" Oct 30 23:52:04.825715 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:04.825549 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwn86\" (UniqueName: \"kubernetes.io/projected/915e2f48-c25a-4bcf-a1d1-edcc5245dfb8-kube-api-access-wwn86\") pod \"payload-processing-e2e-worker-w3-986d37-695476d5cd-99x6j\" (UID: \"915e2f48-c25a-4bcf-a1d1-edcc5245dfb8\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-986d37-695476d5cd-99x6j" Oct 30 23:52:04.826256 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:04.826235 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/915e2f48-c25a-4bcf-a1d1-edcc5245dfb8-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-986d37-695476d5cd-99x6j\" (UID: \"915e2f48-c25a-4bcf-a1d1-edcc5245dfb8\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-986d37-695476d5cd-99x6j" Oct 30 23:52:04.887808 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:04.887773 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwn86\" (UniqueName: \"kubernetes.io/projected/915e2f48-c25a-4bcf-a1d1-edcc5245dfb8-kube-api-access-wwn86\") pod \"payload-processing-e2e-worker-w3-986d37-695476d5cd-99x6j\" (UID: \"915e2f48-c25a-4bcf-a1d1-edcc5245dfb8\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-986d37-695476d5cd-99x6j" Oct 30 23:52:04.893056 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:04.893033 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-986d37-695476d5cd-99x6j" Oct 30 23:52:05.107737 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:05.107599 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-986d37-5d7d64658c-h2tgp"] Oct 30 23:52:05.110882 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:52:05.110836 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc5ca633_21a6_4853_88d1_cccb1e17ad4e.slice/crio-7f5c2ed32a1b765c09ac6a4025bc7e11dd96567981981f062e77e08ea9abc50f WatchSource:0}: Error finding container 7f5c2ed32a1b765c09ac6a4025bc7e11dd96567981981f062e77e08ea9abc50f: Status 404 returned error can't find the container with id 7f5c2ed32a1b765c09ac6a4025bc7e11dd96567981981f062e77e08ea9abc50f Oct 30 23:52:05.231128 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:05.231088 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:52:05.270992 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:05.270968 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-986d37-695476d5cd-99x6j"] Oct 30 23:52:05.272537 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:52:05.272505 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod915e2f48_c25a_4bcf_a1d1_edcc5245dfb8.slice/crio-a7b8e2039573501c1b63f2e913796ce22bf4f71c26569b64363ad3735577fb3d WatchSource:0}: Error finding container a7b8e2039573501c1b63f2e913796ce22bf4f71c26569b64363ad3735577fb3d: Status 404 returned error can't find the container with id a7b8e2039573501c1b63f2e913796ce22bf4f71c26569b64363ad3735577fb3d Oct 30 23:52:05.446311 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:05.443979 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:52:05.826168 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:05.826131 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-986d37-695476d5cd-99x6j" event={"ID":"915e2f48-c25a-4bcf-a1d1-edcc5245dfb8","Type":"ContainerStarted","Data":"24f3926da24a3a209c72cb2885d1b74325abd3425173fa87e16cb25ee7a39d26"} Oct 30 23:52:05.826708 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:05.826201 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-986d37-695476d5cd-99x6j" Oct 30 23:52:05.826708 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:05.826228 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-986d37-695476d5cd-99x6j" event={"ID":"915e2f48-c25a-4bcf-a1d1-edcc5245dfb8","Type":"ContainerStarted","Data":"a7b8e2039573501c1b63f2e913796ce22bf4f71c26569b64363ad3735577fb3d"} Oct 30 23:52:05.827936 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:05.827909 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-986d37-5d7d64658c-h2tgp" event={"ID":"fc5ca633-21a6-4853-88d1-cccb1e17ad4e","Type":"ContainerStarted","Data":"d17f0fe66e125dca99815f10b9d62d3654f065875af1bb42f2d57598170926cd"} Oct 30 23:52:05.828077 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:05.827949 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-986d37-5d7d64658c-h2tgp" event={"ID":"fc5ca633-21a6-4853-88d1-cccb1e17ad4e","Type":"ContainerStarted","Data":"7f5c2ed32a1b765c09ac6a4025bc7e11dd96567981981f062e77e08ea9abc50f"} Oct 30 23:52:05.828077 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:05.827998 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-986d37-5d7d64658c-h2tgp" Oct 30 23:52:05.948351 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:05.948275 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w3-986d37-695476d5cd-99x6j" podStartSLOduration=1.948259979 podStartE2EDuration="1.948259979s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:52:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:52:05.947104244 +0000 UTC m=+638.983435066" watchObservedRunningTime="2026-10-30 23:52:05.948259979 +0000 UTC m=+638.984590845" Oct 30 23:52:06.076357 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:06.076220 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-986d37-5d7d64658c-h2tgp" podStartSLOduration=2.076196008 podStartE2EDuration="2.076196008s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:52:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:52:06.065409628 +0000 UTC m=+639.101740451" watchObservedRunningTime="2026-10-30 23:52:06.076196008 +0000 UTC m=+639.112526830" Oct 30 23:52:06.174894 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:06.174862 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:52:06.332197 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:06.332111 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:52:06.503837 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:06.503802 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:52:08.570956 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:08.570922 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:52:08.765420 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:08.765380 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:52:08.979630 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:08.979595 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-27e82b-5968f9f875-t9qz2"] Oct 30 23:52:08.983788 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:08.983764 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-27e82b-5968f9f875-t9qz2" Oct 30 23:52:09.076378 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:09.058031 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-27e82b-5968f9f875-t9qz2"] Oct 30 23:52:09.084397 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:09.084362 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/687684ad-5b0d-493a-9fa4-cbbe640229fe-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-27e82b-5968f9f875-t9qz2\" (UID: \"687684ad-5b0d-493a-9fa4-cbbe640229fe\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-27e82b-5968f9f875-t9qz2" Oct 30 23:52:09.084593 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:09.084423 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65chd\" (UniqueName: \"kubernetes.io/projected/687684ad-5b0d-493a-9fa4-cbbe640229fe-kube-api-access-65chd\") pod \"payload-pre-processing-e2e-worker-w2-27e82b-5968f9f875-t9qz2\" (UID: \"687684ad-5b0d-493a-9fa4-cbbe640229fe\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-27e82b-5968f9f875-t9qz2" Oct 30 23:52:09.095702 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:09.095633 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w2-27e82b-dockercfg-2tjt7\"" Oct 30 23:52:09.095702 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:09.095633 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-27e82b\"" Oct 30 23:52:09.130485 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:09.130448 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-27e82b-64f94b9586-wb68f"] Oct 30 23:52:09.134465 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:09.134443 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-27e82b-64f94b9586-wb68f" Oct 30 23:52:09.185264 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:09.185227 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ckl8m\" (UniqueName: \"kubernetes.io/projected/59ccea66-1d77-43cd-bd56-e88b44a5e621-kube-api-access-ckl8m\") pod \"payload-processing-e2e-worker-w2-27e82b-64f94b9586-wb68f\" (UID: \"59ccea66-1d77-43cd-bd56-e88b44a5e621\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-27e82b-64f94b9586-wb68f" Oct 30 23:52:09.185431 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:09.185279 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/687684ad-5b0d-493a-9fa4-cbbe640229fe-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-27e82b-5968f9f875-t9qz2\" (UID: \"687684ad-5b0d-493a-9fa4-cbbe640229fe\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-27e82b-5968f9f875-t9qz2" Oct 30 23:52:09.185431 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:09.185359 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65chd\" (UniqueName: \"kubernetes.io/projected/687684ad-5b0d-493a-9fa4-cbbe640229fe-kube-api-access-65chd\") pod \"payload-pre-processing-e2e-worker-w2-27e82b-5968f9f875-t9qz2\" (UID: \"687684ad-5b0d-493a-9fa4-cbbe640229fe\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-27e82b-5968f9f875-t9qz2" Oct 30 23:52:09.185759 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:09.185563 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59ccea66-1d77-43cd-bd56-e88b44a5e621-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-27e82b-64f94b9586-wb68f\" (UID: \"59ccea66-1d77-43cd-bd56-e88b44a5e621\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-27e82b-64f94b9586-wb68f" Oct 30 23:52:09.186303 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:09.186276 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/687684ad-5b0d-493a-9fa4-cbbe640229fe-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-27e82b-5968f9f875-t9qz2\" (UID: \"687684ad-5b0d-493a-9fa4-cbbe640229fe\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-27e82b-5968f9f875-t9qz2" Oct 30 23:52:09.190819 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:09.190793 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-27e82b-64f94b9586-wb68f"] Oct 30 23:52:09.248695 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:09.248628 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-65chd\" (UniqueName: \"kubernetes.io/projected/687684ad-5b0d-493a-9fa4-cbbe640229fe-kube-api-access-65chd\") pod \"payload-pre-processing-e2e-worker-w2-27e82b-5968f9f875-t9qz2\" (UID: \"687684ad-5b0d-493a-9fa4-cbbe640229fe\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-27e82b-5968f9f875-t9qz2" Oct 30 23:52:09.286497 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:09.286462 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ckl8m\" (UniqueName: \"kubernetes.io/projected/59ccea66-1d77-43cd-bd56-e88b44a5e621-kube-api-access-ckl8m\") pod \"payload-processing-e2e-worker-w2-27e82b-64f94b9586-wb68f\" (UID: \"59ccea66-1d77-43cd-bd56-e88b44a5e621\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-27e82b-64f94b9586-wb68f" Oct 30 23:52:09.286645 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:09.286531 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59ccea66-1d77-43cd-bd56-e88b44a5e621-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-27e82b-64f94b9586-wb68f\" (UID: \"59ccea66-1d77-43cd-bd56-e88b44a5e621\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-27e82b-64f94b9586-wb68f" Oct 30 23:52:09.287222 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:09.287204 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59ccea66-1d77-43cd-bd56-e88b44a5e621-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-27e82b-64f94b9586-wb68f\" (UID: \"59ccea66-1d77-43cd-bd56-e88b44a5e621\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-27e82b-64f94b9586-wb68f" Oct 30 23:52:09.298349 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:09.298312 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-27e82b-5968f9f875-t9qz2" Oct 30 23:52:09.349942 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:09.349251 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ckl8m\" (UniqueName: \"kubernetes.io/projected/59ccea66-1d77-43cd-bd56-e88b44a5e621-kube-api-access-ckl8m\") pod \"payload-processing-e2e-worker-w2-27e82b-64f94b9586-wb68f\" (UID: \"59ccea66-1d77-43cd-bd56-e88b44a5e621\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-27e82b-64f94b9586-wb68f" Oct 30 23:52:09.448537 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:09.448503 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-27e82b-64f94b9586-wb68f" Oct 30 23:52:09.676942 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:09.676909 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-27e82b-5968f9f875-t9qz2"] Oct 30 23:52:09.678692 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:52:09.678664 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod687684ad_5b0d_493a_9fa4_cbbe640229fe.slice/crio-ac57ee797300ddb624140b3badd499d024599e7cf270505d3058c55385829085 WatchSource:0}: Error finding container ac57ee797300ddb624140b3badd499d024599e7cf270505d3058c55385829085: Status 404 returned error can't find the container with id ac57ee797300ddb624140b3badd499d024599e7cf270505d3058c55385829085 Oct 30 23:52:09.826830 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:09.826797 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-27e82b-64f94b9586-wb68f"] Oct 30 23:52:09.827809 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:52:09.827778 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59ccea66_1d77_43cd_bd56_e88b44a5e621.slice/crio-bb81b7b3ed577dc8e2d039c0020ea560729abb232c159242f5a0bd4135987103 WatchSource:0}: Error finding container bb81b7b3ed577dc8e2d039c0020ea560729abb232c159242f5a0bd4135987103: Status 404 returned error can't find the container with id bb81b7b3ed577dc8e2d039c0020ea560729abb232c159242f5a0bd4135987103 Oct 30 23:52:09.869309 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:09.869259 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-27e82b-5968f9f875-t9qz2" event={"ID":"687684ad-5b0d-493a-9fa4-cbbe640229fe","Type":"ContainerStarted","Data":"97af6b4131cff76a3d0bf2dd637d64c4132121feb08d0a1acf9a6e4ea6429ee6"} Oct 30 23:52:09.869424 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:09.869355 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-27e82b-5968f9f875-t9qz2" event={"ID":"687684ad-5b0d-493a-9fa4-cbbe640229fe","Type":"ContainerStarted","Data":"ac57ee797300ddb624140b3badd499d024599e7cf270505d3058c55385829085"} Oct 30 23:52:09.869493 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:09.869482 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-27e82b-5968f9f875-t9qz2" Oct 30 23:52:09.870779 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:09.870748 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-27e82b-64f94b9586-wb68f" event={"ID":"59ccea66-1d77-43cd-bd56-e88b44a5e621","Type":"ContainerStarted","Data":"bb81b7b3ed577dc8e2d039c0020ea560729abb232c159242f5a0bd4135987103"} Oct 30 23:52:09.988512 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:09.988455 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-27e82b-5968f9f875-t9qz2" podStartSLOduration=1.988443528 podStartE2EDuration="1.988443528s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:52:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:52:09.987752675 +0000 UTC m=+643.024083525" watchObservedRunningTime="2026-10-30 23:52:09.988443528 +0000 UTC m=+643.024774350" Oct 30 23:52:10.240663 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:10.240621 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:52:10.428149 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:10.428113 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:52:10.578978 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:10.578883 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:52:10.879892 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:10.879798 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-27e82b-64f94b9586-wb68f" event={"ID":"59ccea66-1d77-43cd-bd56-e88b44a5e621","Type":"ContainerStarted","Data":"a8eef16879d922846f405a93c664865feabb85ecddcfb489be76f82b84fef7c9"} Oct 30 23:52:10.880277 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:10.879946 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-27e82b-64f94b9586-wb68f" Oct 30 23:52:11.000891 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:11.000836 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w2-27e82b-64f94b9586-wb68f" podStartSLOduration=2.000821691 podStartE2EDuration="2.000821691s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:52:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:52:10.999156171 +0000 UTC m=+644.035486994" watchObservedRunningTime="2026-10-30 23:52:11.000821691 +0000 UTC m=+644.037152584" Oct 30 23:52:11.005667 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:11.005626 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:52:11.154572 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:11.154482 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:52:11.394449 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:11.394415 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-35a187-7887c98dff-5h96p"] Oct 30 23:52:11.398495 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:11.398476 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-35a187-7887c98dff-5h96p" Oct 30 23:52:11.411587 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:11.411561 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b69ba41-3647-4827-9acc-8f00b05d299a-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-35a187-7887c98dff-5h96p\" (UID: \"2b69ba41-3647-4827-9acc-8f00b05d299a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-35a187-7887c98dff-5h96p" Oct 30 23:52:11.411690 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:11.411591 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7v8k\" (UniqueName: \"kubernetes.io/projected/2b69ba41-3647-4827-9acc-8f00b05d299a-kube-api-access-d7v8k\") pod \"payload-pre-processing-e2e-worker-w6-35a187-7887c98dff-5h96p\" (UID: \"2b69ba41-3647-4827-9acc-8f00b05d299a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-35a187-7887c98dff-5h96p" Oct 30 23:52:11.498191 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:11.498059 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-35a187-7887c98dff-5h96p"] Oct 30 23:52:11.498404 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:11.498347 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w6-35a187\"" Oct 30 23:52:11.513981 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:11.513931 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b69ba41-3647-4827-9acc-8f00b05d299a-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-35a187-7887c98dff-5h96p\" (UID: \"2b69ba41-3647-4827-9acc-8f00b05d299a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-35a187-7887c98dff-5h96p" Oct 30 23:52:11.513981 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:11.513980 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7v8k\" (UniqueName: \"kubernetes.io/projected/2b69ba41-3647-4827-9acc-8f00b05d299a-kube-api-access-d7v8k\") pod \"payload-pre-processing-e2e-worker-w6-35a187-7887c98dff-5h96p\" (UID: \"2b69ba41-3647-4827-9acc-8f00b05d299a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-35a187-7887c98dff-5h96p" Oct 30 23:52:11.514232 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:11.514175 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w6-35a187-dockercfg-jgq68\"" Oct 30 23:52:11.515238 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:11.515212 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b69ba41-3647-4827-9acc-8f00b05d299a-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-35a187-7887c98dff-5h96p\" (UID: \"2b69ba41-3647-4827-9acc-8f00b05d299a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-35a187-7887c98dff-5h96p" Oct 30 23:52:11.577349 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:11.577300 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7v8k\" (UniqueName: \"kubernetes.io/projected/2b69ba41-3647-4827-9acc-8f00b05d299a-kube-api-access-d7v8k\") pod \"payload-pre-processing-e2e-worker-w6-35a187-7887c98dff-5h96p\" (UID: \"2b69ba41-3647-4827-9acc-8f00b05d299a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-35a187-7887c98dff-5h96p" Oct 30 23:52:11.713447 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:11.713413 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-35a187-7887c98dff-5h96p" Oct 30 23:52:12.106102 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:12.106069 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-35a187-7887c98dff-5h96p"] Oct 30 23:52:12.107472 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:52:12.107445 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b69ba41_3647_4827_9acc_8f00b05d299a.slice/crio-e9f0c3286c8a3ec29795f18bf914dffece828606fbea756ec3a21fa1410351c2 WatchSource:0}: Error finding container e9f0c3286c8a3ec29795f18bf914dffece828606fbea756ec3a21fa1410351c2: Status 404 returned error can't find the container with id e9f0c3286c8a3ec29795f18bf914dffece828606fbea756ec3a21fa1410351c2 Oct 30 23:52:12.893033 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:12.892992 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-35a187-7887c98dff-5h96p" event={"ID":"2b69ba41-3647-4827-9acc-8f00b05d299a","Type":"ContainerStarted","Data":"007f441c941b9c34c2417a05dde72d89ab751077d63f5cef4d19c802ede34a78"} Oct 30 23:52:12.893222 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:12.893052 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-35a187-7887c98dff-5h96p" event={"ID":"2b69ba41-3647-4827-9acc-8f00b05d299a","Type":"ContainerStarted","Data":"e9f0c3286c8a3ec29795f18bf914dffece828606fbea756ec3a21fa1410351c2"} Oct 30 23:52:12.893222 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:12.893084 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-35a187-7887c98dff-5h96p" Oct 30 23:52:13.013863 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:13.013811 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-35a187-7887c98dff-5h96p" podStartSLOduration=2.013795937 podStartE2EDuration="2.013795937s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:52:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:52:13.011375436 +0000 UTC m=+646.047706256" watchObservedRunningTime="2026-10-30 23:52:13.013795937 +0000 UTC m=+646.050126819" Oct 30 23:52:15.343343 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:15.343303 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:52:15.541566 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:15.541533 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:52:15.687324 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:15.687287 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:52:16.058686 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:16.058605 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-fe01fa-698f78d94b-4wmlb"] Oct 30 23:52:16.062633 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:16.062616 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-fe01fa-698f78d94b-4wmlb" Oct 30 23:52:16.168238 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:16.168209 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w5-fe01fa\"" Oct 30 23:52:16.168442 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:16.168365 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w5-fe01fa-dockercfg-78qnz\"" Oct 30 23:52:16.171070 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:16.171039 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-fe01fa-698f78d94b-4wmlb"] Oct 30 23:52:16.184646 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:16.184612 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-clv8m\" (UniqueName: \"kubernetes.io/projected/518493d9-43fd-451c-abc6-fa2db3010c48-kube-api-access-clv8m\") pod \"payload-processing-e2e-worker-w5-fe01fa-698f78d94b-4wmlb\" (UID: \"518493d9-43fd-451c-abc6-fa2db3010c48\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-fe01fa-698f78d94b-4wmlb" Oct 30 23:52:16.184777 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:16.184662 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/518493d9-43fd-451c-abc6-fa2db3010c48-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-fe01fa-698f78d94b-4wmlb\" (UID: \"518493d9-43fd-451c-abc6-fa2db3010c48\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-fe01fa-698f78d94b-4wmlb" Oct 30 23:52:16.285810 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:16.285777 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-clv8m\" (UniqueName: \"kubernetes.io/projected/518493d9-43fd-451c-abc6-fa2db3010c48-kube-api-access-clv8m\") pod \"payload-processing-e2e-worker-w5-fe01fa-698f78d94b-4wmlb\" (UID: \"518493d9-43fd-451c-abc6-fa2db3010c48\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-fe01fa-698f78d94b-4wmlb" Oct 30 23:52:16.285989 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:16.285832 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/518493d9-43fd-451c-abc6-fa2db3010c48-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-fe01fa-698f78d94b-4wmlb\" (UID: \"518493d9-43fd-451c-abc6-fa2db3010c48\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-fe01fa-698f78d94b-4wmlb" Oct 30 23:52:16.286650 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:16.286628 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/518493d9-43fd-451c-abc6-fa2db3010c48-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-fe01fa-698f78d94b-4wmlb\" (UID: \"518493d9-43fd-451c-abc6-fa2db3010c48\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-fe01fa-698f78d94b-4wmlb" Oct 30 23:52:16.348078 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:16.348010 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-clv8m\" (UniqueName: \"kubernetes.io/projected/518493d9-43fd-451c-abc6-fa2db3010c48-kube-api-access-clv8m\") pod \"payload-processing-e2e-worker-w5-fe01fa-698f78d94b-4wmlb\" (UID: \"518493d9-43fd-451c-abc6-fa2db3010c48\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-fe01fa-698f78d94b-4wmlb" Oct 30 23:52:16.375836 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:16.375803 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-fe01fa-698f78d94b-4wmlb" Oct 30 23:52:16.753522 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:16.753485 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-fe01fa-698f78d94b-4wmlb"] Oct 30 23:52:16.754816 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:52:16.754786 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod518493d9_43fd_451c_abc6_fa2db3010c48.slice/crio-6ce510ecf01fbfa10ec89b3e89c8c21eb9098d2a787abed48dfbab8351cdd661 WatchSource:0}: Error finding container 6ce510ecf01fbfa10ec89b3e89c8c21eb9098d2a787abed48dfbab8351cdd661: Status 404 returned error can't find the container with id 6ce510ecf01fbfa10ec89b3e89c8c21eb9098d2a787abed48dfbab8351cdd661 Oct 30 23:52:16.836257 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:16.836232 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-986d37-5d7d64658c-h2tgp" Oct 30 23:52:16.836568 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:16.836552 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-986d37-695476d5cd-99x6j" Oct 30 23:52:16.922667 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:16.922627 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-fe01fa-698f78d94b-4wmlb" event={"ID":"518493d9-43fd-451c-abc6-fa2db3010c48","Type":"ContainerStarted","Data":"9f32da5fe52b33933496ee738a53fb3b95afe1e248b6cfe85bd542ea9684a90b"} Oct 30 23:52:16.922844 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:16.922691 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-fe01fa-698f78d94b-4wmlb" event={"ID":"518493d9-43fd-451c-abc6-fa2db3010c48","Type":"ContainerStarted","Data":"6ce510ecf01fbfa10ec89b3e89c8c21eb9098d2a787abed48dfbab8351cdd661"} Oct 30 23:52:16.922844 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:16.922716 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w5-fe01fa-698f78d94b-4wmlb" Oct 30 23:52:17.083880 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:17.083776 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w5-fe01fa-698f78d94b-4wmlb" podStartSLOduration=1.083751222 podStartE2EDuration="1.083751222s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:52:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:52:17.081924804 +0000 UTC m=+650.118255627" watchObservedRunningTime="2026-10-30 23:52:17.083751222 +0000 UTC m=+650.120082064" Oct 30 23:52:17.933169 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:17.933130 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:52:18.142322 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:18.142287 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:52:19.679061 ip-10-0-141-180 kubenswrapper[4927]: E1030 23:52:19.679016 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sqv69" containerName="extractor" Oct 30 23:52:20.239652 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:20.239611 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:52:20.393482 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:20.393447 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:52:20.550039 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:20.549948 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:52:20.698531 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:20.698482 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:52:20.883466 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:20.883391 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-27e82b-5968f9f875-t9qz2" Oct 30 23:52:21.888859 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:21.888830 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-27e82b-64f94b9586-wb68f" Oct 30 23:52:22.479982 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:22.479948 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:52:22.694605 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:22.694566 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:52:22.899536 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:22.896961 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-46b586-669f464d79-fsqx4"] Oct 30 23:52:22.902029 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:22.901999 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-46b586-669f464d79-fsqx4" Oct 30 23:52:22.966522 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:22.966487 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-46b586-669f464d79-fsqx4"] Oct 30 23:52:22.968286 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:22.968249 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7d25\" (UniqueName: \"kubernetes.io/projected/fea168a8-e4b2-46e2-8180-6910723a1982-kube-api-access-b7d25\") pod \"payload-pre-processing-e2e-worker-w0-46b586-669f464d79-fsqx4\" (UID: \"fea168a8-e4b2-46e2-8180-6910723a1982\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-46b586-669f464d79-fsqx4" Oct 30 23:52:22.968555 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:22.968533 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fea168a8-e4b2-46e2-8180-6910723a1982-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-46b586-669f464d79-fsqx4\" (UID: \"fea168a8-e4b2-46e2-8180-6910723a1982\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-46b586-669f464d79-fsqx4" Oct 30 23:52:23.023266 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:23.023231 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-46b586-dockercfg-jp427\"" Oct 30 23:52:23.024022 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:23.023637 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-46b586\"" Oct 30 23:52:23.069217 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:23.069179 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7d25\" (UniqueName: \"kubernetes.io/projected/fea168a8-e4b2-46e2-8180-6910723a1982-kube-api-access-b7d25\") pod \"payload-pre-processing-e2e-worker-w0-46b586-669f464d79-fsqx4\" (UID: \"fea168a8-e4b2-46e2-8180-6910723a1982\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-46b586-669f464d79-fsqx4" Oct 30 23:52:23.069442 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:23.069234 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fea168a8-e4b2-46e2-8180-6910723a1982-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-46b586-669f464d79-fsqx4\" (UID: \"fea168a8-e4b2-46e2-8180-6910723a1982\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-46b586-669f464d79-fsqx4" Oct 30 23:52:23.070012 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:23.069988 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fea168a8-e4b2-46e2-8180-6910723a1982-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-46b586-669f464d79-fsqx4\" (UID: \"fea168a8-e4b2-46e2-8180-6910723a1982\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-46b586-669f464d79-fsqx4" Oct 30 23:52:23.167353 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:23.167060 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7d25\" (UniqueName: \"kubernetes.io/projected/fea168a8-e4b2-46e2-8180-6910723a1982-kube-api-access-b7d25\") pod \"payload-pre-processing-e2e-worker-w0-46b586-669f464d79-fsqx4\" (UID: \"fea168a8-e4b2-46e2-8180-6910723a1982\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-46b586-669f464d79-fsqx4" Oct 30 23:52:23.217364 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:23.216523 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-46b586-669f464d79-fsqx4" Oct 30 23:52:23.643074 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:23.643042 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-46b586-669f464d79-fsqx4"] Oct 30 23:52:23.903496 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:23.903416 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-35a187-7887c98dff-5h96p" Oct 30 23:52:23.985239 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:23.985208 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-46b586-669f464d79-fsqx4" event={"ID":"fea168a8-e4b2-46e2-8180-6910723a1982","Type":"ContainerStarted","Data":"288590708f982a09b365e63976837e3c6f890f81f277361dc85950fb338e2173"} Oct 30 23:52:23.985426 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:23.985265 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-46b586-669f464d79-fsqx4" event={"ID":"fea168a8-e4b2-46e2-8180-6910723a1982","Type":"ContainerStarted","Data":"73dcdd56c3920557bcd6d8d74ad71522b086d38a38c10d64d83abcb6ef53a479"} Oct 30 23:52:23.985426 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:23.985290 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-46b586-669f464d79-fsqx4" Oct 30 23:52:24.147603 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:24.147550 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-46b586-669f464d79-fsqx4" podStartSLOduration=2.147534492 podStartE2EDuration="2.147534492s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:52:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:52:24.146501671 +0000 UTC m=+657.182832494" watchObservedRunningTime="2026-10-30 23:52:24.147534492 +0000 UTC m=+657.183865369" Oct 30 23:52:24.893262 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:24.893224 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:52:25.160111 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:25.160078 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:52:25.325899 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:25.325866 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:52:25.338090 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:25.336145 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-5c3374-f56fc6f79-7b9pp"] Oct 30 23:52:25.341189 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:25.341163 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5c3374-f56fc6f79-7b9pp" Oct 30 23:52:25.399705 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:25.398447 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5c1c1eb-bce5-4ef6-8246-c0638f5c8b6b-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-5c3374-f56fc6f79-7b9pp\" (UID: \"f5c1c1eb-bce5-4ef6-8246-c0638f5c8b6b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5c3374-f56fc6f79-7b9pp" Oct 30 23:52:25.399705 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:25.398503 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xz9mx\" (UniqueName: \"kubernetes.io/projected/f5c1c1eb-bce5-4ef6-8246-c0638f5c8b6b-kube-api-access-xz9mx\") pod \"payload-pre-processing-e2e-worker-w1-5c3374-f56fc6f79-7b9pp\" (UID: \"f5c1c1eb-bce5-4ef6-8246-c0638f5c8b6b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5c3374-f56fc6f79-7b9pp" Oct 30 23:52:25.402855 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:25.402826 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-5c3374-f56fc6f79-7b9pp"] Oct 30 23:52:25.457770 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:25.457688 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w1-5c3374-dockercfg-lndkx\"" Oct 30 23:52:25.458617 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:25.458593 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-5c3374\"" Oct 30 23:52:25.465816 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:25.465790 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-5c3374-7969595c9d-gzhtd"] Oct 30 23:52:25.482805 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:25.480629 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-5c3374-7969595c9d-gzhtd" Oct 30 23:52:25.499560 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:25.499533 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5c1c1eb-bce5-4ef6-8246-c0638f5c8b6b-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-5c3374-f56fc6f79-7b9pp\" (UID: \"f5c1c1eb-bce5-4ef6-8246-c0638f5c8b6b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5c3374-f56fc6f79-7b9pp" Oct 30 23:52:25.499780 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:25.499566 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xz9mx\" (UniqueName: \"kubernetes.io/projected/f5c1c1eb-bce5-4ef6-8246-c0638f5c8b6b-kube-api-access-xz9mx\") pod \"payload-pre-processing-e2e-worker-w1-5c3374-f56fc6f79-7b9pp\" (UID: \"f5c1c1eb-bce5-4ef6-8246-c0638f5c8b6b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5c3374-f56fc6f79-7b9pp" Oct 30 23:52:25.500306 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:25.500282 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5c1c1eb-bce5-4ef6-8246-c0638f5c8b6b-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-5c3374-f56fc6f79-7b9pp\" (UID: \"f5c1c1eb-bce5-4ef6-8246-c0638f5c8b6b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5c3374-f56fc6f79-7b9pp" Oct 30 23:52:25.560304 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:25.550774 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-5c3374-7969595c9d-gzhtd"] Oct 30 23:52:25.563076 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:25.563044 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xz9mx\" (UniqueName: \"kubernetes.io/projected/f5c1c1eb-bce5-4ef6-8246-c0638f5c8b6b-kube-api-access-xz9mx\") pod \"payload-pre-processing-e2e-worker-w1-5c3374-f56fc6f79-7b9pp\" (UID: \"f5c1c1eb-bce5-4ef6-8246-c0638f5c8b6b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5c3374-f56fc6f79-7b9pp" Oct 30 23:52:25.600409 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:25.600380 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8qn57\" (UniqueName: \"kubernetes.io/projected/2a65bb94-e7dd-4240-af0c-4ce7c24bc599-kube-api-access-8qn57\") pod \"payload-processing-e2e-worker-w1-5c3374-7969595c9d-gzhtd\" (UID: \"2a65bb94-e7dd-4240-af0c-4ce7c24bc599\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-5c3374-7969595c9d-gzhtd" Oct 30 23:52:25.600542 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:25.600461 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a65bb94-e7dd-4240-af0c-4ce7c24bc599-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-5c3374-7969595c9d-gzhtd\" (UID: \"2a65bb94-e7dd-4240-af0c-4ce7c24bc599\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-5c3374-7969595c9d-gzhtd" Oct 30 23:52:25.654720 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:25.654686 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5c3374-f56fc6f79-7b9pp" Oct 30 23:52:25.701762 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:25.701716 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qn57\" (UniqueName: \"kubernetes.io/projected/2a65bb94-e7dd-4240-af0c-4ce7c24bc599-kube-api-access-8qn57\") pod \"payload-processing-e2e-worker-w1-5c3374-7969595c9d-gzhtd\" (UID: \"2a65bb94-e7dd-4240-af0c-4ce7c24bc599\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-5c3374-7969595c9d-gzhtd" Oct 30 23:52:25.701918 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:25.701852 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a65bb94-e7dd-4240-af0c-4ce7c24bc599-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-5c3374-7969595c9d-gzhtd\" (UID: \"2a65bb94-e7dd-4240-af0c-4ce7c24bc599\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-5c3374-7969595c9d-gzhtd" Oct 30 23:52:25.702886 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:25.702866 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a65bb94-e7dd-4240-af0c-4ce7c24bc599-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-5c3374-7969595c9d-gzhtd\" (UID: \"2a65bb94-e7dd-4240-af0c-4ce7c24bc599\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-5c3374-7969595c9d-gzhtd" Oct 30 23:52:25.768891 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:25.768848 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qn57\" (UniqueName: \"kubernetes.io/projected/2a65bb94-e7dd-4240-af0c-4ce7c24bc599-kube-api-access-8qn57\") pod \"payload-processing-e2e-worker-w1-5c3374-7969595c9d-gzhtd\" (UID: \"2a65bb94-e7dd-4240-af0c-4ce7c24bc599\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-5c3374-7969595c9d-gzhtd" Oct 30 23:52:25.797203 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:25.797170 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-5c3374-7969595c9d-gzhtd" Oct 30 23:52:26.060431 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:26.060401 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-5c3374-f56fc6f79-7b9pp"] Oct 30 23:52:26.060742 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:52:26.060711 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5c1c1eb_bce5_4ef6_8246_c0638f5c8b6b.slice/crio-b23f7b1a3caf94a6985bc5d55abc05ba7b6bf1e8df6271fd86e359f82377aae3 WatchSource:0}: Error finding container b23f7b1a3caf94a6985bc5d55abc05ba7b6bf1e8df6271fd86e359f82377aae3: Status 404 returned error can't find the container with id b23f7b1a3caf94a6985bc5d55abc05ba7b6bf1e8df6271fd86e359f82377aae3 Oct 30 23:52:26.203104 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:26.203076 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-5c3374-7969595c9d-gzhtd"] Oct 30 23:52:26.204096 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:52:26.204070 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a65bb94_e7dd_4240_af0c_4ce7c24bc599.slice/crio-91a24e6a206b34c1e0747f2d580f16d228404b3dc22b0123043ce3a787d79fb7 WatchSource:0}: Error finding container 91a24e6a206b34c1e0747f2d580f16d228404b3dc22b0123043ce3a787d79fb7: Status 404 returned error can't find the container with id 91a24e6a206b34c1e0747f2d580f16d228404b3dc22b0123043ce3a787d79fb7 Oct 30 23:52:27.004394 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:27.004360 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5c3374-f56fc6f79-7b9pp" event={"ID":"f5c1c1eb-bce5-4ef6-8246-c0638f5c8b6b","Type":"ContainerStarted","Data":"c672e5ecdf6d33493ca13caa3595740645f4f791d76e997b3f60a01689a07573"} Oct 30 23:52:27.004594 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:27.004423 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5c3374-f56fc6f79-7b9pp" event={"ID":"f5c1c1eb-bce5-4ef6-8246-c0638f5c8b6b","Type":"ContainerStarted","Data":"b23f7b1a3caf94a6985bc5d55abc05ba7b6bf1e8df6271fd86e359f82377aae3"} Oct 30 23:52:27.004594 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:27.004573 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5c3374-f56fc6f79-7b9pp" Oct 30 23:52:27.005941 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:27.005916 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-5c3374-7969595c9d-gzhtd" event={"ID":"2a65bb94-e7dd-4240-af0c-4ce7c24bc599","Type":"ContainerStarted","Data":"ad51b490904e9d709e16972f3397ee5aacb4b22fd5ecccd4ac695bf49d1c7695"} Oct 30 23:52:27.006045 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:27.005962 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-5c3374-7969595c9d-gzhtd" event={"ID":"2a65bb94-e7dd-4240-af0c-4ce7c24bc599","Type":"ContainerStarted","Data":"91a24e6a206b34c1e0747f2d580f16d228404b3dc22b0123043ce3a787d79fb7"} Oct 30 23:52:27.006045 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:27.005998 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-5c3374-7969595c9d-gzhtd" Oct 30 23:52:27.134179 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:27.134113 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5c3374-f56fc6f79-7b9pp" podStartSLOduration=2.134094739 podStartE2EDuration="2.134094739s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:52:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:52:27.132611163 +0000 UTC m=+660.168942000" watchObservedRunningTime="2026-10-30 23:52:27.134094739 +0000 UTC m=+660.170425562" Oct 30 23:52:27.937048 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:27.937016 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w5-fe01fa-698f78d94b-4wmlb" Oct 30 23:52:28.057668 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:28.057617 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w1-5c3374-7969595c9d-gzhtd" podStartSLOduration=3.057604592 podStartE2EDuration="3.057604592s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:52:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:52:27.317585213 +0000 UTC m=+660.353916051" watchObservedRunningTime="2026-10-30 23:52:28.057604592 +0000 UTC m=+661.093935458" Oct 30 23:52:29.553379 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:29.553345 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:52:29.764951 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:29.764182 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:52:33.055501 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:33.055471 4927 generic.go:356] "Generic (PLEG): container finished" podID="8844df03-4076-4f22-94bd-49a4514c21ff" containerID="2eb5077930c6b1e1cddac4a30c1a767126179b165ee2d731d11d0be2e41468fe" exitCode=0 Oct 30 23:52:33.055919 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:33.055530 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-678tp" event={"ID":"8844df03-4076-4f22-94bd-49a4514c21ff","Type":"ContainerDied","Data":"2eb5077930c6b1e1cddac4a30c1a767126179b165ee2d731d11d0be2e41468fe"} Oct 30 23:52:33.055919 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:33.055905 4927 scope.go:122] "RemoveContainer" containerID="2eb5077930c6b1e1cddac4a30c1a767126179b165ee2d731d11d0be2e41468fe" Oct 30 23:52:33.057204 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:33.057178 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 23:52:33.057267 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:33.057246 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 23:52:33.057303 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:33.057275 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 23:52:34.068813 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:34.068776 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-678tp" event={"ID":"8844df03-4076-4f22-94bd-49a4514c21ff","Type":"ContainerStarted","Data":"54d3605313529f3a8d87d2eeeb44e571a9060e411a04b7802df8676f9d20bac3"} Oct 30 23:52:34.257148 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:34.257044 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:52:34.431350 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:34.431292 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:52:34.584915 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:34.584873 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:52:34.657547 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:34.657511 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-678tp" Oct 30 23:52:34.657930 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:34.657903 4927 patch_prober.go:29] interesting pod/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-678tp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Oct 30 23:52:34.658054 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:34.657953 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-678tp" podUID="8844df03-4076-4f22-94bd-49a4514c21ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Oct 30 23:52:34.994566 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:34.994533 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-46b586-669f464d79-fsqx4" Oct 30 23:52:35.663359 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:35.657790 4927 patch_prober.go:29] interesting pod/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-678tp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Oct 30 23:52:35.663359 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:35.657855 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-678tp" podUID="8844df03-4076-4f22-94bd-49a4514c21ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Oct 30 23:52:36.653832 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:36.653790 4927 patch_prober.go:29] interesting pod/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-678tp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Oct 30 23:52:36.654130 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:36.654103 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-678tp" podUID="8844df03-4076-4f22-94bd-49a4514c21ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Oct 30 23:52:37.653355 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:37.652730 4927 patch_prober.go:29] interesting pod/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-678tp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Oct 30 23:52:37.653355 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:37.652798 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-678tp" podUID="8844df03-4076-4f22-94bd-49a4514c21ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Oct 30 23:52:38.043353 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:38.042767 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5c3374-f56fc6f79-7b9pp" Oct 30 23:52:38.049209 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:38.048498 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-5c3374-7969595c9d-gzhtd" Oct 30 23:52:38.659358 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:38.658632 4927 patch_prober.go:29] interesting pod/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-678tp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Oct 30 23:52:38.659358 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:38.658705 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-678tp" podUID="8844df03-4076-4f22-94bd-49a4514c21ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Oct 30 23:52:38.961055 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:38.960940 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:52:39.196013 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:39.195974 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:52:39.655430 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:39.655397 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-678tp" Oct 30 23:52:39.655951 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:39.655835 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-678tp" Oct 30 23:52:39.656767 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:39.656747 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-678tp" Oct 30 23:52:40.772207 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:40.772170 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-xgpsz"] Oct 30 23:52:40.777526 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:40.777499 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-xgpsz" Oct 30 23:52:40.791993 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:40.791960 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6b778b5-4861-4a3f-85cb-5524098583f1-credential-socket\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-xgpsz\" (UID: \"e6b778b5-4861-4a3f-85cb-5524098583f1\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-xgpsz" Oct 30 23:52:40.792143 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:40.792070 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mtgd5\" (UniqueName: \"kubernetes.io/projected/e6b778b5-4861-4a3f-85cb-5524098583f1-kube-api-access-mtgd5\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-xgpsz\" (UID: \"e6b778b5-4861-4a3f-85cb-5524098583f1\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-xgpsz" Oct 30 23:52:40.792207 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:40.792156 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6b778b5-4861-4a3f-85cb-5524098583f1-istio-podinfo\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-xgpsz\" (UID: \"e6b778b5-4861-4a3f-85cb-5524098583f1\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-xgpsz" Oct 30 23:52:40.792269 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:40.792227 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6b778b5-4861-4a3f-85cb-5524098583f1-istio-envoy\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-xgpsz\" (UID: \"e6b778b5-4861-4a3f-85cb-5524098583f1\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-xgpsz" Oct 30 23:52:40.792324 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:40.792308 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6b778b5-4861-4a3f-85cb-5524098583f1-istio-token\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-xgpsz\" (UID: \"e6b778b5-4861-4a3f-85cb-5524098583f1\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-xgpsz" Oct 30 23:52:40.792408 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:40.792369 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6b778b5-4861-4a3f-85cb-5524098583f1-istiod-ca-cert\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-xgpsz\" (UID: \"e6b778b5-4861-4a3f-85cb-5524098583f1\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-xgpsz" Oct 30 23:52:40.792408 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:40.792386 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6b778b5-4861-4a3f-85cb-5524098583f1-istio-data\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-xgpsz\" (UID: \"e6b778b5-4861-4a3f-85cb-5524098583f1\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-xgpsz" Oct 30 23:52:40.792543 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:40.792514 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6b778b5-4861-4a3f-85cb-5524098583f1-workload-socket\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-xgpsz\" (UID: \"e6b778b5-4861-4a3f-85cb-5524098583f1\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-xgpsz" Oct 30 23:52:40.792620 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:40.792603 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6b778b5-4861-4a3f-85cb-5524098583f1-workload-certs\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-xgpsz\" (UID: \"e6b778b5-4861-4a3f-85cb-5524098583f1\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-xgpsz" Oct 30 23:52:40.893643 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:40.893603 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6b778b5-4861-4a3f-85cb-5524098583f1-istio-envoy\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-xgpsz\" (UID: \"e6b778b5-4861-4a3f-85cb-5524098583f1\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-xgpsz" Oct 30 23:52:40.893845 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:40.893689 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6b778b5-4861-4a3f-85cb-5524098583f1-istio-token\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-xgpsz\" (UID: \"e6b778b5-4861-4a3f-85cb-5524098583f1\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-xgpsz" Oct 30 23:52:40.893845 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:40.893721 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6b778b5-4861-4a3f-85cb-5524098583f1-istiod-ca-cert\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-xgpsz\" (UID: \"e6b778b5-4861-4a3f-85cb-5524098583f1\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-xgpsz" Oct 30 23:52:40.893845 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:40.893742 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6b778b5-4861-4a3f-85cb-5524098583f1-istio-data\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-xgpsz\" (UID: \"e6b778b5-4861-4a3f-85cb-5524098583f1\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-xgpsz" Oct 30 23:52:40.893845 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:40.893780 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6b778b5-4861-4a3f-85cb-5524098583f1-workload-socket\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-xgpsz\" (UID: \"e6b778b5-4861-4a3f-85cb-5524098583f1\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-xgpsz" Oct 30 23:52:40.893845 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:40.893832 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6b778b5-4861-4a3f-85cb-5524098583f1-workload-certs\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-xgpsz\" (UID: \"e6b778b5-4861-4a3f-85cb-5524098583f1\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-xgpsz" Oct 30 23:52:40.894172 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:40.893871 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6b778b5-4861-4a3f-85cb-5524098583f1-credential-socket\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-xgpsz\" (UID: \"e6b778b5-4861-4a3f-85cb-5524098583f1\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-xgpsz" Oct 30 23:52:40.894172 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:40.893959 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtgd5\" (UniqueName: \"kubernetes.io/projected/e6b778b5-4861-4a3f-85cb-5524098583f1-kube-api-access-mtgd5\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-xgpsz\" (UID: \"e6b778b5-4861-4a3f-85cb-5524098583f1\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-xgpsz" Oct 30 23:52:40.894172 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:40.894000 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6b778b5-4861-4a3f-85cb-5524098583f1-istio-podinfo\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-xgpsz\" (UID: \"e6b778b5-4861-4a3f-85cb-5524098583f1\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-xgpsz" Oct 30 23:52:40.894582 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:40.894444 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6b778b5-4861-4a3f-85cb-5524098583f1-istiod-ca-cert\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-xgpsz\" (UID: \"e6b778b5-4861-4a3f-85cb-5524098583f1\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-xgpsz" Oct 30 23:52:40.894732 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:40.894703 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6b778b5-4861-4a3f-85cb-5524098583f1-workload-certs\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-xgpsz\" (UID: \"e6b778b5-4861-4a3f-85cb-5524098583f1\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-xgpsz" Oct 30 23:52:40.894956 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:40.894931 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6b778b5-4861-4a3f-85cb-5524098583f1-workload-socket\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-xgpsz\" (UID: \"e6b778b5-4861-4a3f-85cb-5524098583f1\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-xgpsz" Oct 30 23:52:40.895199 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:40.895175 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6b778b5-4861-4a3f-85cb-5524098583f1-credential-socket\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-xgpsz\" (UID: \"e6b778b5-4861-4a3f-85cb-5524098583f1\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-xgpsz" Oct 30 23:52:40.895480 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:40.895456 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6b778b5-4861-4a3f-85cb-5524098583f1-istio-data\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-xgpsz\" (UID: \"e6b778b5-4861-4a3f-85cb-5524098583f1\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-xgpsz" Oct 30 23:52:40.896615 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:40.896593 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6b778b5-4861-4a3f-85cb-5524098583f1-istio-envoy\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-xgpsz\" (UID: \"e6b778b5-4861-4a3f-85cb-5524098583f1\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-xgpsz" Oct 30 23:52:40.897029 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:40.897008 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6b778b5-4861-4a3f-85cb-5524098583f1-istio-podinfo\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-xgpsz\" (UID: \"e6b778b5-4861-4a3f-85cb-5524098583f1\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-xgpsz" Oct 30 23:52:40.916781 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:40.916746 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-xgpsz"] Oct 30 23:52:40.967025 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:40.966988 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6b778b5-4861-4a3f-85cb-5524098583f1-istio-token\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-xgpsz\" (UID: \"e6b778b5-4861-4a3f-85cb-5524098583f1\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-xgpsz" Oct 30 23:52:40.967179 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:40.967161 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mtgd5\" (UniqueName: \"kubernetes.io/projected/e6b778b5-4861-4a3f-85cb-5524098583f1-kube-api-access-mtgd5\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-xgpsz\" (UID: \"e6b778b5-4861-4a3f-85cb-5524098583f1\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-xgpsz" Oct 30 23:52:41.092854 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:41.092766 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-xgpsz" Oct 30 23:52:41.542654 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:41.542624 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-xgpsz"] Oct 30 23:52:41.543135 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:52:41.543110 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6b778b5_4861_4a3f_85cb_5524098583f1.slice/crio-c80c8ebf1c96d05ba6484b49689cd0bb0896ec05eb54465160de443bb1e3b6ee WatchSource:0}: Error finding container c80c8ebf1c96d05ba6484b49689cd0bb0896ec05eb54465160de443bb1e3b6ee: Status 404 returned error can't find the container with id c80c8ebf1c96d05ba6484b49689cd0bb0896ec05eb54465160de443bb1e3b6ee Oct 30 23:52:41.546124 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:41.546090 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 23:52:41.546211 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:41.546170 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 23:52:41.546281 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:41.546218 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 23:52:42.164642 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:42.164604 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-xgpsz" event={"ID":"e6b778b5-4861-4a3f-85cb-5524098583f1","Type":"ContainerStarted","Data":"dae44a8554e1269e47872205987bfe082bbc0e2df1fa6679ab5137a80ab6ad25"} Oct 30 23:52:42.165036 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:42.164667 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-xgpsz" event={"ID":"e6b778b5-4861-4a3f-85cb-5524098583f1","Type":"ContainerStarted","Data":"c80c8ebf1c96d05ba6484b49689cd0bb0896ec05eb54465160de443bb1e3b6ee"} Oct 30 23:52:42.401322 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:42.401261 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-xgpsz" podStartSLOduration=2.401245321 podStartE2EDuration="2.401245321s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:52:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:52:42.347530103 +0000 UTC m=+675.383860936" watchObservedRunningTime="2026-10-30 23:52:42.401245321 +0000 UTC m=+675.437576140" Oct 30 23:52:42.779489 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:42.779400 4927 patch_prober.go:29] interesting pod/e2e-worker-w6-35a187-openshift-default-5b54c65f75-94tkp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Oct 30 23:52:42.779489 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:42.779454 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-94tkp" podUID="9f348596-394c-4bfb-ba79-4788da8fc6de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Oct 30 23:52:43.093873 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:43.093794 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-xgpsz" Oct 30 23:52:43.095615 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:43.095590 4927 patch_prober.go:29] interesting pod/e2e-worker-w6-35a187-openshift-default-5b54c65f75-xgpsz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Oct 30 23:52:43.095737 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:43.095636 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-xgpsz" podUID="e6b778b5-4861-4a3f-85cb-5524098583f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Oct 30 23:52:43.172598 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:43.172565 4927 generic.go:356] "Generic (PLEG): container finished" podID="9f348596-394c-4bfb-ba79-4788da8fc6de" containerID="65e5c2072ab717203bf04484e90c092ed6756580261f380bc47fb601fcba23d1" exitCode=0 Oct 30 23:52:43.173109 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:43.172650 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-94tkp" event={"ID":"9f348596-394c-4bfb-ba79-4788da8fc6de","Type":"ContainerDied","Data":"65e5c2072ab717203bf04484e90c092ed6756580261f380bc47fb601fcba23d1"} Oct 30 23:52:43.173109 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:43.173084 4927 scope.go:122] "RemoveContainer" containerID="65e5c2072ab717203bf04484e90c092ed6756580261f380bc47fb601fcba23d1" Oct 30 23:52:43.174438 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:43.174410 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 23:52:43.174508 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:43.174474 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 23:52:43.174549 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:43.174509 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 23:52:43.839274 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:43.839230 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:52:44.093212 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:44.093132 4927 patch_prober.go:29] interesting pod/e2e-worker-w6-35a187-openshift-default-5b54c65f75-xgpsz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Oct 30 23:52:44.093212 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:44.093181 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-xgpsz" podUID="e6b778b5-4861-4a3f-85cb-5524098583f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Oct 30 23:52:44.182857 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:44.182821 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-94tkp" event={"ID":"9f348596-394c-4bfb-ba79-4788da8fc6de","Type":"ContainerStarted","Data":"9ea8e8a0622bd064854d976900a471b591cf0ad747977a10109b496f5289b8df"} Oct 30 23:52:44.597177 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:44.597145 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-94tkp" Oct 30 23:52:44.597535 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:44.597511 4927 patch_prober.go:29] interesting pod/e2e-worker-w6-35a187-openshift-default-5b54c65f75-94tkp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Oct 30 23:52:44.597689 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:44.597667 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-94tkp" podUID="9f348596-394c-4bfb-ba79-4788da8fc6de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Oct 30 23:52:45.093519 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:45.093489 4927 patch_prober.go:29] interesting pod/e2e-worker-w6-35a187-openshift-default-5b54c65f75-xgpsz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Oct 30 23:52:45.093765 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:45.093744 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-xgpsz" podUID="e6b778b5-4861-4a3f-85cb-5524098583f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Oct 30 23:52:45.596870 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:45.596826 4927 patch_prober.go:29] interesting pod/e2e-worker-w6-35a187-openshift-default-5b54c65f75-94tkp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Oct 30 23:52:45.597400 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:45.596884 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-94tkp" podUID="9f348596-394c-4bfb-ba79-4788da8fc6de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Oct 30 23:52:46.093781 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:46.093747 4927 patch_prober.go:29] interesting pod/e2e-worker-w6-35a187-openshift-default-5b54c65f75-xgpsz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Oct 30 23:52:46.093957 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:46.093801 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-xgpsz" podUID="e6b778b5-4861-4a3f-85cb-5524098583f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Oct 30 23:52:46.597521 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:46.597480 4927 patch_prober.go:29] interesting pod/e2e-worker-w6-35a187-openshift-default-5b54c65f75-94tkp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Oct 30 23:52:46.598022 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:46.597533 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-94tkp" podUID="9f348596-394c-4bfb-ba79-4788da8fc6de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Oct 30 23:52:47.101888 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:47.101852 4927 patch_prober.go:29] interesting pod/e2e-worker-w6-35a187-openshift-default-5b54c65f75-xgpsz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Oct 30 23:52:47.102149 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:47.102125 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-xgpsz" podUID="e6b778b5-4861-4a3f-85cb-5524098583f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Oct 30 23:52:47.597327 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:47.597293 4927 patch_prober.go:29] interesting pod/e2e-worker-w6-35a187-openshift-default-5b54c65f75-94tkp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Oct 30 23:52:47.597501 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:47.597370 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-94tkp" podUID="9f348596-394c-4bfb-ba79-4788da8fc6de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Oct 30 23:52:48.097457 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:48.097426 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-xgpsz" Oct 30 23:52:48.097963 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:48.097826 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-xgpsz" Oct 30 23:52:48.098531 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:48.098511 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-xgpsz" Oct 30 23:52:48.580868 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:48.580828 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:52:48.599513 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:48.598699 4927 patch_prober.go:29] interesting pod/e2e-worker-w6-35a187-openshift-default-5b54c65f75-94tkp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Oct 30 23:52:48.604727 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:48.598761 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-94tkp" podUID="9f348596-394c-4bfb-ba79-4788da8fc6de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Oct 30 23:52:49.687307 ip-10-0-141-180 kubenswrapper[4927]: E1030 23:52:49.687271 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sqv69" containerName="extractor" Oct 30 23:52:49.779758 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:49.779728 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-94tkp" Oct 30 23:52:49.780473 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:49.780444 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-94tkp" Oct 30 23:52:49.782168 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:49.782135 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-94tkp" Oct 30 23:52:49.891848 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:49.891799 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:52:50.527233 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:50.527037 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:52:50.689020 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:50.688968 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:52:53.615536 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:53.615496 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:52:54.328006 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:54.327965 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:52:55.099399 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:55.099359 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:52:55.750866 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:55.750834 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:52:56.051351 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:56.051258 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:52:56.281889 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:56.281801 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-tgc8q"] Oct 30 23:52:56.285706 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:56.285682 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-tgc8q" Oct 30 23:52:56.330743 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:56.330666 4927 generic.go:356] "Generic (PLEG): container finished" podID="a23bdae1-1e2a-4e26-8e2b-43d7b50a800e" containerID="66fe27523fed3875d511f1effa6f7cdd362683d0e88c130474cb5f5fb93ff4fd" exitCode=0 Oct 30 23:52:56.330900 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:56.330742 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-jqc5p" event={"ID":"a23bdae1-1e2a-4e26-8e2b-43d7b50a800e","Type":"ContainerDied","Data":"66fe27523fed3875d511f1effa6f7cdd362683d0e88c130474cb5f5fb93ff4fd"} Oct 30 23:52:56.331160 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:56.331143 4927 scope.go:122] "RemoveContainer" containerID="66fe27523fed3875d511f1effa6f7cdd362683d0e88c130474cb5f5fb93ff4fd" Oct 30 23:52:56.332448 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:56.332420 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 23:52:56.332506 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:56.332490 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 23:52:56.332543 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:56.332519 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 23:52:56.342730 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:56.342711 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-46b586-openshift-default-dockercfg-5bltr\"" Oct 30 23:52:56.387890 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:56.387855 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e8b64ac-0b2a-40aa-ab07-25ec13005517-workload-socket\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-tgc8q\" (UID: \"3e8b64ac-0b2a-40aa-ab07-25ec13005517\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-tgc8q" Oct 30 23:52:56.388036 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:56.387987 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e8b64ac-0b2a-40aa-ab07-25ec13005517-istiod-ca-cert\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-tgc8q\" (UID: \"3e8b64ac-0b2a-40aa-ab07-25ec13005517\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-tgc8q" Oct 30 23:52:56.388110 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:56.388082 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e8b64ac-0b2a-40aa-ab07-25ec13005517-istio-podinfo\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-tgc8q\" (UID: \"3e8b64ac-0b2a-40aa-ab07-25ec13005517\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-tgc8q" Oct 30 23:52:56.388175 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:56.388142 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e8b64ac-0b2a-40aa-ab07-25ec13005517-workload-certs\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-tgc8q\" (UID: \"3e8b64ac-0b2a-40aa-ab07-25ec13005517\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-tgc8q" Oct 30 23:52:56.388175 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:56.388168 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e8b64ac-0b2a-40aa-ab07-25ec13005517-istio-token\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-tgc8q\" (UID: \"3e8b64ac-0b2a-40aa-ab07-25ec13005517\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-tgc8q" Oct 30 23:52:56.388295 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:56.388218 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4cqr\" (UniqueName: \"kubernetes.io/projected/3e8b64ac-0b2a-40aa-ab07-25ec13005517-kube-api-access-g4cqr\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-tgc8q\" (UID: \"3e8b64ac-0b2a-40aa-ab07-25ec13005517\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-tgc8q" Oct 30 23:52:56.388295 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:56.388257 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e8b64ac-0b2a-40aa-ab07-25ec13005517-credential-socket\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-tgc8q\" (UID: \"3e8b64ac-0b2a-40aa-ab07-25ec13005517\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-tgc8q" Oct 30 23:52:56.388472 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:56.388294 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e8b64ac-0b2a-40aa-ab07-25ec13005517-istio-data\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-tgc8q\" (UID: \"3e8b64ac-0b2a-40aa-ab07-25ec13005517\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-tgc8q" Oct 30 23:52:56.388472 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:56.388406 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e8b64ac-0b2a-40aa-ab07-25ec13005517-istio-envoy\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-tgc8q\" (UID: \"3e8b64ac-0b2a-40aa-ab07-25ec13005517\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-tgc8q" Oct 30 23:52:56.396962 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:56.396941 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-tgc8q"] Oct 30 23:52:56.490273 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:56.490234 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e8b64ac-0b2a-40aa-ab07-25ec13005517-workload-socket\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-tgc8q\" (UID: \"3e8b64ac-0b2a-40aa-ab07-25ec13005517\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-tgc8q" Oct 30 23:52:56.490458 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:56.490290 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e8b64ac-0b2a-40aa-ab07-25ec13005517-istiod-ca-cert\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-tgc8q\" (UID: \"3e8b64ac-0b2a-40aa-ab07-25ec13005517\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-tgc8q" Oct 30 23:52:56.490458 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:56.490326 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e8b64ac-0b2a-40aa-ab07-25ec13005517-istio-podinfo\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-tgc8q\" (UID: \"3e8b64ac-0b2a-40aa-ab07-25ec13005517\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-tgc8q" Oct 30 23:52:56.490458 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:56.490378 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e8b64ac-0b2a-40aa-ab07-25ec13005517-workload-certs\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-tgc8q\" (UID: \"3e8b64ac-0b2a-40aa-ab07-25ec13005517\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-tgc8q" Oct 30 23:52:56.490458 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:56.490403 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e8b64ac-0b2a-40aa-ab07-25ec13005517-istio-token\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-tgc8q\" (UID: \"3e8b64ac-0b2a-40aa-ab07-25ec13005517\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-tgc8q" Oct 30 23:52:56.490458 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:56.490451 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4cqr\" (UniqueName: \"kubernetes.io/projected/3e8b64ac-0b2a-40aa-ab07-25ec13005517-kube-api-access-g4cqr\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-tgc8q\" (UID: \"3e8b64ac-0b2a-40aa-ab07-25ec13005517\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-tgc8q" Oct 30 23:52:56.490844 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:56.490490 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e8b64ac-0b2a-40aa-ab07-25ec13005517-credential-socket\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-tgc8q\" (UID: \"3e8b64ac-0b2a-40aa-ab07-25ec13005517\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-tgc8q" Oct 30 23:52:56.490844 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:56.490530 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e8b64ac-0b2a-40aa-ab07-25ec13005517-istio-data\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-tgc8q\" (UID: \"3e8b64ac-0b2a-40aa-ab07-25ec13005517\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-tgc8q" Oct 30 23:52:56.490844 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:56.490579 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e8b64ac-0b2a-40aa-ab07-25ec13005517-istio-envoy\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-tgc8q\" (UID: \"3e8b64ac-0b2a-40aa-ab07-25ec13005517\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-tgc8q" Oct 30 23:52:56.490844 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:56.490779 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e8b64ac-0b2a-40aa-ab07-25ec13005517-workload-socket\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-tgc8q\" (UID: \"3e8b64ac-0b2a-40aa-ab07-25ec13005517\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-tgc8q" Oct 30 23:52:56.491061 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:56.490888 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e8b64ac-0b2a-40aa-ab07-25ec13005517-workload-certs\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-tgc8q\" (UID: \"3e8b64ac-0b2a-40aa-ab07-25ec13005517\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-tgc8q" Oct 30 23:52:56.491114 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:56.491095 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e8b64ac-0b2a-40aa-ab07-25ec13005517-istiod-ca-cert\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-tgc8q\" (UID: \"3e8b64ac-0b2a-40aa-ab07-25ec13005517\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-tgc8q" Oct 30 23:52:56.491193 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:56.491162 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e8b64ac-0b2a-40aa-ab07-25ec13005517-credential-socket\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-tgc8q\" (UID: \"3e8b64ac-0b2a-40aa-ab07-25ec13005517\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-tgc8q" Oct 30 23:52:56.491549 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:56.491519 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e8b64ac-0b2a-40aa-ab07-25ec13005517-istio-data\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-tgc8q\" (UID: \"3e8b64ac-0b2a-40aa-ab07-25ec13005517\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-tgc8q" Oct 30 23:52:56.493000 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:56.492967 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e8b64ac-0b2a-40aa-ab07-25ec13005517-istio-envoy\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-tgc8q\" (UID: \"3e8b64ac-0b2a-40aa-ab07-25ec13005517\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-tgc8q" Oct 30 23:52:56.493417 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:56.493394 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e8b64ac-0b2a-40aa-ab07-25ec13005517-istio-podinfo\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-tgc8q\" (UID: \"3e8b64ac-0b2a-40aa-ab07-25ec13005517\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-tgc8q" Oct 30 23:52:56.596301 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:56.596221 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4cqr\" (UniqueName: \"kubernetes.io/projected/3e8b64ac-0b2a-40aa-ab07-25ec13005517-kube-api-access-g4cqr\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-tgc8q\" (UID: \"3e8b64ac-0b2a-40aa-ab07-25ec13005517\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-tgc8q" Oct 30 23:52:56.597240 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:56.597184 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e8b64ac-0b2a-40aa-ab07-25ec13005517-istio-token\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-tgc8q\" (UID: \"3e8b64ac-0b2a-40aa-ab07-25ec13005517\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-tgc8q" Oct 30 23:52:56.600066 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:56.600041 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-tgc8q" Oct 30 23:52:56.864585 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:56.864501 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:52:57.053906 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:57.053882 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-tgc8q"] Oct 30 23:52:57.055930 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:52:57.055897 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e8b64ac_0b2a_40aa_ab07_25ec13005517.slice/crio-eb0f9e5dd4bc9c042b076161a3da97fcb688899ae0642c52b637291ac5256b6d WatchSource:0}: Error finding container eb0f9e5dd4bc9c042b076161a3da97fcb688899ae0642c52b637291ac5256b6d: Status 404 returned error can't find the container with id eb0f9e5dd4bc9c042b076161a3da97fcb688899ae0642c52b637291ac5256b6d Oct 30 23:52:57.058429 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:57.058395 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 23:52:57.058515 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:57.058470 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 23:52:57.058515 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:57.058506 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 23:52:57.344629 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:57.344583 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-tgc8q" event={"ID":"3e8b64ac-0b2a-40aa-ab07-25ec13005517","Type":"ContainerStarted","Data":"d67f2bd16f7e33e47e54489c1a91fb2008dda3cfc24949e624a22d15d0ebf8e1"} Oct 30 23:52:57.345080 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:57.344659 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-tgc8q" event={"ID":"3e8b64ac-0b2a-40aa-ab07-25ec13005517","Type":"ContainerStarted","Data":"eb0f9e5dd4bc9c042b076161a3da97fcb688899ae0642c52b637291ac5256b6d"} Oct 30 23:52:57.347376 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:57.347165 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-jqc5p" event={"ID":"a23bdae1-1e2a-4e26-8e2b-43d7b50a800e","Type":"ContainerStarted","Data":"604282d2cb273a5b341a0126f2894ded0f1f72e73c53d2b17248a900ced00ea5"} Oct 30 23:52:57.480904 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:57.480837 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-tgc8q" podStartSLOduration=1.480822242 podStartE2EDuration="1.480822242s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:52:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:52:57.478259647 +0000 UTC m=+690.514590495" watchObservedRunningTime="2026-10-30 23:52:57.480822242 +0000 UTC m=+690.517153063" Oct 30 23:52:57.600737 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:57.600702 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-tgc8q" Oct 30 23:52:57.778405 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:57.777806 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-jqc5p" Oct 30 23:52:57.944720 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:57.944684 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-jqc5p" Oct 30 23:52:57.948042 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:57.948011 4927 patch_prober.go:29] interesting pod/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-jqc5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Oct 30 23:52:57.948259 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:57.948234 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-jqc5p" podUID="a23bdae1-1e2a-4e26-8e2b-43d7b50a800e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Oct 30 23:52:58.602294 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:58.602255 4927 patch_prober.go:29] interesting pod/e2e-worker-w0-46b586-openshift-default-86c598c96c-tgc8q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Oct 30 23:52:58.602733 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:58.602353 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-tgc8q" podUID="3e8b64ac-0b2a-40aa-ab07-25ec13005517" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Oct 30 23:52:58.944761 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:58.944722 4927 patch_prober.go:29] interesting pod/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-jqc5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Oct 30 23:52:58.945085 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:58.945055 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-jqc5p" podUID="a23bdae1-1e2a-4e26-8e2b-43d7b50a800e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Oct 30 23:52:58.993356 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:58.990496 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:52:59.600852 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:59.600822 4927 patch_prober.go:29] interesting pod/e2e-worker-w0-46b586-openshift-default-86c598c96c-tgc8q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Oct 30 23:52:59.601064 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:59.600878 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-tgc8q" podUID="3e8b64ac-0b2a-40aa-ab07-25ec13005517" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Oct 30 23:52:59.946770 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:59.946732 4927 patch_prober.go:29] interesting pod/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-jqc5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Oct 30 23:52:59.948214 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:52:59.948182 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-jqc5p" podUID="a23bdae1-1e2a-4e26-8e2b-43d7b50a800e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Oct 30 23:53:00.601478 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:53:00.601441 4927 patch_prober.go:29] interesting pod/e2e-worker-w0-46b586-openshift-default-86c598c96c-tgc8q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Oct 30 23:53:00.601652 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:53:00.601507 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-tgc8q" podUID="3e8b64ac-0b2a-40aa-ab07-25ec13005517" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Oct 30 23:53:00.943918 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:53:00.943865 4927 patch_prober.go:29] interesting pod/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-jqc5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Oct 30 23:53:00.944434 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:53:00.944184 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-jqc5p" podUID="a23bdae1-1e2a-4e26-8e2b-43d7b50a800e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Oct 30 23:53:01.601643 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:53:01.601600 4927 patch_prober.go:29] interesting pod/e2e-worker-w0-46b586-openshift-default-86c598c96c-tgc8q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Oct 30 23:53:01.602138 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:53:01.601664 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-tgc8q" podUID="3e8b64ac-0b2a-40aa-ab07-25ec13005517" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Oct 30 23:53:01.739000 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:53:01.738962 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:53:01.946772 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:53:01.946729 4927 patch_prober.go:29] interesting pod/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-jqc5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Oct 30 23:53:01.947068 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:53:01.947040 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-jqc5p" podUID="a23bdae1-1e2a-4e26-8e2b-43d7b50a800e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Oct 30 23:53:02.357941 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:53:02.357848 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:53:02.557469 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:53:02.557428 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:53:02.601034 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:53:02.600988 4927 patch_prober.go:29] interesting pod/e2e-worker-w0-46b586-openshift-default-86c598c96c-tgc8q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Oct 30 23:53:02.601210 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:53:02.601082 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-tgc8q" podUID="3e8b64ac-0b2a-40aa-ab07-25ec13005517" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Oct 30 23:53:02.947574 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:53:02.947538 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-jqc5p" Oct 30 23:53:02.948831 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:53:02.948805 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-jqc5p" Oct 30 23:53:03.605941 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:53:03.605897 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-tgc8q" Oct 30 23:53:03.606310 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:53:03.606288 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-tgc8q" Oct 30 23:53:03.607034 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:53:03.607007 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-tgc8q" Oct 30 23:53:04.214786 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:53:04.214756 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:53:05.886670 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:53:05.886636 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:53:06.582213 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:53:06.582181 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:53:06.821065 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:53:06.821025 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:53:09.366930 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:53:09.366880 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:53:14.143876 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:53:14.143831 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:53:16.363941 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:53:16.363905 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:53:17.061366 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:53:17.061317 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:53:17.304108 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:53:17.304077 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:53:18.763316 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:53:18.763273 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:53:19.320152 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:53:19.320113 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:53:19.678124 ip-10-0-141-180 kubenswrapper[4927]: E1030 23:53:19.678088 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sqv69" containerName="extractor" Oct 30 23:53:20.490352 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:53:20.490290 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:53:20.999310 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:53:20.999274 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:53:21.166842 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:53:21.166800 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:53:21.438632 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:53:21.438596 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:53:24.278959 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:53:24.278923 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:53:26.822920 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:53:26.822877 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:53:27.038798 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:53:27.038744 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:53:27.491152 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:53:27.491100 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:53:27.691372 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:53:27.691288 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:53:29.348387 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:53:29.348350 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:53:34.083608 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:53:34.083573 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:53:35.220020 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:53:35.219976 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:53:38.997963 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:53:38.997927 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:53:43.615715 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:53:43.615678 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:53:46.105125 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:53:46.105091 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:53:48.319719 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:53:48.319681 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:53:49.677911 ip-10-0-141-180 kubenswrapper[4927]: E1030 23:53:49.677878 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sqv69" containerName="extractor" Oct 30 23:53:52.923089 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:53:52.923048 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:53:55.910318 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:53:55.910283 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:53:57.479145 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:53:57.479096 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:54:02.084509 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:54:02.084472 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:54:06.691194 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:54:06.691104 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:54:06.757805 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:54:06.757766 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-46b586/simulator-w0-46b586-kserve-67588b8dbd-z8xmn"] Oct 30 23:54:06.762570 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:54:06.762546 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-46b586/simulator-w0-46b586-kserve-67588b8dbd-z8xmn" Oct 30 23:54:06.822229 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:54:06.822195 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vc5g2\" (UniqueName: \"kubernetes.io/projected/9a25e252-223d-41f2-abf7-632067c77e0b-kube-api-access-vc5g2\") pod \"simulator-w0-46b586-kserve-67588b8dbd-z8xmn\" (UID: \"9a25e252-223d-41f2-abf7-632067c77e0b\") " pod="e2e-models-e2e-worker-w0-46b586/simulator-w0-46b586-kserve-67588b8dbd-z8xmn" Oct 30 23:54:06.822409 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:54:06.822318 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9a25e252-223d-41f2-abf7-632067c77e0b-model-cache\") pod \"simulator-w0-46b586-kserve-67588b8dbd-z8xmn\" (UID: \"9a25e252-223d-41f2-abf7-632067c77e0b\") " pod="e2e-models-e2e-worker-w0-46b586/simulator-w0-46b586-kserve-67588b8dbd-z8xmn" Oct 30 23:54:06.822409 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:54:06.822396 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9a25e252-223d-41f2-abf7-632067c77e0b-dshm\") pod \"simulator-w0-46b586-kserve-67588b8dbd-z8xmn\" (UID: \"9a25e252-223d-41f2-abf7-632067c77e0b\") " pod="e2e-models-e2e-worker-w0-46b586/simulator-w0-46b586-kserve-67588b8dbd-z8xmn" Oct 30 23:54:06.822490 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:54:06.822420 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9a25e252-223d-41f2-abf7-632067c77e0b-home\") pod \"simulator-w0-46b586-kserve-67588b8dbd-z8xmn\" (UID: \"9a25e252-223d-41f2-abf7-632067c77e0b\") " pod="e2e-models-e2e-worker-w0-46b586/simulator-w0-46b586-kserve-67588b8dbd-z8xmn" Oct 30 23:54:06.822490 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:54:06.822439 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9a25e252-223d-41f2-abf7-632067c77e0b-tmp-dir\") pod \"simulator-w0-46b586-kserve-67588b8dbd-z8xmn\" (UID: \"9a25e252-223d-41f2-abf7-632067c77e0b\") " pod="e2e-models-e2e-worker-w0-46b586/simulator-w0-46b586-kserve-67588b8dbd-z8xmn" Oct 30 23:54:06.822599 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:54:06.822564 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9a25e252-223d-41f2-abf7-632067c77e0b-tls-certs\") pod \"simulator-w0-46b586-kserve-67588b8dbd-z8xmn\" (UID: \"9a25e252-223d-41f2-abf7-632067c77e0b\") " pod="e2e-models-e2e-worker-w0-46b586/simulator-w0-46b586-kserve-67588b8dbd-z8xmn" Oct 30 23:54:06.837723 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:54:06.837692 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-46b586/simulator-w0-46b586-kserve-67588b8dbd-z8xmn"] Oct 30 23:54:06.871172 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:54:06.871144 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-46b586\"/\"openshift-service-ca.crt\"" Oct 30 23:54:06.871365 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:54:06.871211 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-46b586\"/\"default-dockercfg-fst44\"" Oct 30 23:54:06.871365 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:54:06.871228 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-46b586\"/\"kube-root-ca.crt\"" Oct 30 23:54:06.871847 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:54:06.871832 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-46b586\"/\"simulator-w0-46b586-kserve-self-signed-certs\"" Oct 30 23:54:06.932396 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:54:06.932319 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9a25e252-223d-41f2-abf7-632067c77e0b-model-cache\") pod \"simulator-w0-46b586-kserve-67588b8dbd-z8xmn\" (UID: \"9a25e252-223d-41f2-abf7-632067c77e0b\") " pod="e2e-models-e2e-worker-w0-46b586/simulator-w0-46b586-kserve-67588b8dbd-z8xmn" Oct 30 23:54:06.932686 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:54:06.932407 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9a25e252-223d-41f2-abf7-632067c77e0b-dshm\") pod \"simulator-w0-46b586-kserve-67588b8dbd-z8xmn\" (UID: \"9a25e252-223d-41f2-abf7-632067c77e0b\") " pod="e2e-models-e2e-worker-w0-46b586/simulator-w0-46b586-kserve-67588b8dbd-z8xmn" Oct 30 23:54:06.932686 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:54:06.932445 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9a25e252-223d-41f2-abf7-632067c77e0b-home\") pod \"simulator-w0-46b586-kserve-67588b8dbd-z8xmn\" (UID: \"9a25e252-223d-41f2-abf7-632067c77e0b\") " pod="e2e-models-e2e-worker-w0-46b586/simulator-w0-46b586-kserve-67588b8dbd-z8xmn" Oct 30 23:54:06.932686 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:54:06.932482 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9a25e252-223d-41f2-abf7-632067c77e0b-tmp-dir\") pod \"simulator-w0-46b586-kserve-67588b8dbd-z8xmn\" (UID: \"9a25e252-223d-41f2-abf7-632067c77e0b\") " pod="e2e-models-e2e-worker-w0-46b586/simulator-w0-46b586-kserve-67588b8dbd-z8xmn" Oct 30 23:54:06.932686 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:54:06.932536 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9a25e252-223d-41f2-abf7-632067c77e0b-tls-certs\") pod \"simulator-w0-46b586-kserve-67588b8dbd-z8xmn\" (UID: \"9a25e252-223d-41f2-abf7-632067c77e0b\") " pod="e2e-models-e2e-worker-w0-46b586/simulator-w0-46b586-kserve-67588b8dbd-z8xmn" Oct 30 23:54:06.932686 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:54:06.932665 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vc5g2\" (UniqueName: \"kubernetes.io/projected/9a25e252-223d-41f2-abf7-632067c77e0b-kube-api-access-vc5g2\") pod \"simulator-w0-46b586-kserve-67588b8dbd-z8xmn\" (UID: \"9a25e252-223d-41f2-abf7-632067c77e0b\") " pod="e2e-models-e2e-worker-w0-46b586/simulator-w0-46b586-kserve-67588b8dbd-z8xmn" Oct 30 23:54:06.937040 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:54:06.933474 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9a25e252-223d-41f2-abf7-632067c77e0b-model-cache\") pod \"simulator-w0-46b586-kserve-67588b8dbd-z8xmn\" (UID: \"9a25e252-223d-41f2-abf7-632067c77e0b\") " pod="e2e-models-e2e-worker-w0-46b586/simulator-w0-46b586-kserve-67588b8dbd-z8xmn" Oct 30 23:54:06.937611 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:54:06.937323 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:54:06.942061 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:54:06.938147 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9a25e252-223d-41f2-abf7-632067c77e0b-tmp-dir\") pod \"simulator-w0-46b586-kserve-67588b8dbd-z8xmn\" (UID: \"9a25e252-223d-41f2-abf7-632067c77e0b\") " pod="e2e-models-e2e-worker-w0-46b586/simulator-w0-46b586-kserve-67588b8dbd-z8xmn" Oct 30 23:54:06.942061 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:54:06.938458 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9a25e252-223d-41f2-abf7-632067c77e0b-home\") pod \"simulator-w0-46b586-kserve-67588b8dbd-z8xmn\" (UID: \"9a25e252-223d-41f2-abf7-632067c77e0b\") " pod="e2e-models-e2e-worker-w0-46b586/simulator-w0-46b586-kserve-67588b8dbd-z8xmn" Oct 30 23:54:06.943827 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:54:06.943785 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9a25e252-223d-41f2-abf7-632067c77e0b-tls-certs\") pod \"simulator-w0-46b586-kserve-67588b8dbd-z8xmn\" (UID: \"9a25e252-223d-41f2-abf7-632067c77e0b\") " pod="e2e-models-e2e-worker-w0-46b586/simulator-w0-46b586-kserve-67588b8dbd-z8xmn" Oct 30 23:54:06.944058 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:54:06.944037 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9a25e252-223d-41f2-abf7-632067c77e0b-dshm\") pod \"simulator-w0-46b586-kserve-67588b8dbd-z8xmn\" (UID: \"9a25e252-223d-41f2-abf7-632067c77e0b\") " pod="e2e-models-e2e-worker-w0-46b586/simulator-w0-46b586-kserve-67588b8dbd-z8xmn" Oct 30 23:54:06.995761 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:54:06.995726 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vc5g2\" (UniqueName: \"kubernetes.io/projected/9a25e252-223d-41f2-abf7-632067c77e0b-kube-api-access-vc5g2\") pod \"simulator-w0-46b586-kserve-67588b8dbd-z8xmn\" (UID: \"9a25e252-223d-41f2-abf7-632067c77e0b\") " pod="e2e-models-e2e-worker-w0-46b586/simulator-w0-46b586-kserve-67588b8dbd-z8xmn" Oct 30 23:54:07.074373 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:54:07.074317 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-46b586/simulator-w0-46b586-kserve-67588b8dbd-z8xmn" Oct 30 23:54:07.461815 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:54:07.461785 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-46b586/simulator-w0-46b586-kserve-67588b8dbd-z8xmn"] Oct 30 23:54:07.462971 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:54:07.462944 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9a25e252_223d_41f2_abf7_632067c77e0b.slice/crio-357a6258b7eb20df4d89943a06535cf9c4b20ecaa34582e2e0ab20eb8dfba36f WatchSource:0}: Error finding container 357a6258b7eb20df4d89943a06535cf9c4b20ecaa34582e2e0ab20eb8dfba36f: Status 404 returned error can't find the container with id 357a6258b7eb20df4d89943a06535cf9c4b20ecaa34582e2e0ab20eb8dfba36f Oct 30 23:54:07.915343 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:54:07.915296 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-46b586/simulator-w0-46b586-kserve-67588b8dbd-z8xmn" event={"ID":"9a25e252-223d-41f2-abf7-632067c77e0b","Type":"ContainerStarted","Data":"5fc487df72f1e3d46eb58939813b297b3c070c8821cfd9ee0a2a6f85ec2c70a6"} Oct 30 23:54:07.915747 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:54:07.915356 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-46b586/simulator-w0-46b586-kserve-67588b8dbd-z8xmn" event={"ID":"9a25e252-223d-41f2-abf7-632067c77e0b","Type":"ContainerStarted","Data":"357a6258b7eb20df4d89943a06535cf9c4b20ecaa34582e2e0ab20eb8dfba36f"} Oct 30 23:54:08.042515 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:54:08.042460 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-46b586/simulator-w0-46b586-kserve-67588b8dbd-z8xmn" podStartSLOduration=2.042446782 podStartE2EDuration="2.042446782s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:54:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:54:08.040635923 +0000 UTC m=+761.076966746" watchObservedRunningTime="2026-10-30 23:54:08.042446782 +0000 UTC m=+761.078777604" Oct 30 23:54:08.074745 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:54:08.074711 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-46b586/simulator-w0-46b586-kserve-67588b8dbd-z8xmn" Oct 30 23:54:11.372739 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:54:11.372695 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:54:13.970506 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:54:13.970466 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:54:15.996787 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:54:15.996754 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:54:17.074604 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:54:17.074566 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-46b586/simulator-w0-46b586-kserve-67588b8dbd-z8xmn" Oct 30 23:54:17.088625 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:54:17.088599 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-46b586/simulator-w0-46b586-kserve-67588b8dbd-z8xmn" Oct 30 23:54:18.002125 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:54:18.002095 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-46b586/simulator-w0-46b586-kserve-67588b8dbd-z8xmn" Oct 30 23:54:19.677893 ip-10-0-141-180 kubenswrapper[4927]: E1030 23:54:19.677858 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sqv69" containerName="extractor" Oct 30 23:54:20.701420 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:54:20.701386 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:54:21.335286 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:54:21.335248 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:54:25.381865 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:54:25.381829 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:54:28.042211 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:54:28.042172 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:54:29.991793 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:54:29.991753 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:54:34.497947 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:54:34.497904 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:54:39.075517 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:54:39.075451 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:54:43.621381 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:54:43.621323 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:54:48.191779 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:54:48.191735 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:54:48.508387 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:54:48.508298 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-5c3374/premium-w1-5c3374-kserve-6c56d4dcd-llwcl"] Oct 30 23:54:48.512231 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:54:48.512214 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-5c3374/premium-w1-5c3374-kserve-6c56d4dcd-llwcl" Oct 30 23:54:48.572609 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:54:48.572573 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-5c3374/premium-w1-5c3374-kserve-6c56d4dcd-llwcl"] Oct 30 23:54:48.589467 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:54:48.589435 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc813f02-e077-4251-8c1e-fea0986b0f36-tmp-dir\") pod \"premium-w1-5c3374-kserve-6c56d4dcd-llwcl\" (UID: \"dc813f02-e077-4251-8c1e-fea0986b0f36\") " pod="e2e-models-e2e-worker-w1-5c3374/premium-w1-5c3374-kserve-6c56d4dcd-llwcl" Oct 30 23:54:48.589632 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:54:48.589479 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc813f02-e077-4251-8c1e-fea0986b0f36-dshm\") pod \"premium-w1-5c3374-kserve-6c56d4dcd-llwcl\" (UID: \"dc813f02-e077-4251-8c1e-fea0986b0f36\") " pod="e2e-models-e2e-worker-w1-5c3374/premium-w1-5c3374-kserve-6c56d4dcd-llwcl" Oct 30 23:54:48.589632 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:54:48.589579 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc813f02-e077-4251-8c1e-fea0986b0f36-model-cache\") pod \"premium-w1-5c3374-kserve-6c56d4dcd-llwcl\" (UID: \"dc813f02-e077-4251-8c1e-fea0986b0f36\") " pod="e2e-models-e2e-worker-w1-5c3374/premium-w1-5c3374-kserve-6c56d4dcd-llwcl" Oct 30 23:54:48.589632 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:54:48.589628 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dc813f02-e077-4251-8c1e-fea0986b0f36-home\") pod \"premium-w1-5c3374-kserve-6c56d4dcd-llwcl\" (UID: \"dc813f02-e077-4251-8c1e-fea0986b0f36\") " pod="e2e-models-e2e-worker-w1-5c3374/premium-w1-5c3374-kserve-6c56d4dcd-llwcl" Oct 30 23:54:48.589773 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:54:48.589723 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hkhrl\" (UniqueName: \"kubernetes.io/projected/dc813f02-e077-4251-8c1e-fea0986b0f36-kube-api-access-hkhrl\") pod \"premium-w1-5c3374-kserve-6c56d4dcd-llwcl\" (UID: \"dc813f02-e077-4251-8c1e-fea0986b0f36\") " pod="e2e-models-e2e-worker-w1-5c3374/premium-w1-5c3374-kserve-6c56d4dcd-llwcl" Oct 30 23:54:48.589773 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:54:48.589746 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc813f02-e077-4251-8c1e-fea0986b0f36-tls-certs\") pod \"premium-w1-5c3374-kserve-6c56d4dcd-llwcl\" (UID: \"dc813f02-e077-4251-8c1e-fea0986b0f36\") " pod="e2e-models-e2e-worker-w1-5c3374/premium-w1-5c3374-kserve-6c56d4dcd-llwcl" Oct 30 23:54:48.607096 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:54:48.607067 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:54:48.619073 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:54:48.619047 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-5c3374\"/\"kube-root-ca.crt\"" Oct 30 23:54:48.619193 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:54:48.619176 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-5c3374\"/\"openshift-service-ca.crt\"" Oct 30 23:54:48.619236 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:54:48.619202 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-5c3374\"/\"default-dockercfg-dhj9d\"" Oct 30 23:54:48.619275 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:54:48.619264 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-5c3374\"/\"premium-w1-5c3374-kserve-self-signed-certs\"" Oct 30 23:54:48.690842 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:54:48.690803 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc813f02-e077-4251-8c1e-fea0986b0f36-tmp-dir\") pod \"premium-w1-5c3374-kserve-6c56d4dcd-llwcl\" (UID: \"dc813f02-e077-4251-8c1e-fea0986b0f36\") " pod="e2e-models-e2e-worker-w1-5c3374/premium-w1-5c3374-kserve-6c56d4dcd-llwcl" Oct 30 23:54:48.691036 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:54:48.690850 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc813f02-e077-4251-8c1e-fea0986b0f36-dshm\") pod \"premium-w1-5c3374-kserve-6c56d4dcd-llwcl\" (UID: \"dc813f02-e077-4251-8c1e-fea0986b0f36\") " pod="e2e-models-e2e-worker-w1-5c3374/premium-w1-5c3374-kserve-6c56d4dcd-llwcl" Oct 30 23:54:48.691036 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:54:48.690888 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc813f02-e077-4251-8c1e-fea0986b0f36-model-cache\") pod \"premium-w1-5c3374-kserve-6c56d4dcd-llwcl\" (UID: \"dc813f02-e077-4251-8c1e-fea0986b0f36\") " pod="e2e-models-e2e-worker-w1-5c3374/premium-w1-5c3374-kserve-6c56d4dcd-llwcl" Oct 30 23:54:48.691036 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:54:48.690908 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dc813f02-e077-4251-8c1e-fea0986b0f36-home\") pod \"premium-w1-5c3374-kserve-6c56d4dcd-llwcl\" (UID: \"dc813f02-e077-4251-8c1e-fea0986b0f36\") " pod="e2e-models-e2e-worker-w1-5c3374/premium-w1-5c3374-kserve-6c56d4dcd-llwcl" Oct 30 23:54:48.691036 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:54:48.690948 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hkhrl\" (UniqueName: \"kubernetes.io/projected/dc813f02-e077-4251-8c1e-fea0986b0f36-kube-api-access-hkhrl\") pod \"premium-w1-5c3374-kserve-6c56d4dcd-llwcl\" (UID: \"dc813f02-e077-4251-8c1e-fea0986b0f36\") " pod="e2e-models-e2e-worker-w1-5c3374/premium-w1-5c3374-kserve-6c56d4dcd-llwcl" Oct 30 23:54:48.691036 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:54:48.690968 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc813f02-e077-4251-8c1e-fea0986b0f36-tls-certs\") pod \"premium-w1-5c3374-kserve-6c56d4dcd-llwcl\" (UID: \"dc813f02-e077-4251-8c1e-fea0986b0f36\") " pod="e2e-models-e2e-worker-w1-5c3374/premium-w1-5c3374-kserve-6c56d4dcd-llwcl" Oct 30 23:54:48.691313 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:54:48.691293 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc813f02-e077-4251-8c1e-fea0986b0f36-tmp-dir\") pod \"premium-w1-5c3374-kserve-6c56d4dcd-llwcl\" (UID: \"dc813f02-e077-4251-8c1e-fea0986b0f36\") " pod="e2e-models-e2e-worker-w1-5c3374/premium-w1-5c3374-kserve-6c56d4dcd-llwcl" Oct 30 23:54:48.691418 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:54:48.691354 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dc813f02-e077-4251-8c1e-fea0986b0f36-home\") pod \"premium-w1-5c3374-kserve-6c56d4dcd-llwcl\" (UID: \"dc813f02-e077-4251-8c1e-fea0986b0f36\") " pod="e2e-models-e2e-worker-w1-5c3374/premium-w1-5c3374-kserve-6c56d4dcd-llwcl" Oct 30 23:54:48.691462 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:54:48.691440 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc813f02-e077-4251-8c1e-fea0986b0f36-model-cache\") pod \"premium-w1-5c3374-kserve-6c56d4dcd-llwcl\" (UID: \"dc813f02-e077-4251-8c1e-fea0986b0f36\") " pod="e2e-models-e2e-worker-w1-5c3374/premium-w1-5c3374-kserve-6c56d4dcd-llwcl" Oct 30 23:54:48.693405 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:54:48.693378 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc813f02-e077-4251-8c1e-fea0986b0f36-dshm\") pod \"premium-w1-5c3374-kserve-6c56d4dcd-llwcl\" (UID: \"dc813f02-e077-4251-8c1e-fea0986b0f36\") " pod="e2e-models-e2e-worker-w1-5c3374/premium-w1-5c3374-kserve-6c56d4dcd-llwcl" Oct 30 23:54:48.693502 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:54:48.693455 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc813f02-e077-4251-8c1e-fea0986b0f36-tls-certs\") pod \"premium-w1-5c3374-kserve-6c56d4dcd-llwcl\" (UID: \"dc813f02-e077-4251-8c1e-fea0986b0f36\") " pod="e2e-models-e2e-worker-w1-5c3374/premium-w1-5c3374-kserve-6c56d4dcd-llwcl" Oct 30 23:54:48.753280 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:54:48.753248 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hkhrl\" (UniqueName: \"kubernetes.io/projected/dc813f02-e077-4251-8c1e-fea0986b0f36-kube-api-access-hkhrl\") pod \"premium-w1-5c3374-kserve-6c56d4dcd-llwcl\" (UID: \"dc813f02-e077-4251-8c1e-fea0986b0f36\") " pod="e2e-models-e2e-worker-w1-5c3374/premium-w1-5c3374-kserve-6c56d4dcd-llwcl" Oct 30 23:54:48.823948 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:54:48.823866 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-5c3374/premium-w1-5c3374-kserve-6c56d4dcd-llwcl" Oct 30 23:54:49.205941 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:54:49.205915 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-5c3374/premium-w1-5c3374-kserve-6c56d4dcd-llwcl"] Oct 30 23:54:49.207346 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:54:49.207302 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddc813f02_e077_4251_8c1e_fea0986b0f36.slice/crio-daf405cfe626671f3e2381c112a39a75147c9cd35d30e06e7dd450ceab0f09ce WatchSource:0}: Error finding container daf405cfe626671f3e2381c112a39a75147c9cd35d30e06e7dd450ceab0f09ce: Status 404 returned error can't find the container with id daf405cfe626671f3e2381c112a39a75147c9cd35d30e06e7dd450ceab0f09ce Oct 30 23:54:49.677886 ip-10-0-141-180 kubenswrapper[4927]: E1030 23:54:49.677850 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sqv69" containerName="extractor" Oct 30 23:54:50.200802 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:54:50.200767 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-5c3374/premium-w1-5c3374-kserve-6c56d4dcd-llwcl" event={"ID":"dc813f02-e077-4251-8c1e-fea0986b0f36","Type":"ContainerStarted","Data":"66afb0750f24146730758a0ff9eb3d95a460a6962908d6133b63e905a86b82eb"} Oct 30 23:54:50.200980 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:54:50.200817 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-5c3374/premium-w1-5c3374-kserve-6c56d4dcd-llwcl" event={"ID":"dc813f02-e077-4251-8c1e-fea0986b0f36","Type":"ContainerStarted","Data":"daf405cfe626671f3e2381c112a39a75147c9cd35d30e06e7dd450ceab0f09ce"} Oct 30 23:54:50.321672 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:54:50.321619 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-5c3374/premium-w1-5c3374-kserve-6c56d4dcd-llwcl" podStartSLOduration=2.321601248 podStartE2EDuration="2.321601248s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:54:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:54:50.319906839 +0000 UTC m=+803.356237664" watchObservedRunningTime="2026-10-30 23:54:50.321601248 +0000 UTC m=+803.357932080" Oct 30 23:54:50.824442 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:54:50.824409 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5c3374/premium-w1-5c3374-kserve-6c56d4dcd-llwcl" Oct 30 23:54:52.788524 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:54:52.788487 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:54:57.400830 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:54:57.400798 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:54:58.825005 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:54:58.824973 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-5c3374/premium-w1-5c3374-kserve-6c56d4dcd-llwcl" Oct 30 23:54:58.842358 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:54:58.842309 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-5c3374/premium-w1-5c3374-kserve-6c56d4dcd-llwcl" Oct 30 23:54:59.288074 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:54:59.288040 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5c3374/premium-w1-5c3374-kserve-6c56d4dcd-llwcl" Oct 30 23:55:02.091775 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:55:02.091736 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:55:02.444812 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:55:02.444780 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:55:02.648414 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:55:02.648378 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:55:06.693238 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:55:06.693203 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:55:09.134864 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:55:09.134827 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:55:11.518830 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:55:11.518796 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:55:16.367707 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:55:16.367670 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:55:19.668913 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:55:19.668880 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:55:19.682167 ip-10-0-141-180 kubenswrapper[4927]: E1030 23:55:19.682133 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sqv69" containerName="extractor" Oct 30 23:55:21.108368 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:55:21.108321 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:55:21.712512 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:55:21.712474 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:55:25.773367 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:55:25.773320 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:55:30.453991 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:55:30.451741 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:55:31.116484 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:55:31.116447 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:55:32.631269 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:55:32.631232 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-986d37/premium-w3-986d37-kserve-5c7c7757fb-r99j7"] Oct 30 23:55:32.636317 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:55:32.636297 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-986d37/premium-w3-986d37-kserve-5c7c7757fb-r99j7" Oct 30 23:55:32.697382 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:55:32.697324 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-986d37/premium-w3-986d37-kserve-5c7c7757fb-r99j7"] Oct 30 23:55:32.744460 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:55:32.744426 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-986d37\"/\"premium-w3-986d37-kserve-self-signed-certs\"" Oct 30 23:55:32.744460 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:55:32.744450 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-986d37\"/\"kube-root-ca.crt\"" Oct 30 23:55:32.744614 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:55:32.744427 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-986d37\"/\"default-dockercfg-ns2t6\"" Oct 30 23:55:32.744614 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:55:32.744470 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-986d37\"/\"openshift-service-ca.crt\"" Oct 30 23:55:32.786774 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:55:32.786485 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:55:32.832688 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:55:32.832649 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4e38df7-355f-4b47-a0b5-7e0b907f3326-tmp-dir\") pod \"premium-w3-986d37-kserve-5c7c7757fb-r99j7\" (UID: \"f4e38df7-355f-4b47-a0b5-7e0b907f3326\") " pod="e2e-models-e2e-worker-w3-986d37/premium-w3-986d37-kserve-5c7c7757fb-r99j7" Oct 30 23:55:32.832891 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:55:32.832789 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nfpgq\" (UniqueName: \"kubernetes.io/projected/f4e38df7-355f-4b47-a0b5-7e0b907f3326-kube-api-access-nfpgq\") pod \"premium-w3-986d37-kserve-5c7c7757fb-r99j7\" (UID: \"f4e38df7-355f-4b47-a0b5-7e0b907f3326\") " pod="e2e-models-e2e-worker-w3-986d37/premium-w3-986d37-kserve-5c7c7757fb-r99j7" Oct 30 23:55:32.832891 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:55:32.832830 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f4e38df7-355f-4b47-a0b5-7e0b907f3326-tls-certs\") pod \"premium-w3-986d37-kserve-5c7c7757fb-r99j7\" (UID: \"f4e38df7-355f-4b47-a0b5-7e0b907f3326\") " pod="e2e-models-e2e-worker-w3-986d37/premium-w3-986d37-kserve-5c7c7757fb-r99j7" Oct 30 23:55:32.833007 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:55:32.832982 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f4e38df7-355f-4b47-a0b5-7e0b907f3326-model-cache\") pod \"premium-w3-986d37-kserve-5c7c7757fb-r99j7\" (UID: \"f4e38df7-355f-4b47-a0b5-7e0b907f3326\") " pod="e2e-models-e2e-worker-w3-986d37/premium-w3-986d37-kserve-5c7c7757fb-r99j7" Oct 30 23:55:32.833057 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:55:32.833008 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f4e38df7-355f-4b47-a0b5-7e0b907f3326-dshm\") pod \"premium-w3-986d37-kserve-5c7c7757fb-r99j7\" (UID: \"f4e38df7-355f-4b47-a0b5-7e0b907f3326\") " pod="e2e-models-e2e-worker-w3-986d37/premium-w3-986d37-kserve-5c7c7757fb-r99j7" Oct 30 23:55:32.833057 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:55:32.833042 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f4e38df7-355f-4b47-a0b5-7e0b907f3326-home\") pod \"premium-w3-986d37-kserve-5c7c7757fb-r99j7\" (UID: \"f4e38df7-355f-4b47-a0b5-7e0b907f3326\") " pod="e2e-models-e2e-worker-w3-986d37/premium-w3-986d37-kserve-5c7c7757fb-r99j7" Oct 30 23:55:32.934315 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:55:32.934271 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nfpgq\" (UniqueName: \"kubernetes.io/projected/f4e38df7-355f-4b47-a0b5-7e0b907f3326-kube-api-access-nfpgq\") pod \"premium-w3-986d37-kserve-5c7c7757fb-r99j7\" (UID: \"f4e38df7-355f-4b47-a0b5-7e0b907f3326\") " pod="e2e-models-e2e-worker-w3-986d37/premium-w3-986d37-kserve-5c7c7757fb-r99j7" Oct 30 23:55:32.934543 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:55:32.934322 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f4e38df7-355f-4b47-a0b5-7e0b907f3326-tls-certs\") pod \"premium-w3-986d37-kserve-5c7c7757fb-r99j7\" (UID: \"f4e38df7-355f-4b47-a0b5-7e0b907f3326\") " pod="e2e-models-e2e-worker-w3-986d37/premium-w3-986d37-kserve-5c7c7757fb-r99j7" Oct 30 23:55:32.934543 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:55:32.934391 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f4e38df7-355f-4b47-a0b5-7e0b907f3326-model-cache\") pod \"premium-w3-986d37-kserve-5c7c7757fb-r99j7\" (UID: \"f4e38df7-355f-4b47-a0b5-7e0b907f3326\") " pod="e2e-models-e2e-worker-w3-986d37/premium-w3-986d37-kserve-5c7c7757fb-r99j7" Oct 30 23:55:32.934543 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:55:32.934410 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f4e38df7-355f-4b47-a0b5-7e0b907f3326-dshm\") pod \"premium-w3-986d37-kserve-5c7c7757fb-r99j7\" (UID: \"f4e38df7-355f-4b47-a0b5-7e0b907f3326\") " pod="e2e-models-e2e-worker-w3-986d37/premium-w3-986d37-kserve-5c7c7757fb-r99j7" Oct 30 23:55:32.934543 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:55:32.934446 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f4e38df7-355f-4b47-a0b5-7e0b907f3326-home\") pod \"premium-w3-986d37-kserve-5c7c7757fb-r99j7\" (UID: \"f4e38df7-355f-4b47-a0b5-7e0b907f3326\") " pod="e2e-models-e2e-worker-w3-986d37/premium-w3-986d37-kserve-5c7c7757fb-r99j7" Oct 30 23:55:32.934543 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:55:32.934521 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4e38df7-355f-4b47-a0b5-7e0b907f3326-tmp-dir\") pod \"premium-w3-986d37-kserve-5c7c7757fb-r99j7\" (UID: \"f4e38df7-355f-4b47-a0b5-7e0b907f3326\") " pod="e2e-models-e2e-worker-w3-986d37/premium-w3-986d37-kserve-5c7c7757fb-r99j7" Oct 30 23:55:32.934851 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:55:32.934831 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f4e38df7-355f-4b47-a0b5-7e0b907f3326-model-cache\") pod \"premium-w3-986d37-kserve-5c7c7757fb-r99j7\" (UID: \"f4e38df7-355f-4b47-a0b5-7e0b907f3326\") " pod="e2e-models-e2e-worker-w3-986d37/premium-w3-986d37-kserve-5c7c7757fb-r99j7" Oct 30 23:55:32.934921 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:55:32.934896 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f4e38df7-355f-4b47-a0b5-7e0b907f3326-home\") pod \"premium-w3-986d37-kserve-5c7c7757fb-r99j7\" (UID: \"f4e38df7-355f-4b47-a0b5-7e0b907f3326\") " pod="e2e-models-e2e-worker-w3-986d37/premium-w3-986d37-kserve-5c7c7757fb-r99j7" Oct 30 23:55:32.934970 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:55:32.934934 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4e38df7-355f-4b47-a0b5-7e0b907f3326-tmp-dir\") pod \"premium-w3-986d37-kserve-5c7c7757fb-r99j7\" (UID: \"f4e38df7-355f-4b47-a0b5-7e0b907f3326\") " pod="e2e-models-e2e-worker-w3-986d37/premium-w3-986d37-kserve-5c7c7757fb-r99j7" Oct 30 23:55:32.936807 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:55:32.936780 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f4e38df7-355f-4b47-a0b5-7e0b907f3326-dshm\") pod \"premium-w3-986d37-kserve-5c7c7757fb-r99j7\" (UID: \"f4e38df7-355f-4b47-a0b5-7e0b907f3326\") " pod="e2e-models-e2e-worker-w3-986d37/premium-w3-986d37-kserve-5c7c7757fb-r99j7" Oct 30 23:55:32.937028 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:55:32.937008 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f4e38df7-355f-4b47-a0b5-7e0b907f3326-tls-certs\") pod \"premium-w3-986d37-kserve-5c7c7757fb-r99j7\" (UID: \"f4e38df7-355f-4b47-a0b5-7e0b907f3326\") " pod="e2e-models-e2e-worker-w3-986d37/premium-w3-986d37-kserve-5c7c7757fb-r99j7" Oct 30 23:55:33.019385 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:55:33.019293 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nfpgq\" (UniqueName: \"kubernetes.io/projected/f4e38df7-355f-4b47-a0b5-7e0b907f3326-kube-api-access-nfpgq\") pod \"premium-w3-986d37-kserve-5c7c7757fb-r99j7\" (UID: \"f4e38df7-355f-4b47-a0b5-7e0b907f3326\") " pod="e2e-models-e2e-worker-w3-986d37/premium-w3-986d37-kserve-5c7c7757fb-r99j7" Oct 30 23:55:33.280140 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:55:33.280049 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-986d37/premium-w3-986d37-kserve-5c7c7757fb-r99j7" Oct 30 23:55:33.661805 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:55:33.661780 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-986d37/premium-w3-986d37-kserve-5c7c7757fb-r99j7"] Oct 30 23:55:33.663532 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:55:33.663497 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf4e38df7_355f_4b47_a0b5_7e0b907f3326.slice/crio-9d3db928da248abff45ca14834a2c8a6ac095af7911b08dc38e46163fac18215 WatchSource:0}: Error finding container 9d3db928da248abff45ca14834a2c8a6ac095af7911b08dc38e46163fac18215: Status 404 returned error can't find the container with id 9d3db928da248abff45ca14834a2c8a6ac095af7911b08dc38e46163fac18215 Oct 30 23:55:34.510653 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:55:34.510614 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-986d37/premium-w3-986d37-kserve-5c7c7757fb-r99j7" event={"ID":"f4e38df7-355f-4b47-a0b5-7e0b907f3326","Type":"ContainerStarted","Data":"46b55086ef6e0ea80c41e66e0fe1951219feee8d5b2011565d514504d8de75e2"} Oct 30 23:55:34.510858 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:55:34.510665 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-986d37/premium-w3-986d37-kserve-5c7c7757fb-r99j7" event={"ID":"f4e38df7-355f-4b47-a0b5-7e0b907f3326","Type":"ContainerStarted","Data":"9d3db928da248abff45ca14834a2c8a6ac095af7911b08dc38e46163fac18215"} Oct 30 23:55:34.633597 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:55:34.633544 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-986d37/premium-w3-986d37-kserve-5c7c7757fb-r99j7" podStartSLOduration=2.633529996 podStartE2EDuration="2.633529996s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:55:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:55:34.630513235 +0000 UTC m=+847.666844059" watchObservedRunningTime="2026-10-30 23:55:34.633529996 +0000 UTC m=+847.669860818" Oct 30 23:55:35.011201 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:55:35.011166 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:55:35.280852 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:55:35.280763 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-986d37/premium-w3-986d37-kserve-5c7c7757fb-r99j7" Oct 30 23:55:39.679918 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:55:39.679835 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:55:41.806588 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:55:41.806552 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-7gm84"] Oct 30 23:55:41.810666 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:55:41.810643 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-7gm84" Oct 30 23:55:41.858872 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:55:41.858840 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45977fb0-7f5d-4f55-9530-112d60f269ab-istio-envoy\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-7gm84\" (UID: \"45977fb0-7f5d-4f55-9530-112d60f269ab\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-7gm84" Oct 30 23:55:41.859027 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:55:41.858885 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45977fb0-7f5d-4f55-9530-112d60f269ab-istio-podinfo\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-7gm84\" (UID: \"45977fb0-7f5d-4f55-9530-112d60f269ab\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-7gm84" Oct 30 23:55:41.859027 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:55:41.858904 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45977fb0-7f5d-4f55-9530-112d60f269ab-credential-socket\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-7gm84\" (UID: \"45977fb0-7f5d-4f55-9530-112d60f269ab\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-7gm84" Oct 30 23:55:41.859099 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:55:41.859020 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45977fb0-7f5d-4f55-9530-112d60f269ab-istiod-ca-cert\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-7gm84\" (UID: \"45977fb0-7f5d-4f55-9530-112d60f269ab\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-7gm84" Oct 30 23:55:41.859099 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:55:41.859065 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45977fb0-7f5d-4f55-9530-112d60f269ab-istio-token\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-7gm84\" (UID: \"45977fb0-7f5d-4f55-9530-112d60f269ab\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-7gm84" Oct 30 23:55:41.859163 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:55:41.859133 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45977fb0-7f5d-4f55-9530-112d60f269ab-istio-data\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-7gm84\" (UID: \"45977fb0-7f5d-4f55-9530-112d60f269ab\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-7gm84" Oct 30 23:55:41.859197 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:55:41.859166 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45977fb0-7f5d-4f55-9530-112d60f269ab-workload-socket\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-7gm84\" (UID: \"45977fb0-7f5d-4f55-9530-112d60f269ab\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-7gm84" Oct 30 23:55:41.859197 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:55:41.859190 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w56pf\" (UniqueName: \"kubernetes.io/projected/45977fb0-7f5d-4f55-9530-112d60f269ab-kube-api-access-w56pf\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-7gm84\" (UID: \"45977fb0-7f5d-4f55-9530-112d60f269ab\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-7gm84" Oct 30 23:55:41.859259 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:55:41.859240 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45977fb0-7f5d-4f55-9530-112d60f269ab-workload-certs\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-7gm84\" (UID: \"45977fb0-7f5d-4f55-9530-112d60f269ab\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-7gm84" Oct 30 23:55:41.865950 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:55:41.865930 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-27e82b-openshift-default-dockercfg-2pkcf\"" Oct 30 23:55:41.921686 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:55:41.921650 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-7gm84"] Oct 30 23:55:41.960592 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:55:41.960555 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45977fb0-7f5d-4f55-9530-112d60f269ab-istio-envoy\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-7gm84\" (UID: \"45977fb0-7f5d-4f55-9530-112d60f269ab\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-7gm84" Oct 30 23:55:41.960787 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:55:41.960603 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45977fb0-7f5d-4f55-9530-112d60f269ab-istio-podinfo\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-7gm84\" (UID: \"45977fb0-7f5d-4f55-9530-112d60f269ab\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-7gm84" Oct 30 23:55:41.960787 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:55:41.960622 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45977fb0-7f5d-4f55-9530-112d60f269ab-credential-socket\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-7gm84\" (UID: \"45977fb0-7f5d-4f55-9530-112d60f269ab\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-7gm84" Oct 30 23:55:41.960787 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:55:41.960751 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45977fb0-7f5d-4f55-9530-112d60f269ab-istiod-ca-cert\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-7gm84\" (UID: \"45977fb0-7f5d-4f55-9530-112d60f269ab\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-7gm84" Oct 30 23:55:41.961040 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:55:41.961015 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45977fb0-7f5d-4f55-9530-112d60f269ab-istio-token\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-7gm84\" (UID: \"45977fb0-7f5d-4f55-9530-112d60f269ab\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-7gm84" Oct 30 23:55:41.961271 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:55:41.961246 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45977fb0-7f5d-4f55-9530-112d60f269ab-istio-data\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-7gm84\" (UID: \"45977fb0-7f5d-4f55-9530-112d60f269ab\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-7gm84" Oct 30 23:55:41.961383 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:55:41.961115 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45977fb0-7f5d-4f55-9530-112d60f269ab-credential-socket\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-7gm84\" (UID: \"45977fb0-7f5d-4f55-9530-112d60f269ab\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-7gm84" Oct 30 23:55:41.961383 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:55:41.961306 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45977fb0-7f5d-4f55-9530-112d60f269ab-workload-socket\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-7gm84\" (UID: \"45977fb0-7f5d-4f55-9530-112d60f269ab\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-7gm84" Oct 30 23:55:41.961383 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:55:41.961354 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w56pf\" (UniqueName: \"kubernetes.io/projected/45977fb0-7f5d-4f55-9530-112d60f269ab-kube-api-access-w56pf\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-7gm84\" (UID: \"45977fb0-7f5d-4f55-9530-112d60f269ab\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-7gm84" Oct 30 23:55:41.961561 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:55:41.961386 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45977fb0-7f5d-4f55-9530-112d60f269ab-workload-certs\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-7gm84\" (UID: \"45977fb0-7f5d-4f55-9530-112d60f269ab\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-7gm84" Oct 30 23:55:41.961808 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:55:41.961779 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45977fb0-7f5d-4f55-9530-112d60f269ab-istiod-ca-cert\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-7gm84\" (UID: \"45977fb0-7f5d-4f55-9530-112d60f269ab\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-7gm84" Oct 30 23:55:41.961892 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:55:41.961805 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45977fb0-7f5d-4f55-9530-112d60f269ab-istio-data\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-7gm84\" (UID: \"45977fb0-7f5d-4f55-9530-112d60f269ab\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-7gm84" Oct 30 23:55:41.961985 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:55:41.961959 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45977fb0-7f5d-4f55-9530-112d60f269ab-workload-certs\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-7gm84\" (UID: \"45977fb0-7f5d-4f55-9530-112d60f269ab\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-7gm84" Oct 30 23:55:41.962025 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:55:41.961980 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45977fb0-7f5d-4f55-9530-112d60f269ab-workload-socket\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-7gm84\" (UID: \"45977fb0-7f5d-4f55-9530-112d60f269ab\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-7gm84" Oct 30 23:55:41.963074 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:55:41.963052 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45977fb0-7f5d-4f55-9530-112d60f269ab-istio-envoy\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-7gm84\" (UID: \"45977fb0-7f5d-4f55-9530-112d60f269ab\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-7gm84" Oct 30 23:55:41.963226 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:55:41.963207 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45977fb0-7f5d-4f55-9530-112d60f269ab-istio-podinfo\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-7gm84\" (UID: \"45977fb0-7f5d-4f55-9530-112d60f269ab\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-7gm84" Oct 30 23:55:42.025006 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:55:42.024972 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45977fb0-7f5d-4f55-9530-112d60f269ab-istio-token\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-7gm84\" (UID: \"45977fb0-7f5d-4f55-9530-112d60f269ab\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-7gm84" Oct 30 23:55:42.026298 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:55:42.026268 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w56pf\" (UniqueName: \"kubernetes.io/projected/45977fb0-7f5d-4f55-9530-112d60f269ab-kube-api-access-w56pf\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-7gm84\" (UID: \"45977fb0-7f5d-4f55-9530-112d60f269ab\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-7gm84" Oct 30 23:55:42.126626 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:55:42.126546 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-7gm84" Oct 30 23:55:42.565396 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:55:42.565370 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-7gm84"] Oct 30 23:55:42.566542 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:55:42.566516 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45977fb0_7f5d_4f55_9530_112d60f269ab.slice/crio-c8f56af937bf936720bb0359806ee16cec1b02c5fe9d9345f78b93a58d757268 WatchSource:0}: Error finding container c8f56af937bf936720bb0359806ee16cec1b02c5fe9d9345f78b93a58d757268: Status 404 returned error can't find the container with id c8f56af937bf936720bb0359806ee16cec1b02c5fe9d9345f78b93a58d757268 Oct 30 23:55:42.568710 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:55:42.568674 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 23:55:42.568792 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:55:42.568751 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 23:55:42.568792 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:55:42.568784 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 23:55:43.281059 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:55:43.281024 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-986d37/premium-w3-986d37-kserve-5c7c7757fb-r99j7" Oct 30 23:55:43.293438 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:55:43.293414 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-986d37/premium-w3-986d37-kserve-5c7c7757fb-r99j7" Oct 30 23:55:43.570394 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:55:43.570289 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-7gm84" event={"ID":"45977fb0-7f5d-4f55-9530-112d60f269ab","Type":"ContainerStarted","Data":"646c05c17da8a4f530f45bf9bb4e8348afb3fbe87e9040d1ee2cea4e92dccb62"} Oct 30 23:55:43.570394 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:55:43.570367 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-7gm84" event={"ID":"45977fb0-7f5d-4f55-9530-112d60f269ab","Type":"ContainerStarted","Data":"c8f56af937bf936720bb0359806ee16cec1b02c5fe9d9345f78b93a58d757268"} Oct 30 23:55:43.581543 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:55:43.581515 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-986d37/premium-w3-986d37-kserve-5c7c7757fb-r99j7" Oct 30 23:55:43.845641 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:55:43.845525 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-7gm84" podStartSLOduration=2.845510518 podStartE2EDuration="2.845510518s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:55:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:55:43.722628443 +0000 UTC m=+856.758959290" watchObservedRunningTime="2026-10-30 23:55:43.845510518 +0000 UTC m=+856.881841340" Oct 30 23:55:44.131951 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:55:44.131863 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-7gm84" Oct 30 23:55:44.132278 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:55:44.132252 4927 patch_prober.go:29] interesting pod/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-7gm84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Oct 30 23:55:44.132472 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:55:44.132435 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-7gm84" podUID="45977fb0-7f5d-4f55-9530-112d60f269ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Oct 30 23:55:44.398664 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:55:44.398573 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:55:45.127731 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:55:45.127695 4927 patch_prober.go:29] interesting pod/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-7gm84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Oct 30 23:55:45.128009 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:55:45.127981 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-7gm84" podUID="45977fb0-7f5d-4f55-9530-112d60f269ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Oct 30 23:55:46.127380 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:55:46.127326 4927 patch_prober.go:29] interesting pod/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-7gm84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Oct 30 23:55:46.127964 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:55:46.127932 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-7gm84" podUID="45977fb0-7f5d-4f55-9530-112d60f269ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Oct 30 23:55:46.246363 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:55:46.243055 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:55:47.128759 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:55:47.128717 4927 patch_prober.go:29] interesting pod/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-7gm84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Oct 30 23:55:47.129391 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:55:47.129354 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-7gm84" podUID="45977fb0-7f5d-4f55-9530-112d60f269ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Oct 30 23:55:48.130723 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:55:48.130679 4927 patch_prober.go:29] interesting pod/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-7gm84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Oct 30 23:55:48.131379 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:55:48.131321 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-7gm84" podUID="45977fb0-7f5d-4f55-9530-112d60f269ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Oct 30 23:55:49.004594 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:55:49.004555 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:55:49.332305 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:55:49.332220 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-7gm84" Oct 30 23:55:49.333027 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:55:49.332998 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-7gm84" Oct 30 23:55:49.333858 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:55:49.333837 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-7gm84" Oct 30 23:55:49.687508 ip-10-0-141-180 kubenswrapper[4927]: E1030 23:55:49.687465 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sqv69" containerName="extractor" Oct 30 23:55:53.633780 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:55:53.633736 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:55:54.505072 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:55:54.505036 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a414f1b5-nwttq"] Oct 30 23:55:54.509247 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:55:54.509223 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a414f1b5-nwttq" Oct 30 23:55:54.579390 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:55:54.573790 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a414f1b5-nwttq"] Oct 30 23:55:54.620097 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:55:54.620061 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 23:55:54.620097 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:55:54.620090 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4fdzs\"" Oct 30 23:55:54.628637 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:55:54.628610 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa953e7a-5616-4ec2-aea5-c554ddcc648c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a414f1b5-nwttq\" (UID: \"aa953e7a-5616-4ec2-aea5-c554ddcc648c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a414f1b5-nwttq" Oct 30 23:55:54.628726 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:55:54.628664 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa953e7a-5616-4ec2-aea5-c554ddcc648c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a414f1b5-nwttq\" (UID: \"aa953e7a-5616-4ec2-aea5-c554ddcc648c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a414f1b5-nwttq" Oct 30 23:55:54.628848 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:55:54.628826 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fzww6\" (UniqueName: \"kubernetes.io/projected/aa953e7a-5616-4ec2-aea5-c554ddcc648c-kube-api-access-fzww6\") pod \"maas-api-revoke-sub-a414f1b5-nwttq\" (UID: \"aa953e7a-5616-4ec2-aea5-c554ddcc648c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a414f1b5-nwttq" Oct 30 23:55:54.729980 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:55:54.729940 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa953e7a-5616-4ec2-aea5-c554ddcc648c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a414f1b5-nwttq\" (UID: \"aa953e7a-5616-4ec2-aea5-c554ddcc648c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a414f1b5-nwttq" Oct 30 23:55:54.730468 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:55:54.730007 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa953e7a-5616-4ec2-aea5-c554ddcc648c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a414f1b5-nwttq\" (UID: \"aa953e7a-5616-4ec2-aea5-c554ddcc648c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a414f1b5-nwttq" Oct 30 23:55:54.730468 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:55:54.730070 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fzww6\" (UniqueName: \"kubernetes.io/projected/aa953e7a-5616-4ec2-aea5-c554ddcc648c-kube-api-access-fzww6\") pod \"maas-api-revoke-sub-a414f1b5-nwttq\" (UID: \"aa953e7a-5616-4ec2-aea5-c554ddcc648c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a414f1b5-nwttq" Oct 30 23:55:54.730657 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:55:54.730636 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa953e7a-5616-4ec2-aea5-c554ddcc648c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a414f1b5-nwttq\" (UID: \"aa953e7a-5616-4ec2-aea5-c554ddcc648c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a414f1b5-nwttq" Oct 30 23:55:54.732344 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:55:54.732308 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa953e7a-5616-4ec2-aea5-c554ddcc648c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a414f1b5-nwttq\" (UID: \"aa953e7a-5616-4ec2-aea5-c554ddcc648c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a414f1b5-nwttq" Oct 30 23:55:54.793722 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:55:54.793651 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fzww6\" (UniqueName: \"kubernetes.io/projected/aa953e7a-5616-4ec2-aea5-c554ddcc648c-kube-api-access-fzww6\") pod \"maas-api-revoke-sub-a414f1b5-nwttq\" (UID: \"aa953e7a-5616-4ec2-aea5-c554ddcc648c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a414f1b5-nwttq" Oct 30 23:55:54.825420 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:55:54.825394 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a414f1b5-nwttq" Oct 30 23:55:55.207545 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:55:55.207513 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa953e7a_5616_4ec2_aea5_c554ddcc648c.slice/crio-71975ed6577be418cad8c7fe7b955dc54a1b1e9c85b7281dba3392a1b2ce3f78 WatchSource:0}: Error finding container 71975ed6577be418cad8c7fe7b955dc54a1b1e9c85b7281dba3392a1b2ce3f78: Status 404 returned error can't find the container with id 71975ed6577be418cad8c7fe7b955dc54a1b1e9c85b7281dba3392a1b2ce3f78 Oct 30 23:55:55.207958 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:55:55.207933 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a414f1b5-nwttq"] Oct 30 23:55:55.209795 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:55:55.209778 4927 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 23:55:55.656444 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:55:55.656360 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a414f1b5-nwttq" event={"ID":"aa953e7a-5616-4ec2-aea5-c554ddcc648c","Type":"ContainerStarted","Data":"71975ed6577be418cad8c7fe7b955dc54a1b1e9c85b7281dba3392a1b2ce3f78"} Oct 30 23:55:58.305061 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:55:58.304958 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:56:00.169347 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:00.169303 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-pnm97"] Oct 30 23:56:00.169771 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:00.169578 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-pnm97" podUID="495da7b4-1e63-46fa-ac83-6d3bb75692f8" containerName="istio-proxy" containerID="cri-o://0ac36170caeaed36c8f33fe01cbef7a150c4d67109850ec0c78261b4d55968cf" gracePeriod=30 Oct 30 23:56:00.721283 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:00.721238 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a414f1b5-nwttq" event={"ID":"aa953e7a-5616-4ec2-aea5-c554ddcc648c","Type":"ContainerStarted","Data":"4e9c7b64f8c0d9d6856d9c2c56b270022e186a136fc71417eedb574253b51145"} Oct 30 23:56:00.846748 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:00.846692 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a414f1b5-nwttq" podStartSLOduration=2.200356779 podStartE2EDuration="6.846679516s" totalImagesPullingTime="4.646322737s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:55:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:56:00.843321328 +0000 UTC m=+873.879652151" watchObservedRunningTime="2026-10-30 23:56:00.846679516 +0000 UTC m=+873.883010337" Oct 30 23:56:01.730623 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:01.730584 4927 generic.go:356] "Generic (PLEG): container finished" podID="aa953e7a-5616-4ec2-aea5-c554ddcc648c" containerID="4e9c7b64f8c0d9d6856d9c2c56b270022e186a136fc71417eedb574253b51145" exitCode=0 Oct 30 23:56:01.731038 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:01.730675 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a414f1b5-nwttq" event={"ID":"aa953e7a-5616-4ec2-aea5-c554ddcc648c","Type":"ContainerDied","Data":"4e9c7b64f8c0d9d6856d9c2c56b270022e186a136fc71417eedb574253b51145"} Oct 30 23:56:02.882089 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:02.882061 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a414f1b5-nwttq" Oct 30 23:56:02.974771 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:02.974735 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa953e7a-5616-4ec2-aea5-c554ddcc648c-kube-api-access-fzww6\" (UniqueName: \"kubernetes.io/projected/aa953e7a-5616-4ec2-aea5-c554ddcc648c-kube-api-access-fzww6\") pod \"aa953e7a-5616-4ec2-aea5-c554ddcc648c\" (UID: \"aa953e7a-5616-4ec2-aea5-c554ddcc648c\") " Oct 30 23:56:02.974946 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:02.974781 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aa953e7a-5616-4ec2-aea5-c554ddcc648c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa953e7a-5616-4ec2-aea5-c554ddcc648c-maas-api-xks-ca\") pod \"aa953e7a-5616-4ec2-aea5-c554ddcc648c\" (UID: \"aa953e7a-5616-4ec2-aea5-c554ddcc648c\") " Oct 30 23:56:02.974946 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:02.974912 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa953e7a-5616-4ec2-aea5-c554ddcc648c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa953e7a-5616-4ec2-aea5-c554ddcc648c-maas-api-openshift-ca\") pod \"aa953e7a-5616-4ec2-aea5-c554ddcc648c\" (UID: \"aa953e7a-5616-4ec2-aea5-c554ddcc648c\") " Oct 30 23:56:02.975367 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:02.975315 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa953e7a-5616-4ec2-aea5-c554ddcc648c-maas-api-openshift-ca" pod "aa953e7a-5616-4ec2-aea5-c554ddcc648c" (UID: "aa953e7a-5616-4ec2-aea5-c554ddcc648c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:56:02.976954 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:02.976934 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa953e7a-5616-4ec2-aea5-c554ddcc648c-kube-api-access-fzww6" pod "aa953e7a-5616-4ec2-aea5-c554ddcc648c" (UID: "aa953e7a-5616-4ec2-aea5-c554ddcc648c"). InnerVolumeSpecName "kube-api-access-fzww6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:56:02.977123 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:02.977100 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa953e7a-5616-4ec2-aea5-c554ddcc648c-maas-api-xks-ca" pod "aa953e7a-5616-4ec2-aea5-c554ddcc648c" (UID: "aa953e7a-5616-4ec2-aea5-c554ddcc648c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 23:56:03.076159 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:03.076078 4927 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa953e7a-5616-4ec2-aea5-c554ddcc648c-maas-api-openshift-ca\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 23:56:03.076159 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:03.076109 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fzww6\" (UniqueName: \"kubernetes.io/projected/aa953e7a-5616-4ec2-aea5-c554ddcc648c-kube-api-access-fzww6\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 23:56:03.076159 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:03.076123 4927 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa953e7a-5616-4ec2-aea5-c554ddcc648c-maas-api-xks-ca\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 23:56:03.151259 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:03.151220 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:56:03.746405 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:03.746371 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a414f1b5-nwttq" Oct 30 23:56:03.746571 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:03.746373 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a414f1b5-nwttq" event={"ID":"aa953e7a-5616-4ec2-aea5-c554ddcc648c","Type":"ContainerDied","Data":"71975ed6577be418cad8c7fe7b955dc54a1b1e9c85b7281dba3392a1b2ce3f78"} Oct 30 23:56:03.746571 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:03.746509 4927 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="71975ed6577be418cad8c7fe7b955dc54a1b1e9c85b7281dba3392a1b2ce3f78" Oct 30 23:56:04.482047 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:04.482009 4927 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-pnm97 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 23:56:04.482447 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:04.482074 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-pnm97" podUID="495da7b4-1e63-46fa-ac83-6d3bb75692f8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 23:56:05.444737 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:05.444706 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-pnm97" Oct 30 23:56:05.501080 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:05.501002 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/495da7b4-1e63-46fa-ac83-6d3bb75692f8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/495da7b4-1e63-46fa-ac83-6d3bb75692f8-istio-envoy\") pod \"495da7b4-1e63-46fa-ac83-6d3bb75692f8\" (UID: \"495da7b4-1e63-46fa-ac83-6d3bb75692f8\") " Oct 30 23:56:05.501525 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:05.501116 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/495da7b4-1e63-46fa-ac83-6d3bb75692f8-kube-api-access-hhhh8\" (UniqueName: \"kubernetes.io/projected/495da7b4-1e63-46fa-ac83-6d3bb75692f8-kube-api-access-hhhh8\") pod \"495da7b4-1e63-46fa-ac83-6d3bb75692f8\" (UID: \"495da7b4-1e63-46fa-ac83-6d3bb75692f8\") " Oct 30 23:56:05.501525 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:05.501140 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/495da7b4-1e63-46fa-ac83-6d3bb75692f8-istio-token\" (UniqueName: \"kubernetes.io/projected/495da7b4-1e63-46fa-ac83-6d3bb75692f8-istio-token\") pod \"495da7b4-1e63-46fa-ac83-6d3bb75692f8\" (UID: \"495da7b4-1e63-46fa-ac83-6d3bb75692f8\") " Oct 30 23:56:05.501525 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:05.501182 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/495da7b4-1e63-46fa-ac83-6d3bb75692f8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/495da7b4-1e63-46fa-ac83-6d3bb75692f8-credential-socket\") pod \"495da7b4-1e63-46fa-ac83-6d3bb75692f8\" (UID: \"495da7b4-1e63-46fa-ac83-6d3bb75692f8\") " Oct 30 23:56:05.501525 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:05.501212 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/495da7b4-1e63-46fa-ac83-6d3bb75692f8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/495da7b4-1e63-46fa-ac83-6d3bb75692f8-workload-certs\") pod \"495da7b4-1e63-46fa-ac83-6d3bb75692f8\" (UID: \"495da7b4-1e63-46fa-ac83-6d3bb75692f8\") " Oct 30 23:56:05.501525 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:05.501259 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/495da7b4-1e63-46fa-ac83-6d3bb75692f8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/495da7b4-1e63-46fa-ac83-6d3bb75692f8-istiod-ca-cert\") pod \"495da7b4-1e63-46fa-ac83-6d3bb75692f8\" (UID: \"495da7b4-1e63-46fa-ac83-6d3bb75692f8\") " Oct 30 23:56:05.501525 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:05.501305 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/495da7b4-1e63-46fa-ac83-6d3bb75692f8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/495da7b4-1e63-46fa-ac83-6d3bb75692f8-istio-data\") pod \"495da7b4-1e63-46fa-ac83-6d3bb75692f8\" (UID: \"495da7b4-1e63-46fa-ac83-6d3bb75692f8\") " Oct 30 23:56:05.501525 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:05.501392 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/495da7b4-1e63-46fa-ac83-6d3bb75692f8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/495da7b4-1e63-46fa-ac83-6d3bb75692f8-workload-socket\") pod \"495da7b4-1e63-46fa-ac83-6d3bb75692f8\" (UID: \"495da7b4-1e63-46fa-ac83-6d3bb75692f8\") " Oct 30 23:56:05.501525 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:05.501450 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/495da7b4-1e63-46fa-ac83-6d3bb75692f8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/495da7b4-1e63-46fa-ac83-6d3bb75692f8-istio-podinfo\") pod \"495da7b4-1e63-46fa-ac83-6d3bb75692f8\" (UID: \"495da7b4-1e63-46fa-ac83-6d3bb75692f8\") " Oct 30 23:56:05.501937 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:05.501497 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/495da7b4-1e63-46fa-ac83-6d3bb75692f8-workload-certs" pod "495da7b4-1e63-46fa-ac83-6d3bb75692f8" (UID: "495da7b4-1e63-46fa-ac83-6d3bb75692f8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:56:05.501937 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:05.501544 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/495da7b4-1e63-46fa-ac83-6d3bb75692f8-credential-socket" pod "495da7b4-1e63-46fa-ac83-6d3bb75692f8" (UID: "495da7b4-1e63-46fa-ac83-6d3bb75692f8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:56:05.501937 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:05.501742 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/495da7b4-1e63-46fa-ac83-6d3bb75692f8-workload-socket" pod "495da7b4-1e63-46fa-ac83-6d3bb75692f8" (UID: "495da7b4-1e63-46fa-ac83-6d3bb75692f8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:56:05.501937 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:05.501767 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/495da7b4-1e63-46fa-ac83-6d3bb75692f8-istiod-ca-cert" pod "495da7b4-1e63-46fa-ac83-6d3bb75692f8" (UID: "495da7b4-1e63-46fa-ac83-6d3bb75692f8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:56:05.501937 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:05.501796 4927 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/495da7b4-1e63-46fa-ac83-6d3bb75692f8-credential-socket\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 23:56:05.501937 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:05.501819 4927 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/495da7b4-1e63-46fa-ac83-6d3bb75692f8-workload-certs\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 23:56:05.501937 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:05.501917 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/495da7b4-1e63-46fa-ac83-6d3bb75692f8-istio-data" pod "495da7b4-1e63-46fa-ac83-6d3bb75692f8" (UID: "495da7b4-1e63-46fa-ac83-6d3bb75692f8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:56:05.503527 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:05.503505 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/495da7b4-1e63-46fa-ac83-6d3bb75692f8-kube-api-access-hhhh8" pod "495da7b4-1e63-46fa-ac83-6d3bb75692f8" (UID: "495da7b4-1e63-46fa-ac83-6d3bb75692f8"). InnerVolumeSpecName "kube-api-access-hhhh8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:56:05.503636 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:05.503552 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/495da7b4-1e63-46fa-ac83-6d3bb75692f8-istio-podinfo" pod "495da7b4-1e63-46fa-ac83-6d3bb75692f8" (UID: "495da7b4-1e63-46fa-ac83-6d3bb75692f8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 23:56:05.503686 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:05.503634 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/495da7b4-1e63-46fa-ac83-6d3bb75692f8-istio-envoy" pod "495da7b4-1e63-46fa-ac83-6d3bb75692f8" (UID: "495da7b4-1e63-46fa-ac83-6d3bb75692f8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:56:05.503725 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:05.503704 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/495da7b4-1e63-46fa-ac83-6d3bb75692f8-istio-token" pod "495da7b4-1e63-46fa-ac83-6d3bb75692f8" (UID: "495da7b4-1e63-46fa-ac83-6d3bb75692f8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:56:05.602706 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:05.602677 4927 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/495da7b4-1e63-46fa-ac83-6d3bb75692f8-istiod-ca-cert\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 23:56:05.602706 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:05.602703 4927 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/495da7b4-1e63-46fa-ac83-6d3bb75692f8-istio-data\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 23:56:05.602706 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:05.602713 4927 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/495da7b4-1e63-46fa-ac83-6d3bb75692f8-workload-socket\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 23:56:05.602940 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:05.602722 4927 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/495da7b4-1e63-46fa-ac83-6d3bb75692f8-istio-podinfo\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 23:56:05.602940 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:05.602730 4927 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/495da7b4-1e63-46fa-ac83-6d3bb75692f8-istio-envoy\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 23:56:05.602940 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:05.602739 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hhhh8\" (UniqueName: \"kubernetes.io/projected/495da7b4-1e63-46fa-ac83-6d3bb75692f8-kube-api-access-hhhh8\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 23:56:05.602940 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:05.602748 4927 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/495da7b4-1e63-46fa-ac83-6d3bb75692f8-istio-token\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 23:56:05.761934 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:05.761841 4927 generic.go:356] "Generic (PLEG): container finished" podID="495da7b4-1e63-46fa-ac83-6d3bb75692f8" containerID="0ac36170caeaed36c8f33fe01cbef7a150c4d67109850ec0c78261b4d55968cf" exitCode=0 Oct 30 23:56:05.762107 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:05.761961 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-pnm97" Oct 30 23:56:05.762107 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:05.761978 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-pnm97" event={"ID":"495da7b4-1e63-46fa-ac83-6d3bb75692f8","Type":"ContainerDied","Data":"0ac36170caeaed36c8f33fe01cbef7a150c4d67109850ec0c78261b4d55968cf"} Oct 30 23:56:05.762107 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:05.762038 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-pnm97" event={"ID":"495da7b4-1e63-46fa-ac83-6d3bb75692f8","Type":"ContainerDied","Data":"633dc6e2b09285bc2bc9eceedccedacb5d5df5e2a3ada3112e76e39670b63ad9"} Oct 30 23:56:05.762107 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:05.762061 4927 scope.go:122] "RemoveContainer" containerID="0ac36170caeaed36c8f33fe01cbef7a150c4d67109850ec0c78261b4d55968cf" Oct 30 23:56:05.776825 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:05.775033 4927 scope.go:122] "RemoveContainer" containerID="3c0baafe8176b8985b703751e8d2118f0442fe67f81363c6f1c35c195218077d" Oct 30 23:56:05.785136 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:05.785114 4927 scope.go:122] "RemoveContainer" containerID="0ac36170caeaed36c8f33fe01cbef7a150c4d67109850ec0c78261b4d55968cf" Oct 30 23:56:05.785622 ip-10-0-141-180 kubenswrapper[4927]: E1030 23:56:05.785584 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ac36170caeaed36c8f33fe01cbef7a150c4d67109850ec0c78261b4d55968cf\": container with ID starting with 0ac36170caeaed36c8f33fe01cbef7a150c4d67109850ec0c78261b4d55968cf not found: ID does not exist" containerID="0ac36170caeaed36c8f33fe01cbef7a150c4d67109850ec0c78261b4d55968cf" Oct 30 23:56:05.785799 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:05.785636 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ac36170caeaed36c8f33fe01cbef7a150c4d67109850ec0c78261b4d55968cf"} err="failed to get container status \"0ac36170caeaed36c8f33fe01cbef7a150c4d67109850ec0c78261b4d55968cf\": rpc error: code = NotFound desc = could not find container \"0ac36170caeaed36c8f33fe01cbef7a150c4d67109850ec0c78261b4d55968cf\": container with ID starting with 0ac36170caeaed36c8f33fe01cbef7a150c4d67109850ec0c78261b4d55968cf not found: ID does not exist" Oct 30 23:56:05.785799 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:05.785664 4927 scope.go:122] "RemoveContainer" containerID="3c0baafe8176b8985b703751e8d2118f0442fe67f81363c6f1c35c195218077d" Oct 30 23:56:05.786039 ip-10-0-141-180 kubenswrapper[4927]: E1030 23:56:05.786008 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c0baafe8176b8985b703751e8d2118f0442fe67f81363c6f1c35c195218077d\": container with ID starting with 3c0baafe8176b8985b703751e8d2118f0442fe67f81363c6f1c35c195218077d not found: ID does not exist" containerID="3c0baafe8176b8985b703751e8d2118f0442fe67f81363c6f1c35c195218077d" Oct 30 23:56:05.786106 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:05.786050 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c0baafe8176b8985b703751e8d2118f0442fe67f81363c6f1c35c195218077d"} err="failed to get container status \"3c0baafe8176b8985b703751e8d2118f0442fe67f81363c6f1c35c195218077d\": rpc error: code = NotFound desc = could not find container \"3c0baafe8176b8985b703751e8d2118f0442fe67f81363c6f1c35c195218077d\": container with ID starting with 3c0baafe8176b8985b703751e8d2118f0442fe67f81363c6f1c35c195218077d not found: ID does not exist" Oct 30 23:56:05.956054 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:05.955999 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-pnm97"] Oct 30 23:56:05.963005 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:05.962954 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-pnm97"] Oct 30 23:56:06.525681 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:06.525646 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b0a3b296-6qrvt"] Oct 30 23:56:06.526170 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:06.526156 4927 cpu_manager.go:414] "containerMap: removing container" podUID="495da7b4-1e63-46fa-ac83-6d3bb75692f8" containerName="istio-proxy" Oct 30 23:56:06.526219 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:06.526171 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="495da7b4-1e63-46fa-ac83-6d3bb75692f8" containerName="istio-proxy" Oct 30 23:56:06.526254 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:06.526217 4927 cpu_manager.go:414] "containerMap: removing container" podUID="aa953e7a-5616-4ec2-aea5-c554ddcc648c" containerName="revoke-keys" Oct 30 23:56:06.526254 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:06.526225 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa953e7a-5616-4ec2-aea5-c554ddcc648c" containerName="revoke-keys" Oct 30 23:56:06.526317 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:06.526310 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="495da7b4-1e63-46fa-ac83-6d3bb75692f8" containerName="istio-proxy" Oct 30 23:56:06.526373 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:06.526319 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa953e7a-5616-4ec2-aea5-c554ddcc648c" containerName="revoke-keys" Oct 30 23:56:06.526373 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:06.526340 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="495da7b4-1e63-46fa-ac83-6d3bb75692f8" containerName="istio-proxy" Oct 30 23:56:06.531087 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:06.531069 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0a3b296-6qrvt" Oct 30 23:56:06.586391 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:06.586354 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 23:56:06.590007 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:06.589973 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b0a3b296-6qrvt"] Oct 30 23:56:06.612265 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:06.612236 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dcqt2\" (UniqueName: \"kubernetes.io/projected/0a2ebfe5-08c3-4d7e-a32c-e5ee5ef4673e-kube-api-access-dcqt2\") pod \"maas-api-revoke-sub-b0a3b296-6qrvt\" (UID: \"0a2ebfe5-08c3-4d7e-a32c-e5ee5ef4673e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0a3b296-6qrvt" Oct 30 23:56:06.612423 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:06.612274 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a2ebfe5-08c3-4d7e-a32c-e5ee5ef4673e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b0a3b296-6qrvt\" (UID: \"0a2ebfe5-08c3-4d7e-a32c-e5ee5ef4673e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0a3b296-6qrvt" Oct 30 23:56:06.612464 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:06.612421 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a2ebfe5-08c3-4d7e-a32c-e5ee5ef4673e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b0a3b296-6qrvt\" (UID: \"0a2ebfe5-08c3-4d7e-a32c-e5ee5ef4673e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0a3b296-6qrvt" Oct 30 23:56:06.635858 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:06.635832 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4fdzs\"" Oct 30 23:56:06.713822 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:06.713780 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a2ebfe5-08c3-4d7e-a32c-e5ee5ef4673e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b0a3b296-6qrvt\" (UID: \"0a2ebfe5-08c3-4d7e-a32c-e5ee5ef4673e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0a3b296-6qrvt" Oct 30 23:56:06.713962 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:06.713882 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dcqt2\" (UniqueName: \"kubernetes.io/projected/0a2ebfe5-08c3-4d7e-a32c-e5ee5ef4673e-kube-api-access-dcqt2\") pod \"maas-api-revoke-sub-b0a3b296-6qrvt\" (UID: \"0a2ebfe5-08c3-4d7e-a32c-e5ee5ef4673e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0a3b296-6qrvt" Oct 30 23:56:06.713962 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:06.713904 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a2ebfe5-08c3-4d7e-a32c-e5ee5ef4673e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b0a3b296-6qrvt\" (UID: \"0a2ebfe5-08c3-4d7e-a32c-e5ee5ef4673e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0a3b296-6qrvt" Oct 30 23:56:06.714583 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:06.714563 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a2ebfe5-08c3-4d7e-a32c-e5ee5ef4673e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b0a3b296-6qrvt\" (UID: \"0a2ebfe5-08c3-4d7e-a32c-e5ee5ef4673e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0a3b296-6qrvt" Oct 30 23:56:06.716160 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:06.716140 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a2ebfe5-08c3-4d7e-a32c-e5ee5ef4673e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b0a3b296-6qrvt\" (UID: \"0a2ebfe5-08c3-4d7e-a32c-e5ee5ef4673e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0a3b296-6qrvt" Oct 30 23:56:06.777691 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:06.777617 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dcqt2\" (UniqueName: \"kubernetes.io/projected/0a2ebfe5-08c3-4d7e-a32c-e5ee5ef4673e-kube-api-access-dcqt2\") pod \"maas-api-revoke-sub-b0a3b296-6qrvt\" (UID: \"0a2ebfe5-08c3-4d7e-a32c-e5ee5ef4673e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0a3b296-6qrvt" Oct 30 23:56:06.844081 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:06.844051 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0a3b296-6qrvt" Oct 30 23:56:07.086235 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:07.086157 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="495da7b4-1e63-46fa-ac83-6d3bb75692f8" path="/var/lib/kubelet/pods/495da7b4-1e63-46fa-ac83-6d3bb75692f8/volumes" Oct 30 23:56:07.221995 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:07.221966 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b0a3b296-6qrvt"] Oct 30 23:56:07.223623 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:56:07.223597 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a2ebfe5_08c3_4d7e_a32c_e5ee5ef4673e.slice/crio-5a07db9a742a9d0d14e167c559e7852c1efbcc2f86c8b887258ca5a2414dc46f WatchSource:0}: Error finding container 5a07db9a742a9d0d14e167c559e7852c1efbcc2f86c8b887258ca5a2414dc46f: Status 404 returned error can't find the container with id 5a07db9a742a9d0d14e167c559e7852c1efbcc2f86c8b887258ca5a2414dc46f Oct 30 23:56:07.776448 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:07.776362 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0a3b296-6qrvt" event={"ID":"0a2ebfe5-08c3-4d7e-a32c-e5ee5ef4673e","Type":"ContainerStarted","Data":"21816518bcb866b47e586e81fb59288492cc740e563e318142f0cacd50859734"} Oct 30 23:56:07.776448 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:07.776415 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0a3b296-6qrvt" event={"ID":"0a2ebfe5-08c3-4d7e-a32c-e5ee5ef4673e","Type":"ContainerStarted","Data":"5a07db9a742a9d0d14e167c559e7852c1efbcc2f86c8b887258ca5a2414dc46f"} Oct 30 23:56:07.795401 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:07.795373 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:56:07.910607 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:07.910534 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0a3b296-6qrvt" podStartSLOduration=1.910515926 podStartE2EDuration="1.910515926s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:56:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:56:07.908229783 +0000 UTC m=+880.944560633" watchObservedRunningTime="2026-10-30 23:56:07.910515926 +0000 UTC m=+880.946846748" Oct 30 23:56:08.784407 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:08.784370 4927 generic.go:356] "Generic (PLEG): container finished" podID="0a2ebfe5-08c3-4d7e-a32c-e5ee5ef4673e" containerID="21816518bcb866b47e586e81fb59288492cc740e563e318142f0cacd50859734" exitCode=0 Oct 30 23:56:08.784804 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:08.784457 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0a3b296-6qrvt" event={"ID":"0a2ebfe5-08c3-4d7e-a32c-e5ee5ef4673e","Type":"ContainerDied","Data":"21816518bcb866b47e586e81fb59288492cc740e563e318142f0cacd50859734"} Oct 30 23:56:09.957833 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:09.957806 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0a3b296-6qrvt" Oct 30 23:56:10.055958 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:10.055873 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a2ebfe5-08c3-4d7e-a32c-e5ee5ef4673e-kube-api-access-dcqt2\" (UniqueName: \"kubernetes.io/projected/0a2ebfe5-08c3-4d7e-a32c-e5ee5ef4673e-kube-api-access-dcqt2\") pod \"0a2ebfe5-08c3-4d7e-a32c-e5ee5ef4673e\" (UID: \"0a2ebfe5-08c3-4d7e-a32c-e5ee5ef4673e\") " Oct 30 23:56:10.056108 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:10.056014 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0a2ebfe5-08c3-4d7e-a32c-e5ee5ef4673e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a2ebfe5-08c3-4d7e-a32c-e5ee5ef4673e-maas-api-xks-ca\") pod \"0a2ebfe5-08c3-4d7e-a32c-e5ee5ef4673e\" (UID: \"0a2ebfe5-08c3-4d7e-a32c-e5ee5ef4673e\") " Oct 30 23:56:10.056108 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:10.056066 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a2ebfe5-08c3-4d7e-a32c-e5ee5ef4673e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a2ebfe5-08c3-4d7e-a32c-e5ee5ef4673e-maas-api-openshift-ca\") pod \"0a2ebfe5-08c3-4d7e-a32c-e5ee5ef4673e\" (UID: \"0a2ebfe5-08c3-4d7e-a32c-e5ee5ef4673e\") " Oct 30 23:56:10.056494 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:10.056459 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a2ebfe5-08c3-4d7e-a32c-e5ee5ef4673e-maas-api-openshift-ca" pod "0a2ebfe5-08c3-4d7e-a32c-e5ee5ef4673e" (UID: "0a2ebfe5-08c3-4d7e-a32c-e5ee5ef4673e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:56:10.058214 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:10.058184 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a2ebfe5-08c3-4d7e-a32c-e5ee5ef4673e-maas-api-xks-ca" pod "0a2ebfe5-08c3-4d7e-a32c-e5ee5ef4673e" (UID: "0a2ebfe5-08c3-4d7e-a32c-e5ee5ef4673e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 23:56:10.058308 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:10.058189 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a2ebfe5-08c3-4d7e-a32c-e5ee5ef4673e-kube-api-access-dcqt2" pod "0a2ebfe5-08c3-4d7e-a32c-e5ee5ef4673e" (UID: "0a2ebfe5-08c3-4d7e-a32c-e5ee5ef4673e"). InnerVolumeSpecName "kube-api-access-dcqt2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:56:10.157306 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:10.157267 4927 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a2ebfe5-08c3-4d7e-a32c-e5ee5ef4673e-maas-api-xks-ca\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 23:56:10.157306 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:10.157303 4927 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a2ebfe5-08c3-4d7e-a32c-e5ee5ef4673e-maas-api-openshift-ca\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 23:56:10.157563 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:10.157319 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dcqt2\" (UniqueName: \"kubernetes.io/projected/0a2ebfe5-08c3-4d7e-a32c-e5ee5ef4673e-kube-api-access-dcqt2\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 23:56:10.223130 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:10.223094 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d7c68c1c-gjckg"] Oct 30 23:56:10.223676 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:10.223661 4927 cpu_manager.go:414] "containerMap: removing container" podUID="495da7b4-1e63-46fa-ac83-6d3bb75692f8" containerName="istio-proxy" Oct 30 23:56:10.223728 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:10.223677 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="495da7b4-1e63-46fa-ac83-6d3bb75692f8" containerName="istio-proxy" Oct 30 23:56:10.223766 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:10.223727 4927 cpu_manager.go:414] "containerMap: removing container" podUID="0a2ebfe5-08c3-4d7e-a32c-e5ee5ef4673e" containerName="revoke-keys" Oct 30 23:56:10.223766 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:10.223734 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a2ebfe5-08c3-4d7e-a32c-e5ee5ef4673e" containerName="revoke-keys" Oct 30 23:56:10.223847 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:10.223836 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a2ebfe5-08c3-4d7e-a32c-e5ee5ef4673e" containerName="revoke-keys" Oct 30 23:56:10.226887 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:10.226870 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7c68c1c-gjckg" Oct 30 23:56:10.258714 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:10.258684 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24htg\" (UniqueName: \"kubernetes.io/projected/298bd1f1-9653-4535-9e79-682eb391f6d6-kube-api-access-24htg\") pod \"maas-api-revoke-sub-d7c68c1c-gjckg\" (UID: \"298bd1f1-9653-4535-9e79-682eb391f6d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7c68c1c-gjckg" Oct 30 23:56:10.258848 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:10.258731 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/298bd1f1-9653-4535-9e79-682eb391f6d6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d7c68c1c-gjckg\" (UID: \"298bd1f1-9653-4535-9e79-682eb391f6d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7c68c1c-gjckg" Oct 30 23:56:10.258912 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:10.258888 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/298bd1f1-9653-4535-9e79-682eb391f6d6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d7c68c1c-gjckg\" (UID: \"298bd1f1-9653-4535-9e79-682eb391f6d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7c68c1c-gjckg" Oct 30 23:56:10.286122 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:10.286088 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d7c68c1c-gjckg"] Oct 30 23:56:10.360487 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:10.360411 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24htg\" (UniqueName: \"kubernetes.io/projected/298bd1f1-9653-4535-9e79-682eb391f6d6-kube-api-access-24htg\") pod \"maas-api-revoke-sub-d7c68c1c-gjckg\" (UID: \"298bd1f1-9653-4535-9e79-682eb391f6d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7c68c1c-gjckg" Oct 30 23:56:10.360487 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:10.360458 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/298bd1f1-9653-4535-9e79-682eb391f6d6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d7c68c1c-gjckg\" (UID: \"298bd1f1-9653-4535-9e79-682eb391f6d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7c68c1c-gjckg" Oct 30 23:56:10.360666 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:10.360515 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/298bd1f1-9653-4535-9e79-682eb391f6d6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d7c68c1c-gjckg\" (UID: \"298bd1f1-9653-4535-9e79-682eb391f6d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7c68c1c-gjckg" Oct 30 23:56:10.361133 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:10.361112 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/298bd1f1-9653-4535-9e79-682eb391f6d6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d7c68c1c-gjckg\" (UID: \"298bd1f1-9653-4535-9e79-682eb391f6d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7c68c1c-gjckg" Oct 30 23:56:10.362870 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:10.362847 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/298bd1f1-9653-4535-9e79-682eb391f6d6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d7c68c1c-gjckg\" (UID: \"298bd1f1-9653-4535-9e79-682eb391f6d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7c68c1c-gjckg" Oct 30 23:56:10.423956 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:10.423925 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-24htg\" (UniqueName: \"kubernetes.io/projected/298bd1f1-9653-4535-9e79-682eb391f6d6-kube-api-access-24htg\") pod \"maas-api-revoke-sub-d7c68c1c-gjckg\" (UID: \"298bd1f1-9653-4535-9e79-682eb391f6d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7c68c1c-gjckg" Oct 30 23:56:10.541158 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:10.541120 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7c68c1c-gjckg" Oct 30 23:56:10.799002 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:10.798973 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0a3b296-6qrvt" Oct 30 23:56:10.799156 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:10.799006 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0a3b296-6qrvt" event={"ID":"0a2ebfe5-08c3-4d7e-a32c-e5ee5ef4673e","Type":"ContainerDied","Data":"5a07db9a742a9d0d14e167c559e7852c1efbcc2f86c8b887258ca5a2414dc46f"} Oct 30 23:56:10.799156 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:10.799058 4927 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5a07db9a742a9d0d14e167c559e7852c1efbcc2f86c8b887258ca5a2414dc46f" Oct 30 23:56:10.922256 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:10.922226 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d7c68c1c-gjckg"] Oct 30 23:56:10.923393 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:56:10.923371 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod298bd1f1_9653_4535_9e79_682eb391f6d6.slice/crio-5fd41f486e7c1c313269e6681c87c881498e731e43019c4526ef0411cc542a0e WatchSource:0}: Error finding container 5fd41f486e7c1c313269e6681c87c881498e731e43019c4526ef0411cc542a0e: Status 404 returned error can't find the container with id 5fd41f486e7c1c313269e6681c87c881498e731e43019c4526ef0411cc542a0e Oct 30 23:56:11.806471 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:11.806434 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7c68c1c-gjckg" event={"ID":"298bd1f1-9653-4535-9e79-682eb391f6d6","Type":"ContainerStarted","Data":"6e986a43dd884073b95d832fd7f709546742cb06280a1c11e76bcf8f0789e02f"} Oct 30 23:56:11.806962 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:11.806496 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7c68c1c-gjckg" event={"ID":"298bd1f1-9653-4535-9e79-682eb391f6d6","Type":"ContainerStarted","Data":"5fd41f486e7c1c313269e6681c87c881498e731e43019c4526ef0411cc542a0e"} Oct 30 23:56:11.926864 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:11.926815 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7c68c1c-gjckg" podStartSLOduration=1.9268002709999998 podStartE2EDuration="1.926800271s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:56:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:56:11.924707782 +0000 UTC m=+884.961038615" watchObservedRunningTime="2026-10-30 23:56:11.926800271 +0000 UTC m=+884.963131092" Oct 30 23:56:12.590009 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:12.589975 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:56:12.814321 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:12.814283 4927 generic.go:356] "Generic (PLEG): container finished" podID="298bd1f1-9653-4535-9e79-682eb391f6d6" containerID="6e986a43dd884073b95d832fd7f709546742cb06280a1c11e76bcf8f0789e02f" exitCode=0 Oct 30 23:56:12.814711 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:12.814376 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7c68c1c-gjckg" event={"ID":"298bd1f1-9653-4535-9e79-682eb391f6d6","Type":"ContainerDied","Data":"6e986a43dd884073b95d832fd7f709546742cb06280a1c11e76bcf8f0789e02f"} Oct 30 23:56:13.982545 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:13.982516 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7c68c1c-gjckg" Oct 30 23:56:14.103562 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:14.103475 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/298bd1f1-9653-4535-9e79-682eb391f6d6-kube-api-access-24htg\" (UniqueName: \"kubernetes.io/projected/298bd1f1-9653-4535-9e79-682eb391f6d6-kube-api-access-24htg\") pod \"298bd1f1-9653-4535-9e79-682eb391f6d6\" (UID: \"298bd1f1-9653-4535-9e79-682eb391f6d6\") " Oct 30 23:56:14.103562 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:14.103542 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/298bd1f1-9653-4535-9e79-682eb391f6d6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/298bd1f1-9653-4535-9e79-682eb391f6d6-maas-api-openshift-ca\") pod \"298bd1f1-9653-4535-9e79-682eb391f6d6\" (UID: \"298bd1f1-9653-4535-9e79-682eb391f6d6\") " Oct 30 23:56:14.103809 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:14.103718 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/298bd1f1-9653-4535-9e79-682eb391f6d6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/298bd1f1-9653-4535-9e79-682eb391f6d6-maas-api-xks-ca\") pod \"298bd1f1-9653-4535-9e79-682eb391f6d6\" (UID: \"298bd1f1-9653-4535-9e79-682eb391f6d6\") " Oct 30 23:56:14.103946 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:14.103923 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/298bd1f1-9653-4535-9e79-682eb391f6d6-maas-api-openshift-ca" pod "298bd1f1-9653-4535-9e79-682eb391f6d6" (UID: "298bd1f1-9653-4535-9e79-682eb391f6d6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:56:14.104076 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:14.104058 4927 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/298bd1f1-9653-4535-9e79-682eb391f6d6-maas-api-openshift-ca\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 23:56:14.106246 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:14.106081 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/298bd1f1-9653-4535-9e79-682eb391f6d6-maas-api-xks-ca" pod "298bd1f1-9653-4535-9e79-682eb391f6d6" (UID: "298bd1f1-9653-4535-9e79-682eb391f6d6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 23:56:14.106400 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:14.106316 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/298bd1f1-9653-4535-9e79-682eb391f6d6-kube-api-access-24htg" pod "298bd1f1-9653-4535-9e79-682eb391f6d6" (UID: "298bd1f1-9653-4535-9e79-682eb391f6d6"). InnerVolumeSpecName "kube-api-access-24htg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:56:14.205147 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:14.205094 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-24htg\" (UniqueName: \"kubernetes.io/projected/298bd1f1-9653-4535-9e79-682eb391f6d6-kube-api-access-24htg\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 23:56:14.205147 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:14.205128 4927 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/298bd1f1-9653-4535-9e79-682eb391f6d6-maas-api-xks-ca\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 23:56:14.831273 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:14.831230 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7c68c1c-gjckg" event={"ID":"298bd1f1-9653-4535-9e79-682eb391f6d6","Type":"ContainerDied","Data":"5fd41f486e7c1c313269e6681c87c881498e731e43019c4526ef0411cc542a0e"} Oct 30 23:56:14.831273 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:14.831263 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7c68c1c-gjckg" Oct 30 23:56:14.831496 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:14.831295 4927 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5fd41f486e7c1c313269e6681c87c881498e731e43019c4526ef0411cc542a0e" Oct 30 23:56:17.326913 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:17.326882 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:56:19.680688 ip-10-0-141-180 kubenswrapper[4927]: E1030 23:56:19.680637 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sqv69" containerName="extractor" Oct 30 23:56:21.292967 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:21.292932 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:56:21.518283 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:21.518242 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:56:22.050707 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:22.050666 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:56:22.185691 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:22.185647 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2821202e-ql97w"] Oct 30 23:56:22.186266 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:22.186246 4927 cpu_manager.go:414] "containerMap: removing container" podUID="298bd1f1-9653-4535-9e79-682eb391f6d6" containerName="revoke-keys" Oct 30 23:56:22.186384 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:22.186268 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="298bd1f1-9653-4535-9e79-682eb391f6d6" containerName="revoke-keys" Oct 30 23:56:22.186487 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:22.186473 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="298bd1f1-9653-4535-9e79-682eb391f6d6" containerName="revoke-keys" Oct 30 23:56:22.190783 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:22.190759 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2821202e-ql97w" Oct 30 23:56:22.248837 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:22.248804 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 23:56:22.303659 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:22.303577 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2821202e-ql97w"] Oct 30 23:56:22.327397 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:22.327364 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df44c490-7c92-4af4-908f-d4a01e2ab80c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2821202e-ql97w\" (UID: \"df44c490-7c92-4af4-908f-d4a01e2ab80c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2821202e-ql97w" Oct 30 23:56:22.327569 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:22.327474 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pn2mr\" (UniqueName: \"kubernetes.io/projected/df44c490-7c92-4af4-908f-d4a01e2ab80c-kube-api-access-pn2mr\") pod \"maas-api-revoke-sub-2821202e-ql97w\" (UID: \"df44c490-7c92-4af4-908f-d4a01e2ab80c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2821202e-ql97w" Oct 30 23:56:22.327569 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:22.327540 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df44c490-7c92-4af4-908f-d4a01e2ab80c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2821202e-ql97w\" (UID: \"df44c490-7c92-4af4-908f-d4a01e2ab80c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2821202e-ql97w" Oct 30 23:56:22.353396 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:22.353372 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4fdzs\"" Oct 30 23:56:22.369897 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:22.369868 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:56:22.428842 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:22.428802 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df44c490-7c92-4af4-908f-d4a01e2ab80c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2821202e-ql97w\" (UID: \"df44c490-7c92-4af4-908f-d4a01e2ab80c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2821202e-ql97w" Oct 30 23:56:22.429015 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:22.428853 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df44c490-7c92-4af4-908f-d4a01e2ab80c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2821202e-ql97w\" (UID: \"df44c490-7c92-4af4-908f-d4a01e2ab80c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2821202e-ql97w" Oct 30 23:56:22.429015 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:22.428949 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pn2mr\" (UniqueName: \"kubernetes.io/projected/df44c490-7c92-4af4-908f-d4a01e2ab80c-kube-api-access-pn2mr\") pod \"maas-api-revoke-sub-2821202e-ql97w\" (UID: \"df44c490-7c92-4af4-908f-d4a01e2ab80c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2821202e-ql97w" Oct 30 23:56:22.429673 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:22.429645 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df44c490-7c92-4af4-908f-d4a01e2ab80c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2821202e-ql97w\" (UID: \"df44c490-7c92-4af4-908f-d4a01e2ab80c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2821202e-ql97w" Oct 30 23:56:22.431269 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:22.431246 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df44c490-7c92-4af4-908f-d4a01e2ab80c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2821202e-ql97w\" (UID: \"df44c490-7c92-4af4-908f-d4a01e2ab80c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2821202e-ql97w" Oct 30 23:56:22.492936 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:22.492897 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pn2mr\" (UniqueName: \"kubernetes.io/projected/df44c490-7c92-4af4-908f-d4a01e2ab80c-kube-api-access-pn2mr\") pod \"maas-api-revoke-sub-2821202e-ql97w\" (UID: \"df44c490-7c92-4af4-908f-d4a01e2ab80c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2821202e-ql97w" Oct 30 23:56:22.504822 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:22.504782 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2821202e-ql97w" Oct 30 23:56:22.794135 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:22.794103 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:56:22.887184 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:56:22.887154 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf44c490_7c92_4af4_908f_d4a01e2ab80c.slice/crio-b1e7338aad8cedcf2a7a765dfd461fc36bfbfb0f5a4606f62772fa0188e26d58 WatchSource:0}: Error finding container b1e7338aad8cedcf2a7a765dfd461fc36bfbfb0f5a4606f62772fa0188e26d58: Status 404 returned error can't find the container with id b1e7338aad8cedcf2a7a765dfd461fc36bfbfb0f5a4606f62772fa0188e26d58 Oct 30 23:56:22.887184 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:22.887173 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2821202e-ql97w"] Oct 30 23:56:22.914226 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:22.914187 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2821202e-ql97w" event={"ID":"df44c490-7c92-4af4-908f-d4a01e2ab80c","Type":"ContainerStarted","Data":"b1e7338aad8cedcf2a7a765dfd461fc36bfbfb0f5a4606f62772fa0188e26d58"} Oct 30 23:56:23.921936 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:23.921900 4927 generic.go:356] "Generic (PLEG): container finished" podID="df44c490-7c92-4af4-908f-d4a01e2ab80c" containerID="b7cc1b138ee189acede2bb5ab38d61da87fb37e3b1461cec887402c08b103cab" exitCode=0 Oct 30 23:56:23.922307 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:23.921960 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2821202e-ql97w" event={"ID":"df44c490-7c92-4af4-908f-d4a01e2ab80c","Type":"ContainerDied","Data":"b7cc1b138ee189acede2bb5ab38d61da87fb37e3b1461cec887402c08b103cab"} Oct 30 23:56:25.084049 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:25.084025 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2821202e-ql97w" Oct 30 23:56:25.261170 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:25.261085 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df44c490-7c92-4af4-908f-d4a01e2ab80c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df44c490-7c92-4af4-908f-d4a01e2ab80c-maas-api-openshift-ca\") pod \"df44c490-7c92-4af4-908f-d4a01e2ab80c\" (UID: \"df44c490-7c92-4af4-908f-d4a01e2ab80c\") " Oct 30 23:56:25.261354 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:25.261187 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df44c490-7c92-4af4-908f-d4a01e2ab80c-kube-api-access-pn2mr\" (UniqueName: \"kubernetes.io/projected/df44c490-7c92-4af4-908f-d4a01e2ab80c-kube-api-access-pn2mr\") pod \"df44c490-7c92-4af4-908f-d4a01e2ab80c\" (UID: \"df44c490-7c92-4af4-908f-d4a01e2ab80c\") " Oct 30 23:56:25.261354 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:25.261239 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/df44c490-7c92-4af4-908f-d4a01e2ab80c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df44c490-7c92-4af4-908f-d4a01e2ab80c-maas-api-xks-ca\") pod \"df44c490-7c92-4af4-908f-d4a01e2ab80c\" (UID: \"df44c490-7c92-4af4-908f-d4a01e2ab80c\") " Oct 30 23:56:25.261605 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:25.261548 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df44c490-7c92-4af4-908f-d4a01e2ab80c-maas-api-openshift-ca" pod "df44c490-7c92-4af4-908f-d4a01e2ab80c" (UID: "df44c490-7c92-4af4-908f-d4a01e2ab80c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:56:25.263429 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:25.263403 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df44c490-7c92-4af4-908f-d4a01e2ab80c-kube-api-access-pn2mr" pod "df44c490-7c92-4af4-908f-d4a01e2ab80c" (UID: "df44c490-7c92-4af4-908f-d4a01e2ab80c"). InnerVolumeSpecName "kube-api-access-pn2mr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:56:25.263520 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:25.263460 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df44c490-7c92-4af4-908f-d4a01e2ab80c-maas-api-xks-ca" pod "df44c490-7c92-4af4-908f-d4a01e2ab80c" (UID: "df44c490-7c92-4af4-908f-d4a01e2ab80c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 23:56:25.362942 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:25.362908 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pn2mr\" (UniqueName: \"kubernetes.io/projected/df44c490-7c92-4af4-908f-d4a01e2ab80c-kube-api-access-pn2mr\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 23:56:25.362942 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:25.362937 4927 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df44c490-7c92-4af4-908f-d4a01e2ab80c-maas-api-xks-ca\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 23:56:25.362942 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:25.362949 4927 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df44c490-7c92-4af4-908f-d4a01e2ab80c-maas-api-openshift-ca\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 23:56:25.937044 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:25.937014 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2821202e-ql97w" Oct 30 23:56:25.937211 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:25.937047 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2821202e-ql97w" event={"ID":"df44c490-7c92-4af4-908f-d4a01e2ab80c","Type":"ContainerDied","Data":"b1e7338aad8cedcf2a7a765dfd461fc36bfbfb0f5a4606f62772fa0188e26d58"} Oct 30 23:56:25.937211 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:25.937101 4927 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b1e7338aad8cedcf2a7a765dfd461fc36bfbfb0f5a4606f62772fa0188e26d58" Oct 30 23:56:26.691312 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:26.691273 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:56:27.222753 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:27.222721 4927 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-pqt2c_6382dfb5-a363-47a2-bc79-0b96f7bec0c3/console-operator/3.log" Oct 30 23:56:27.222949 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:27.222911 4927 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-pqt2c_6382dfb5-a363-47a2-bc79-0b96f7bec0c3/console-operator/3.log" Oct 30 23:56:30.632878 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:30.632838 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3f61fa33-cbsnw"] Oct 30 23:56:30.633736 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:30.633714 4927 cpu_manager.go:414] "containerMap: removing container" podUID="df44c490-7c92-4af4-908f-d4a01e2ab80c" containerName="revoke-keys" Oct 30 23:56:30.633792 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:30.633739 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df44c490-7c92-4af4-908f-d4a01e2ab80c" containerName="revoke-keys" Oct 30 23:56:30.633967 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:30.633869 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="df44c490-7c92-4af4-908f-d4a01e2ab80c" containerName="revoke-keys" Oct 30 23:56:30.643309 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:30.643285 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f61fa33-cbsnw" Oct 30 23:56:30.694149 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:30.694119 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3f61fa33-cbsnw"] Oct 30 23:56:30.720752 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:30.720716 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5q8rt\" (UniqueName: \"kubernetes.io/projected/7e1e0858-406a-4fea-91c4-acf5d505d0b9-kube-api-access-5q8rt\") pod \"maas-api-revoke-sub-3f61fa33-cbsnw\" (UID: \"7e1e0858-406a-4fea-91c4-acf5d505d0b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f61fa33-cbsnw" Oct 30 23:56:30.720947 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:30.720848 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e1e0858-406a-4fea-91c4-acf5d505d0b9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3f61fa33-cbsnw\" (UID: \"7e1e0858-406a-4fea-91c4-acf5d505d0b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f61fa33-cbsnw" Oct 30 23:56:30.720947 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:30.720906 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e1e0858-406a-4fea-91c4-acf5d505d0b9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3f61fa33-cbsnw\" (UID: \"7e1e0858-406a-4fea-91c4-acf5d505d0b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f61fa33-cbsnw" Oct 30 23:56:30.740572 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:30.740545 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 23:56:30.740760 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:30.740583 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4fdzs\"" Oct 30 23:56:30.822406 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:30.822375 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e1e0858-406a-4fea-91c4-acf5d505d0b9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3f61fa33-cbsnw\" (UID: \"7e1e0858-406a-4fea-91c4-acf5d505d0b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f61fa33-cbsnw" Oct 30 23:56:30.822591 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:30.822439 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e1e0858-406a-4fea-91c4-acf5d505d0b9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3f61fa33-cbsnw\" (UID: \"7e1e0858-406a-4fea-91c4-acf5d505d0b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f61fa33-cbsnw" Oct 30 23:56:30.822591 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:30.822494 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5q8rt\" (UniqueName: \"kubernetes.io/projected/7e1e0858-406a-4fea-91c4-acf5d505d0b9-kube-api-access-5q8rt\") pod \"maas-api-revoke-sub-3f61fa33-cbsnw\" (UID: \"7e1e0858-406a-4fea-91c4-acf5d505d0b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f61fa33-cbsnw" Oct 30 23:56:30.823152 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:30.823120 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e1e0858-406a-4fea-91c4-acf5d505d0b9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3f61fa33-cbsnw\" (UID: \"7e1e0858-406a-4fea-91c4-acf5d505d0b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f61fa33-cbsnw" Oct 30 23:56:30.824829 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:30.824803 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e1e0858-406a-4fea-91c4-acf5d505d0b9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3f61fa33-cbsnw\" (UID: \"7e1e0858-406a-4fea-91c4-acf5d505d0b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f61fa33-cbsnw" Oct 30 23:56:30.885347 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:30.885267 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5q8rt\" (UniqueName: \"kubernetes.io/projected/7e1e0858-406a-4fea-91c4-acf5d505d0b9-kube-api-access-5q8rt\") pod \"maas-api-revoke-sub-3f61fa33-cbsnw\" (UID: \"7e1e0858-406a-4fea-91c4-acf5d505d0b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f61fa33-cbsnw" Oct 30 23:56:30.956902 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:30.956863 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f61fa33-cbsnw" Oct 30 23:56:31.343356 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:31.343303 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3f61fa33-cbsnw"] Oct 30 23:56:31.344196 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:56:31.344160 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e1e0858_406a_4fea_91c4_acf5d505d0b9.slice/crio-05f36ab98e5973136a0bbd002963ce2fd653f384e72a6cb3cb56349c43824d9e WatchSource:0}: Error finding container 05f36ab98e5973136a0bbd002963ce2fd653f384e72a6cb3cb56349c43824d9e: Status 404 returned error can't find the container with id 05f36ab98e5973136a0bbd002963ce2fd653f384e72a6cb3cb56349c43824d9e Oct 30 23:56:31.462673 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:31.462635 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:56:31.990306 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:31.990268 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f61fa33-cbsnw" event={"ID":"7e1e0858-406a-4fea-91c4-acf5d505d0b9","Type":"ContainerStarted","Data":"a5dd2ee348c74a1893b88d729dc14b12a7b0ddbf0a953f09df7cdc6d019b6790"} Oct 30 23:56:31.990751 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:31.990362 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f61fa33-cbsnw" event={"ID":"7e1e0858-406a-4fea-91c4-acf5d505d0b9","Type":"ContainerStarted","Data":"05f36ab98e5973136a0bbd002963ce2fd653f384e72a6cb3cb56349c43824d9e"} Oct 30 23:56:32.110735 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:32.110683 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f61fa33-cbsnw" podStartSLOduration=2.110663258 podStartE2EDuration="2.110663258s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:56:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:56:32.108880954 +0000 UTC m=+905.145211796" watchObservedRunningTime="2026-10-30 23:56:32.110663258 +0000 UTC m=+905.146994097" Oct 30 23:56:33.007635 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:33.007597 4927 generic.go:356] "Generic (PLEG): container finished" podID="7e1e0858-406a-4fea-91c4-acf5d505d0b9" containerID="a5dd2ee348c74a1893b88d729dc14b12a7b0ddbf0a953f09df7cdc6d019b6790" exitCode=0 Oct 30 23:56:33.008192 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:33.008169 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f61fa33-cbsnw" event={"ID":"7e1e0858-406a-4fea-91c4-acf5d505d0b9","Type":"ContainerDied","Data":"a5dd2ee348c74a1893b88d729dc14b12a7b0ddbf0a953f09df7cdc6d019b6790"} Oct 30 23:56:34.182837 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:34.182809 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f61fa33-cbsnw" Oct 30 23:56:34.266070 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:34.265983 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e1e0858-406a-4fea-91c4-acf5d505d0b9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e1e0858-406a-4fea-91c4-acf5d505d0b9-maas-api-openshift-ca\") pod \"7e1e0858-406a-4fea-91c4-acf5d505d0b9\" (UID: \"7e1e0858-406a-4fea-91c4-acf5d505d0b9\") " Oct 30 23:56:34.266070 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:34.266058 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e1e0858-406a-4fea-91c4-acf5d505d0b9-kube-api-access-5q8rt\" (UniqueName: \"kubernetes.io/projected/7e1e0858-406a-4fea-91c4-acf5d505d0b9-kube-api-access-5q8rt\") pod \"7e1e0858-406a-4fea-91c4-acf5d505d0b9\" (UID: \"7e1e0858-406a-4fea-91c4-acf5d505d0b9\") " Oct 30 23:56:34.266263 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:34.266102 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7e1e0858-406a-4fea-91c4-acf5d505d0b9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e1e0858-406a-4fea-91c4-acf5d505d0b9-maas-api-xks-ca\") pod \"7e1e0858-406a-4fea-91c4-acf5d505d0b9\" (UID: \"7e1e0858-406a-4fea-91c4-acf5d505d0b9\") " Oct 30 23:56:34.266447 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:34.266403 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e1e0858-406a-4fea-91c4-acf5d505d0b9-maas-api-openshift-ca" pod "7e1e0858-406a-4fea-91c4-acf5d505d0b9" (UID: "7e1e0858-406a-4fea-91c4-acf5d505d0b9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:56:34.266536 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:34.266525 4927 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e1e0858-406a-4fea-91c4-acf5d505d0b9-maas-api-openshift-ca\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 23:56:34.268105 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:34.268080 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e1e0858-406a-4fea-91c4-acf5d505d0b9-kube-api-access-5q8rt" pod "7e1e0858-406a-4fea-91c4-acf5d505d0b9" (UID: "7e1e0858-406a-4fea-91c4-acf5d505d0b9"). InnerVolumeSpecName "kube-api-access-5q8rt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:56:34.268520 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:34.268495 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e1e0858-406a-4fea-91c4-acf5d505d0b9-maas-api-xks-ca" pod "7e1e0858-406a-4fea-91c4-acf5d505d0b9" (UID: "7e1e0858-406a-4fea-91c4-acf5d505d0b9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 23:56:34.368038 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:34.367961 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5q8rt\" (UniqueName: \"kubernetes.io/projected/7e1e0858-406a-4fea-91c4-acf5d505d0b9-kube-api-access-5q8rt\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 23:56:34.368038 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:34.368010 4927 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e1e0858-406a-4fea-91c4-acf5d505d0b9-maas-api-xks-ca\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 23:56:35.026426 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:35.026390 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f61fa33-cbsnw" Oct 30 23:56:35.026626 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:35.026432 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f61fa33-cbsnw" event={"ID":"7e1e0858-406a-4fea-91c4-acf5d505d0b9","Type":"ContainerDied","Data":"05f36ab98e5973136a0bbd002963ce2fd653f384e72a6cb3cb56349c43824d9e"} Oct 30 23:56:35.026626 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:35.026479 4927 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="05f36ab98e5973136a0bbd002963ce2fd653f384e72a6cb3cb56349c43824d9e" Oct 30 23:56:35.347076 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:35.346989 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-29043c97-pmqhh"] Oct 30 23:56:35.347578 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:35.347562 4927 cpu_manager.go:414] "containerMap: removing container" podUID="7e1e0858-406a-4fea-91c4-acf5d505d0b9" containerName="revoke-keys" Oct 30 23:56:35.347647 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:35.347578 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e1e0858-406a-4fea-91c4-acf5d505d0b9" containerName="revoke-keys" Oct 30 23:56:35.347703 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:35.347691 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e1e0858-406a-4fea-91c4-acf5d505d0b9" containerName="revoke-keys" Oct 30 23:56:35.351371 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:35.351327 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29043c97-pmqhh" Oct 30 23:56:35.381183 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:35.381154 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba94b314-c7ea-4a07-9084-98f7a36edc1a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-29043c97-pmqhh\" (UID: \"ba94b314-c7ea-4a07-9084-98f7a36edc1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29043c97-pmqhh" Oct 30 23:56:35.381367 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:35.381193 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba94b314-c7ea-4a07-9084-98f7a36edc1a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-29043c97-pmqhh\" (UID: \"ba94b314-c7ea-4a07-9084-98f7a36edc1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29043c97-pmqhh" Oct 30 23:56:35.381367 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:35.381303 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r2hhx\" (UniqueName: \"kubernetes.io/projected/ba94b314-c7ea-4a07-9084-98f7a36edc1a-kube-api-access-r2hhx\") pod \"maas-api-revoke-sub-29043c97-pmqhh\" (UID: \"ba94b314-c7ea-4a07-9084-98f7a36edc1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29043c97-pmqhh" Oct 30 23:56:35.407592 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:35.407561 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4fdzs\"" Oct 30 23:56:35.407856 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:35.407829 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 23:56:35.410406 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:35.410381 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-29043c97-pmqhh"] Oct 30 23:56:35.482858 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:35.482811 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba94b314-c7ea-4a07-9084-98f7a36edc1a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-29043c97-pmqhh\" (UID: \"ba94b314-c7ea-4a07-9084-98f7a36edc1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29043c97-pmqhh" Oct 30 23:56:35.483022 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:35.482898 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2hhx\" (UniqueName: \"kubernetes.io/projected/ba94b314-c7ea-4a07-9084-98f7a36edc1a-kube-api-access-r2hhx\") pod \"maas-api-revoke-sub-29043c97-pmqhh\" (UID: \"ba94b314-c7ea-4a07-9084-98f7a36edc1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29043c97-pmqhh" Oct 30 23:56:35.483082 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:35.483066 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba94b314-c7ea-4a07-9084-98f7a36edc1a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-29043c97-pmqhh\" (UID: \"ba94b314-c7ea-4a07-9084-98f7a36edc1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29043c97-pmqhh" Oct 30 23:56:35.483517 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:35.483493 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba94b314-c7ea-4a07-9084-98f7a36edc1a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-29043c97-pmqhh\" (UID: \"ba94b314-c7ea-4a07-9084-98f7a36edc1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29043c97-pmqhh" Oct 30 23:56:35.485340 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:35.485302 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba94b314-c7ea-4a07-9084-98f7a36edc1a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-29043c97-pmqhh\" (UID: \"ba94b314-c7ea-4a07-9084-98f7a36edc1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29043c97-pmqhh" Oct 30 23:56:35.545299 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:35.545261 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2hhx\" (UniqueName: \"kubernetes.io/projected/ba94b314-c7ea-4a07-9084-98f7a36edc1a-kube-api-access-r2hhx\") pod \"maas-api-revoke-sub-29043c97-pmqhh\" (UID: \"ba94b314-c7ea-4a07-9084-98f7a36edc1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29043c97-pmqhh" Oct 30 23:56:35.667945 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:35.667912 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29043c97-pmqhh" Oct 30 23:56:36.002573 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:36.002483 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:56:36.049791 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:36.049754 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-29043c97-pmqhh"] Oct 30 23:56:36.054248 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:56:36.054214 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba94b314_c7ea_4a07_9084_98f7a36edc1a.slice/crio-ff436c5615793b74d808bf6b99d44af3bd0fda3bbfc76e7f3d269a28a7f8d423 WatchSource:0}: Error finding container ff436c5615793b74d808bf6b99d44af3bd0fda3bbfc76e7f3d269a28a7f8d423: Status 404 returned error can't find the container with id ff436c5615793b74d808bf6b99d44af3bd0fda3bbfc76e7f3d269a28a7f8d423 Oct 30 23:56:37.055825 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:37.054768 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29043c97-pmqhh" event={"ID":"ba94b314-c7ea-4a07-9084-98f7a36edc1a","Type":"ContainerStarted","Data":"59199777c694eb7ed10cbe8e1086608848bb52e103b9d2a524c675897b93d072"} Oct 30 23:56:37.055825 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:37.054846 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29043c97-pmqhh" event={"ID":"ba94b314-c7ea-4a07-9084-98f7a36edc1a","Type":"ContainerStarted","Data":"ff436c5615793b74d808bf6b99d44af3bd0fda3bbfc76e7f3d269a28a7f8d423"} Oct 30 23:56:37.182602 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:37.182532 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29043c97-pmqhh" podStartSLOduration=2.182512674 podStartE2EDuration="2.182512674s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:56:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:56:37.179640569 +0000 UTC m=+910.215971424" watchObservedRunningTime="2026-10-30 23:56:37.182512674 +0000 UTC m=+910.218843497" Oct 30 23:56:38.064401 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:38.064366 4927 generic.go:356] "Generic (PLEG): container finished" podID="ba94b314-c7ea-4a07-9084-98f7a36edc1a" containerID="59199777c694eb7ed10cbe8e1086608848bb52e103b9d2a524c675897b93d072" exitCode=0 Oct 30 23:56:38.064860 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:38.064453 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29043c97-pmqhh" event={"ID":"ba94b314-c7ea-4a07-9084-98f7a36edc1a","Type":"ContainerDied","Data":"59199777c694eb7ed10cbe8e1086608848bb52e103b9d2a524c675897b93d072"} Oct 30 23:56:39.250937 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:39.250905 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29043c97-pmqhh" Oct 30 23:56:39.348462 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:39.348381 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba94b314-c7ea-4a07-9084-98f7a36edc1a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba94b314-c7ea-4a07-9084-98f7a36edc1a-maas-api-openshift-ca\") pod \"ba94b314-c7ea-4a07-9084-98f7a36edc1a\" (UID: \"ba94b314-c7ea-4a07-9084-98f7a36edc1a\") " Oct 30 23:56:39.348608 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:39.348481 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ba94b314-c7ea-4a07-9084-98f7a36edc1a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba94b314-c7ea-4a07-9084-98f7a36edc1a-maas-api-xks-ca\") pod \"ba94b314-c7ea-4a07-9084-98f7a36edc1a\" (UID: \"ba94b314-c7ea-4a07-9084-98f7a36edc1a\") " Oct 30 23:56:39.348608 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:39.348530 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba94b314-c7ea-4a07-9084-98f7a36edc1a-kube-api-access-r2hhx\" (UniqueName: \"kubernetes.io/projected/ba94b314-c7ea-4a07-9084-98f7a36edc1a-kube-api-access-r2hhx\") pod \"ba94b314-c7ea-4a07-9084-98f7a36edc1a\" (UID: \"ba94b314-c7ea-4a07-9084-98f7a36edc1a\") " Oct 30 23:56:39.348749 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:39.348725 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba94b314-c7ea-4a07-9084-98f7a36edc1a-maas-api-openshift-ca" pod "ba94b314-c7ea-4a07-9084-98f7a36edc1a" (UID: "ba94b314-c7ea-4a07-9084-98f7a36edc1a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:56:39.348813 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:39.348794 4927 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba94b314-c7ea-4a07-9084-98f7a36edc1a-maas-api-openshift-ca\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 23:56:39.350659 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:39.350623 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba94b314-c7ea-4a07-9084-98f7a36edc1a-kube-api-access-r2hhx" pod "ba94b314-c7ea-4a07-9084-98f7a36edc1a" (UID: "ba94b314-c7ea-4a07-9084-98f7a36edc1a"). InnerVolumeSpecName "kube-api-access-r2hhx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:56:39.350659 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:39.350646 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ba94b314-c7ea-4a07-9084-98f7a36edc1a-maas-api-xks-ca" pod "ba94b314-c7ea-4a07-9084-98f7a36edc1a" (UID: "ba94b314-c7ea-4a07-9084-98f7a36edc1a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 23:56:39.449719 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:39.449677 4927 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba94b314-c7ea-4a07-9084-98f7a36edc1a-maas-api-xks-ca\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 23:56:39.449719 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:39.449714 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r2hhx\" (UniqueName: \"kubernetes.io/projected/ba94b314-c7ea-4a07-9084-98f7a36edc1a-kube-api-access-r2hhx\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 23:56:40.080790 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:40.080746 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29043c97-pmqhh" event={"ID":"ba94b314-c7ea-4a07-9084-98f7a36edc1a","Type":"ContainerDied","Data":"ff436c5615793b74d808bf6b99d44af3bd0fda3bbfc76e7f3d269a28a7f8d423"} Oct 30 23:56:40.080937 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:40.080806 4927 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ff436c5615793b74d808bf6b99d44af3bd0fda3bbfc76e7f3d269a28a7f8d423" Oct 30 23:56:40.080937 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:40.080767 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29043c97-pmqhh" Oct 30 23:56:40.615775 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:40.615738 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:56:41.465425 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:41.465388 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-719bfdf0-t6rgr"] Oct 30 23:56:41.465935 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:41.465922 4927 cpu_manager.go:414] "containerMap: removing container" podUID="ba94b314-c7ea-4a07-9084-98f7a36edc1a" containerName="revoke-keys" Oct 30 23:56:41.465976 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:41.465937 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba94b314-c7ea-4a07-9084-98f7a36edc1a" containerName="revoke-keys" Oct 30 23:56:41.466025 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:41.466016 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba94b314-c7ea-4a07-9084-98f7a36edc1a" containerName="revoke-keys" Oct 30 23:56:41.469537 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:41.469522 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-719bfdf0-t6rgr" Oct 30 23:56:41.532650 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:41.532612 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-719bfdf0-t6rgr"] Oct 30 23:56:41.572589 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:41.572555 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fg5hp\" (UniqueName: \"kubernetes.io/projected/1385f845-a8bf-46a4-8c44-1974c26c0c65-kube-api-access-fg5hp\") pod \"maas-api-revoke-sub-719bfdf0-t6rgr\" (UID: \"1385f845-a8bf-46a4-8c44-1974c26c0c65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-719bfdf0-t6rgr" Oct 30 23:56:41.572784 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:41.572719 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1385f845-a8bf-46a4-8c44-1974c26c0c65-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-719bfdf0-t6rgr\" (UID: \"1385f845-a8bf-46a4-8c44-1974c26c0c65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-719bfdf0-t6rgr" Oct 30 23:56:41.572784 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:41.572774 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1385f845-a8bf-46a4-8c44-1974c26c0c65-maas-api-xks-ca\") pod \"maas-api-revoke-sub-719bfdf0-t6rgr\" (UID: \"1385f845-a8bf-46a4-8c44-1974c26c0c65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-719bfdf0-t6rgr" Oct 30 23:56:41.579106 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:41.579078 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 23:56:41.579367 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:41.579326 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4fdzs\"" Oct 30 23:56:41.673775 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:41.673740 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fg5hp\" (UniqueName: \"kubernetes.io/projected/1385f845-a8bf-46a4-8c44-1974c26c0c65-kube-api-access-fg5hp\") pod \"maas-api-revoke-sub-719bfdf0-t6rgr\" (UID: \"1385f845-a8bf-46a4-8c44-1974c26c0c65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-719bfdf0-t6rgr" Oct 30 23:56:41.674184 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:41.673873 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1385f845-a8bf-46a4-8c44-1974c26c0c65-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-719bfdf0-t6rgr\" (UID: \"1385f845-a8bf-46a4-8c44-1974c26c0c65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-719bfdf0-t6rgr" Oct 30 23:56:41.674184 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:41.673924 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1385f845-a8bf-46a4-8c44-1974c26c0c65-maas-api-xks-ca\") pod \"maas-api-revoke-sub-719bfdf0-t6rgr\" (UID: \"1385f845-a8bf-46a4-8c44-1974c26c0c65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-719bfdf0-t6rgr" Oct 30 23:56:41.674736 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:41.674712 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1385f845-a8bf-46a4-8c44-1974c26c0c65-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-719bfdf0-t6rgr\" (UID: \"1385f845-a8bf-46a4-8c44-1974c26c0c65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-719bfdf0-t6rgr" Oct 30 23:56:41.676398 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:41.676372 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1385f845-a8bf-46a4-8c44-1974c26c0c65-maas-api-xks-ca\") pod \"maas-api-revoke-sub-719bfdf0-t6rgr\" (UID: \"1385f845-a8bf-46a4-8c44-1974c26c0c65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-719bfdf0-t6rgr" Oct 30 23:56:41.736859 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:41.736783 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fg5hp\" (UniqueName: \"kubernetes.io/projected/1385f845-a8bf-46a4-8c44-1974c26c0c65-kube-api-access-fg5hp\") pod \"maas-api-revoke-sub-719bfdf0-t6rgr\" (UID: \"1385f845-a8bf-46a4-8c44-1974c26c0c65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-719bfdf0-t6rgr" Oct 30 23:56:41.783112 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:41.783078 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-719bfdf0-t6rgr" Oct 30 23:56:42.174680 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:56:42.174644 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1385f845_a8bf_46a4_8c44_1974c26c0c65.slice/crio-dc1b62d29f4e5337792e3bb05d1a72793366323a92b7973cadc73438ef50d168 WatchSource:0}: Error finding container dc1b62d29f4e5337792e3bb05d1a72793366323a92b7973cadc73438ef50d168: Status 404 returned error can't find the container with id dc1b62d29f4e5337792e3bb05d1a72793366323a92b7973cadc73438ef50d168 Oct 30 23:56:42.174886 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:42.174857 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-719bfdf0-t6rgr"] Oct 30 23:56:43.100543 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:43.100513 4927 generic.go:356] "Generic (PLEG): container finished" podID="1385f845-a8bf-46a4-8c44-1974c26c0c65" containerID="6422c129c8065907b4ea48d95adf2c558278dfa9428ae3496d96e4329b817e48" exitCode=0 Oct 30 23:56:43.100857 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:43.100594 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-719bfdf0-t6rgr" event={"ID":"1385f845-a8bf-46a4-8c44-1974c26c0c65","Type":"ContainerDied","Data":"6422c129c8065907b4ea48d95adf2c558278dfa9428ae3496d96e4329b817e48"} Oct 30 23:56:43.100857 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:43.100645 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-719bfdf0-t6rgr" event={"ID":"1385f845-a8bf-46a4-8c44-1974c26c0c65","Type":"ContainerStarted","Data":"dc1b62d29f4e5337792e3bb05d1a72793366323a92b7973cadc73438ef50d168"} Oct 30 23:56:44.281229 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:44.281198 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-719bfdf0-t6rgr" Oct 30 23:56:44.300467 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:44.300435 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1385f845-a8bf-46a4-8c44-1974c26c0c65-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1385f845-a8bf-46a4-8c44-1974c26c0c65-maas-api-xks-ca\") pod \"1385f845-a8bf-46a4-8c44-1974c26c0c65\" (UID: \"1385f845-a8bf-46a4-8c44-1974c26c0c65\") " Oct 30 23:56:44.300617 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:44.300505 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1385f845-a8bf-46a4-8c44-1974c26c0c65-kube-api-access-fg5hp\" (UniqueName: \"kubernetes.io/projected/1385f845-a8bf-46a4-8c44-1974c26c0c65-kube-api-access-fg5hp\") pod \"1385f845-a8bf-46a4-8c44-1974c26c0c65\" (UID: \"1385f845-a8bf-46a4-8c44-1974c26c0c65\") " Oct 30 23:56:44.300667 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:44.300623 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1385f845-a8bf-46a4-8c44-1974c26c0c65-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1385f845-a8bf-46a4-8c44-1974c26c0c65-maas-api-openshift-ca\") pod \"1385f845-a8bf-46a4-8c44-1974c26c0c65\" (UID: \"1385f845-a8bf-46a4-8c44-1974c26c0c65\") " Oct 30 23:56:44.300963 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:44.300929 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1385f845-a8bf-46a4-8c44-1974c26c0c65-maas-api-openshift-ca" pod "1385f845-a8bf-46a4-8c44-1974c26c0c65" (UID: "1385f845-a8bf-46a4-8c44-1974c26c0c65"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:56:44.301087 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:44.301068 4927 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1385f845-a8bf-46a4-8c44-1974c26c0c65-maas-api-openshift-ca\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 23:56:44.302675 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:44.302651 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1385f845-a8bf-46a4-8c44-1974c26c0c65-kube-api-access-fg5hp" pod "1385f845-a8bf-46a4-8c44-1974c26c0c65" (UID: "1385f845-a8bf-46a4-8c44-1974c26c0c65"). InnerVolumeSpecName "kube-api-access-fg5hp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:56:44.302938 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:44.302915 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1385f845-a8bf-46a4-8c44-1974c26c0c65-maas-api-xks-ca" pod "1385f845-a8bf-46a4-8c44-1974c26c0c65" (UID: "1385f845-a8bf-46a4-8c44-1974c26c0c65"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 23:56:44.401878 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:44.401800 4927 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1385f845-a8bf-46a4-8c44-1974c26c0c65-maas-api-xks-ca\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 23:56:44.401878 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:44.401828 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fg5hp\" (UniqueName: \"kubernetes.io/projected/1385f845-a8bf-46a4-8c44-1974c26c0c65-kube-api-access-fg5hp\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 23:56:45.115418 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:45.115377 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-719bfdf0-t6rgr" event={"ID":"1385f845-a8bf-46a4-8c44-1974c26c0c65","Type":"ContainerDied","Data":"dc1b62d29f4e5337792e3bb05d1a72793366323a92b7973cadc73438ef50d168"} Oct 30 23:56:45.115608 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:45.115441 4927 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dc1b62d29f4e5337792e3bb05d1a72793366323a92b7973cadc73438ef50d168" Oct 30 23:56:45.115608 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:45.115390 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-719bfdf0-t6rgr" Oct 30 23:56:45.327407 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:45.327367 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:56:49.678299 ip-10-0-141-180 kubenswrapper[4927]: E1030 23:56:49.678255 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sqv69" containerName="extractor" Oct 30 23:56:50.018081 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:50.017986 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:56:54.675466 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:54.675437 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:56:57.112531 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:57.112488 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d5cc73ad-pf2kb"] Oct 30 23:56:57.113097 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:57.113078 4927 cpu_manager.go:414] "containerMap: removing container" podUID="1385f845-a8bf-46a4-8c44-1974c26c0c65" containerName="revoke-keys" Oct 30 23:56:57.113168 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:57.113098 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1385f845-a8bf-46a4-8c44-1974c26c0c65" containerName="revoke-keys" Oct 30 23:56:57.113227 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:57.113197 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="1385f845-a8bf-46a4-8c44-1974c26c0c65" containerName="revoke-keys" Oct 30 23:56:57.117805 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:57.117784 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5cc73ad-pf2kb" Oct 30 23:56:57.195825 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:57.195790 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d5cc73ad-pf2kb"] Oct 30 23:56:57.239772 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:57.239743 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 23:56:57.239943 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:57.239852 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4fdzs\"" Oct 30 23:56:57.258592 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:57.258556 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7m2w\" (UniqueName: \"kubernetes.io/projected/833799d2-5e63-4445-9e34-42bbb7e13359-kube-api-access-b7m2w\") pod \"maas-api-revoke-sub-d5cc73ad-pf2kb\" (UID: \"833799d2-5e63-4445-9e34-42bbb7e13359\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5cc73ad-pf2kb" Oct 30 23:56:57.258733 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:57.258676 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/833799d2-5e63-4445-9e34-42bbb7e13359-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d5cc73ad-pf2kb\" (UID: \"833799d2-5e63-4445-9e34-42bbb7e13359\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5cc73ad-pf2kb" Oct 30 23:56:57.258799 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:57.258742 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/833799d2-5e63-4445-9e34-42bbb7e13359-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d5cc73ad-pf2kb\" (UID: \"833799d2-5e63-4445-9e34-42bbb7e13359\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5cc73ad-pf2kb" Oct 30 23:56:57.359549 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:57.359518 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/833799d2-5e63-4445-9e34-42bbb7e13359-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d5cc73ad-pf2kb\" (UID: \"833799d2-5e63-4445-9e34-42bbb7e13359\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5cc73ad-pf2kb" Oct 30 23:56:57.359703 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:57.359684 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/833799d2-5e63-4445-9e34-42bbb7e13359-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d5cc73ad-pf2kb\" (UID: \"833799d2-5e63-4445-9e34-42bbb7e13359\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5cc73ad-pf2kb" Oct 30 23:56:57.359845 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:57.359826 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7m2w\" (UniqueName: \"kubernetes.io/projected/833799d2-5e63-4445-9e34-42bbb7e13359-kube-api-access-b7m2w\") pod \"maas-api-revoke-sub-d5cc73ad-pf2kb\" (UID: \"833799d2-5e63-4445-9e34-42bbb7e13359\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5cc73ad-pf2kb" Oct 30 23:56:57.360289 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:57.360264 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/833799d2-5e63-4445-9e34-42bbb7e13359-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d5cc73ad-pf2kb\" (UID: \"833799d2-5e63-4445-9e34-42bbb7e13359\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5cc73ad-pf2kb" Oct 30 23:56:57.362127 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:57.362105 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/833799d2-5e63-4445-9e34-42bbb7e13359-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d5cc73ad-pf2kb\" (UID: \"833799d2-5e63-4445-9e34-42bbb7e13359\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5cc73ad-pf2kb" Oct 30 23:56:57.421983 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:57.421961 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7m2w\" (UniqueName: \"kubernetes.io/projected/833799d2-5e63-4445-9e34-42bbb7e13359-kube-api-access-b7m2w\") pod \"maas-api-revoke-sub-d5cc73ad-pf2kb\" (UID: \"833799d2-5e63-4445-9e34-42bbb7e13359\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5cc73ad-pf2kb" Oct 30 23:56:57.431498 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:57.431469 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5cc73ad-pf2kb" Oct 30 23:56:57.832113 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:57.831062 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d5cc73ad-pf2kb"] Oct 30 23:56:57.835179 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:56:57.835146 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod833799d2_5e63_4445_9e34_42bbb7e13359.slice/crio-068ee39067536350c835fd96f67da87590a9493d45a5299508cd7be7cd01b88d WatchSource:0}: Error finding container 068ee39067536350c835fd96f67da87590a9493d45a5299508cd7be7cd01b88d: Status 404 returned error can't find the container with id 068ee39067536350c835fd96f67da87590a9493d45a5299508cd7be7cd01b88d Oct 30 23:56:58.224104 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:58.223435 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5cc73ad-pf2kb" event={"ID":"833799d2-5e63-4445-9e34-42bbb7e13359","Type":"ContainerStarted","Data":"32f0b272a7ef173c0be0f24ba3215a3f055cbd986f858eb00d742eec60bf56a3"} Oct 30 23:56:58.224104 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:58.223501 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5cc73ad-pf2kb" event={"ID":"833799d2-5e63-4445-9e34-42bbb7e13359","Type":"ContainerStarted","Data":"068ee39067536350c835fd96f67da87590a9493d45a5299508cd7be7cd01b88d"} Oct 30 23:56:58.329412 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:58.329378 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c43c9a94-nxv5l"] Oct 30 23:56:58.333394 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:58.333374 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c43c9a94-nxv5l" Oct 30 23:56:58.345579 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:58.345519 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5cc73ad-pf2kb" podStartSLOduration=1.345501349 podStartE2EDuration="1.345501349s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:56:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:56:58.34185734 +0000 UTC m=+931.378188163" watchObservedRunningTime="2026-10-30 23:56:58.345501349 +0000 UTC m=+931.381832173" Oct 30 23:56:58.394225 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:58.394140 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c43c9a94-nxv5l"] Oct 30 23:56:58.472907 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:58.472872 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c6b629f-f032-469a-a06f-399b6a1a19c8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c43c9a94-nxv5l\" (UID: \"1c6b629f-f032-469a-a06f-399b6a1a19c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c43c9a94-nxv5l" Oct 30 23:56:58.473101 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:58.473000 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqmdk\" (UniqueName: \"kubernetes.io/projected/1c6b629f-f032-469a-a06f-399b6a1a19c8-kube-api-access-gqmdk\") pod \"maas-api-revoke-sub-c43c9a94-nxv5l\" (UID: \"1c6b629f-f032-469a-a06f-399b6a1a19c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c43c9a94-nxv5l" Oct 30 23:56:58.473101 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:58.473052 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c6b629f-f032-469a-a06f-399b6a1a19c8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c43c9a94-nxv5l\" (UID: \"1c6b629f-f032-469a-a06f-399b6a1a19c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c43c9a94-nxv5l" Oct 30 23:56:58.574569 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:58.574528 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c6b629f-f032-469a-a06f-399b6a1a19c8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c43c9a94-nxv5l\" (UID: \"1c6b629f-f032-469a-a06f-399b6a1a19c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c43c9a94-nxv5l" Oct 30 23:56:58.574759 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:58.574621 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqmdk\" (UniqueName: \"kubernetes.io/projected/1c6b629f-f032-469a-a06f-399b6a1a19c8-kube-api-access-gqmdk\") pod \"maas-api-revoke-sub-c43c9a94-nxv5l\" (UID: \"1c6b629f-f032-469a-a06f-399b6a1a19c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c43c9a94-nxv5l" Oct 30 23:56:58.574759 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:58.574655 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c6b629f-f032-469a-a06f-399b6a1a19c8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c43c9a94-nxv5l\" (UID: \"1c6b629f-f032-469a-a06f-399b6a1a19c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c43c9a94-nxv5l" Oct 30 23:56:58.575347 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:58.575305 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c6b629f-f032-469a-a06f-399b6a1a19c8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c43c9a94-nxv5l\" (UID: \"1c6b629f-f032-469a-a06f-399b6a1a19c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c43c9a94-nxv5l" Oct 30 23:56:58.576910 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:58.576885 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c6b629f-f032-469a-a06f-399b6a1a19c8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c43c9a94-nxv5l\" (UID: \"1c6b629f-f032-469a-a06f-399b6a1a19c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c43c9a94-nxv5l" Oct 30 23:56:58.638111 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:58.638077 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqmdk\" (UniqueName: \"kubernetes.io/projected/1c6b629f-f032-469a-a06f-399b6a1a19c8-kube-api-access-gqmdk\") pod \"maas-api-revoke-sub-c43c9a94-nxv5l\" (UID: \"1c6b629f-f032-469a-a06f-399b6a1a19c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c43c9a94-nxv5l" Oct 30 23:56:58.647904 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:58.647839 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c43c9a94-nxv5l" Oct 30 23:56:58.735551 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:58.735521 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-20c2fd60-6jhdm"] Oct 30 23:56:58.740133 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:58.740108 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20c2fd60-6jhdm" Oct 30 23:56:58.797372 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:58.790408 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-20c2fd60-6jhdm"] Oct 30 23:56:58.877735 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:58.877690 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1aa39837-dbe0-4478-a514-b4afc328c705-maas-api-xks-ca\") pod \"maas-api-revoke-sub-20c2fd60-6jhdm\" (UID: \"1aa39837-dbe0-4478-a514-b4afc328c705\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20c2fd60-6jhdm" Oct 30 23:56:58.877933 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:58.877760 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhxkg\" (UniqueName: \"kubernetes.io/projected/1aa39837-dbe0-4478-a514-b4afc328c705-kube-api-access-hhxkg\") pod \"maas-api-revoke-sub-20c2fd60-6jhdm\" (UID: \"1aa39837-dbe0-4478-a514-b4afc328c705\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20c2fd60-6jhdm" Oct 30 23:56:58.877933 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:58.877878 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1aa39837-dbe0-4478-a514-b4afc328c705-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-20c2fd60-6jhdm\" (UID: \"1aa39837-dbe0-4478-a514-b4afc328c705\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20c2fd60-6jhdm" Oct 30 23:56:58.979560 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:58.979527 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1aa39837-dbe0-4478-a514-b4afc328c705-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-20c2fd60-6jhdm\" (UID: \"1aa39837-dbe0-4478-a514-b4afc328c705\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20c2fd60-6jhdm" Oct 30 23:56:58.979754 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:58.979665 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1aa39837-dbe0-4478-a514-b4afc328c705-maas-api-xks-ca\") pod \"maas-api-revoke-sub-20c2fd60-6jhdm\" (UID: \"1aa39837-dbe0-4478-a514-b4afc328c705\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20c2fd60-6jhdm" Oct 30 23:56:58.979754 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:58.979725 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhxkg\" (UniqueName: \"kubernetes.io/projected/1aa39837-dbe0-4478-a514-b4afc328c705-kube-api-access-hhxkg\") pod \"maas-api-revoke-sub-20c2fd60-6jhdm\" (UID: \"1aa39837-dbe0-4478-a514-b4afc328c705\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20c2fd60-6jhdm" Oct 30 23:56:58.980237 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:58.980215 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1aa39837-dbe0-4478-a514-b4afc328c705-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-20c2fd60-6jhdm\" (UID: \"1aa39837-dbe0-4478-a514-b4afc328c705\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20c2fd60-6jhdm" Oct 30 23:56:58.981929 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:58.981910 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1aa39837-dbe0-4478-a514-b4afc328c705-maas-api-xks-ca\") pod \"maas-api-revoke-sub-20c2fd60-6jhdm\" (UID: \"1aa39837-dbe0-4478-a514-b4afc328c705\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20c2fd60-6jhdm" Oct 30 23:56:59.034458 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:59.034428 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c43c9a94-nxv5l"] Oct 30 23:56:59.035360 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:56:59.035316 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c6b629f_f032_469a_a06f_399b6a1a19c8.slice/crio-761c647c439332fd4ff995da2277eb6cfb67bc52f5447438d727038383241879 WatchSource:0}: Error finding container 761c647c439332fd4ff995da2277eb6cfb67bc52f5447438d727038383241879: Status 404 returned error can't find the container with id 761c647c439332fd4ff995da2277eb6cfb67bc52f5447438d727038383241879 Oct 30 23:56:59.042777 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:59.042749 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhxkg\" (UniqueName: \"kubernetes.io/projected/1aa39837-dbe0-4478-a514-b4afc328c705-kube-api-access-hhxkg\") pod \"maas-api-revoke-sub-20c2fd60-6jhdm\" (UID: \"1aa39837-dbe0-4478-a514-b4afc328c705\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20c2fd60-6jhdm" Oct 30 23:56:59.069795 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:59.067500 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20c2fd60-6jhdm" Oct 30 23:56:59.231912 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:59.231840 4927 generic.go:356] "Generic (PLEG): container finished" podID="833799d2-5e63-4445-9e34-42bbb7e13359" containerID="32f0b272a7ef173c0be0f24ba3215a3f055cbd986f858eb00d742eec60bf56a3" exitCode=0 Oct 30 23:56:59.232282 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:59.231925 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5cc73ad-pf2kb" event={"ID":"833799d2-5e63-4445-9e34-42bbb7e13359","Type":"ContainerDied","Data":"32f0b272a7ef173c0be0f24ba3215a3f055cbd986f858eb00d742eec60bf56a3"} Oct 30 23:56:59.233325 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:59.233302 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c43c9a94-nxv5l" event={"ID":"1c6b629f-f032-469a-a06f-399b6a1a19c8","Type":"ContainerStarted","Data":"761c647c439332fd4ff995da2277eb6cfb67bc52f5447438d727038383241879"} Oct 30 23:56:59.437791 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:59.437748 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:56:59.462805 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:56:59.462775 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-20c2fd60-6jhdm"] Oct 30 23:56:59.476592 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:56:59.465560 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1aa39837_dbe0_4478_a514_b4afc328c705.slice/crio-9490e339e6e189dc258fb10d3e650c2a1c61725b44de2b732690662a629be1e5 WatchSource:0}: Error finding container 9490e339e6e189dc258fb10d3e650c2a1c61725b44de2b732690662a629be1e5: Status 404 returned error can't find the container with id 9490e339e6e189dc258fb10d3e650c2a1c61725b44de2b732690662a629be1e5 Oct 30 23:57:00.242831 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:00.242798 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20c2fd60-6jhdm" event={"ID":"1aa39837-dbe0-4478-a514-b4afc328c705","Type":"ContainerStarted","Data":"888de4f50d24dcad4afad7f9ff2a9763835d2b6dd87987c07328775148cc797b"} Oct 30 23:57:00.243301 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:00.242856 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20c2fd60-6jhdm" event={"ID":"1aa39837-dbe0-4478-a514-b4afc328c705","Type":"ContainerStarted","Data":"9490e339e6e189dc258fb10d3e650c2a1c61725b44de2b732690662a629be1e5"} Oct 30 23:57:00.244853 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:00.244825 4927 generic.go:356] "Generic (PLEG): container finished" podID="1c6b629f-f032-469a-a06f-399b6a1a19c8" containerID="bc7456083fece25a72cbce4e4227c27fd540933b478ba3e6b5b25ba1e46aa2b2" exitCode=0 Oct 30 23:57:00.244974 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:00.244916 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c43c9a94-nxv5l" event={"ID":"1c6b629f-f032-469a-a06f-399b6a1a19c8","Type":"ContainerDied","Data":"bc7456083fece25a72cbce4e4227c27fd540933b478ba3e6b5b25ba1e46aa2b2"} Oct 30 23:57:00.365121 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:00.365061 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20c2fd60-6jhdm" podStartSLOduration=2.365046206 podStartE2EDuration="2.365046206s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:56:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:57:00.362765559 +0000 UTC m=+933.399096382" watchObservedRunningTime="2026-10-30 23:57:00.365046206 +0000 UTC m=+933.401377027" Oct 30 23:57:00.405027 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:00.405003 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5cc73ad-pf2kb" Oct 30 23:57:00.502213 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:00.502146 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/833799d2-5e63-4445-9e34-42bbb7e13359-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/833799d2-5e63-4445-9e34-42bbb7e13359-maas-api-openshift-ca\") pod \"833799d2-5e63-4445-9e34-42bbb7e13359\" (UID: \"833799d2-5e63-4445-9e34-42bbb7e13359\") " Oct 30 23:57:00.502360 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:00.502315 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/833799d2-5e63-4445-9e34-42bbb7e13359-kube-api-access-b7m2w\" (UniqueName: \"kubernetes.io/projected/833799d2-5e63-4445-9e34-42bbb7e13359-kube-api-access-b7m2w\") pod \"833799d2-5e63-4445-9e34-42bbb7e13359\" (UID: \"833799d2-5e63-4445-9e34-42bbb7e13359\") " Oct 30 23:57:00.502466 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:00.502448 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/833799d2-5e63-4445-9e34-42bbb7e13359-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/833799d2-5e63-4445-9e34-42bbb7e13359-maas-api-xks-ca\") pod \"833799d2-5e63-4445-9e34-42bbb7e13359\" (UID: \"833799d2-5e63-4445-9e34-42bbb7e13359\") " Oct 30 23:57:00.502516 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:00.502475 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/833799d2-5e63-4445-9e34-42bbb7e13359-maas-api-openshift-ca" pod "833799d2-5e63-4445-9e34-42bbb7e13359" (UID: "833799d2-5e63-4445-9e34-42bbb7e13359"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:57:00.502858 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:00.502837 4927 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/833799d2-5e63-4445-9e34-42bbb7e13359-maas-api-openshift-ca\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 23:57:00.504295 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:00.504274 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/833799d2-5e63-4445-9e34-42bbb7e13359-maas-api-xks-ca" pod "833799d2-5e63-4445-9e34-42bbb7e13359" (UID: "833799d2-5e63-4445-9e34-42bbb7e13359"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 23:57:00.504427 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:00.504348 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/833799d2-5e63-4445-9e34-42bbb7e13359-kube-api-access-b7m2w" pod "833799d2-5e63-4445-9e34-42bbb7e13359" (UID: "833799d2-5e63-4445-9e34-42bbb7e13359"). InnerVolumeSpecName "kube-api-access-b7m2w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:57:00.604270 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:00.604240 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b7m2w\" (UniqueName: \"kubernetes.io/projected/833799d2-5e63-4445-9e34-42bbb7e13359-kube-api-access-b7m2w\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 23:57:00.604270 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:00.604271 4927 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/833799d2-5e63-4445-9e34-42bbb7e13359-maas-api-xks-ca\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 23:57:01.253701 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:01.253667 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5cc73ad-pf2kb" event={"ID":"833799d2-5e63-4445-9e34-42bbb7e13359","Type":"ContainerDied","Data":"068ee39067536350c835fd96f67da87590a9493d45a5299508cd7be7cd01b88d"} Oct 30 23:57:01.254200 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:01.253729 4927 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="068ee39067536350c835fd96f67da87590a9493d45a5299508cd7be7cd01b88d" Oct 30 23:57:01.254200 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:01.253678 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5cc73ad-pf2kb" Oct 30 23:57:01.255478 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:01.255451 4927 generic.go:356] "Generic (PLEG): container finished" podID="1aa39837-dbe0-4478-a514-b4afc328c705" containerID="888de4f50d24dcad4afad7f9ff2a9763835d2b6dd87987c07328775148cc797b" exitCode=0 Oct 30 23:57:01.255625 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:01.255531 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20c2fd60-6jhdm" event={"ID":"1aa39837-dbe0-4478-a514-b4afc328c705","Type":"ContainerDied","Data":"888de4f50d24dcad4afad7f9ff2a9763835d2b6dd87987c07328775148cc797b"} Oct 30 23:57:01.401441 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:01.401418 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c43c9a94-nxv5l" Oct 30 23:57:01.513800 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:01.513767 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c6b629f-f032-469a-a06f-399b6a1a19c8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c6b629f-f032-469a-a06f-399b6a1a19c8-maas-api-openshift-ca\") pod \"1c6b629f-f032-469a-a06f-399b6a1a19c8\" (UID: \"1c6b629f-f032-469a-a06f-399b6a1a19c8\") " Oct 30 23:57:01.513800 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:01.513803 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c6b629f-f032-469a-a06f-399b6a1a19c8-kube-api-access-gqmdk\" (UniqueName: \"kubernetes.io/projected/1c6b629f-f032-469a-a06f-399b6a1a19c8-kube-api-access-gqmdk\") pod \"1c6b629f-f032-469a-a06f-399b6a1a19c8\" (UID: \"1c6b629f-f032-469a-a06f-399b6a1a19c8\") " Oct 30 23:57:01.514048 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:01.514013 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1c6b629f-f032-469a-a06f-399b6a1a19c8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c6b629f-f032-469a-a06f-399b6a1a19c8-maas-api-xks-ca\") pod \"1c6b629f-f032-469a-a06f-399b6a1a19c8\" (UID: \"1c6b629f-f032-469a-a06f-399b6a1a19c8\") " Oct 30 23:57:01.514129 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:01.514101 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c6b629f-f032-469a-a06f-399b6a1a19c8-maas-api-openshift-ca" pod "1c6b629f-f032-469a-a06f-399b6a1a19c8" (UID: "1c6b629f-f032-469a-a06f-399b6a1a19c8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:57:01.514413 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:01.514394 4927 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c6b629f-f032-469a-a06f-399b6a1a19c8-maas-api-openshift-ca\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 23:57:01.516148 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:01.516121 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c6b629f-f032-469a-a06f-399b6a1a19c8-kube-api-access-gqmdk" pod "1c6b629f-f032-469a-a06f-399b6a1a19c8" (UID: "1c6b629f-f032-469a-a06f-399b6a1a19c8"). InnerVolumeSpecName "kube-api-access-gqmdk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:57:01.516296 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:01.516280 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c6b629f-f032-469a-a06f-399b6a1a19c8-maas-api-xks-ca" pod "1c6b629f-f032-469a-a06f-399b6a1a19c8" (UID: "1c6b629f-f032-469a-a06f-399b6a1a19c8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 23:57:01.615492 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:01.615457 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gqmdk\" (UniqueName: \"kubernetes.io/projected/1c6b629f-f032-469a-a06f-399b6a1a19c8-kube-api-access-gqmdk\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 23:57:01.615492 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:01.615485 4927 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c6b629f-f032-469a-a06f-399b6a1a19c8-maas-api-xks-ca\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 23:57:02.268234 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:02.268198 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c43c9a94-nxv5l" Oct 30 23:57:02.268234 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:02.268202 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c43c9a94-nxv5l" event={"ID":"1c6b629f-f032-469a-a06f-399b6a1a19c8","Type":"ContainerDied","Data":"761c647c439332fd4ff995da2277eb6cfb67bc52f5447438d727038383241879"} Oct 30 23:57:02.268776 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:02.268298 4927 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="761c647c439332fd4ff995da2277eb6cfb67bc52f5447438d727038383241879" Oct 30 23:57:02.425957 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:02.425926 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20c2fd60-6jhdm" Oct 30 23:57:02.526110 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:02.526025 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1aa39837-dbe0-4478-a514-b4afc328c705-kube-api-access-hhxkg\" (UniqueName: \"kubernetes.io/projected/1aa39837-dbe0-4478-a514-b4afc328c705-kube-api-access-hhxkg\") pod \"1aa39837-dbe0-4478-a514-b4afc328c705\" (UID: \"1aa39837-dbe0-4478-a514-b4afc328c705\") " Oct 30 23:57:02.526272 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:02.526144 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1aa39837-dbe0-4478-a514-b4afc328c705-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1aa39837-dbe0-4478-a514-b4afc328c705-maas-api-openshift-ca\") pod \"1aa39837-dbe0-4478-a514-b4afc328c705\" (UID: \"1aa39837-dbe0-4478-a514-b4afc328c705\") " Oct 30 23:57:02.526272 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:02.526204 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1aa39837-dbe0-4478-a514-b4afc328c705-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1aa39837-dbe0-4478-a514-b4afc328c705-maas-api-xks-ca\") pod \"1aa39837-dbe0-4478-a514-b4afc328c705\" (UID: \"1aa39837-dbe0-4478-a514-b4afc328c705\") " Oct 30 23:57:02.526615 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:02.526581 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1aa39837-dbe0-4478-a514-b4afc328c705-maas-api-openshift-ca" pod "1aa39837-dbe0-4478-a514-b4afc328c705" (UID: "1aa39837-dbe0-4478-a514-b4afc328c705"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:57:02.528348 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:02.528303 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1aa39837-dbe0-4478-a514-b4afc328c705-maas-api-xks-ca" pod "1aa39837-dbe0-4478-a514-b4afc328c705" (UID: "1aa39837-dbe0-4478-a514-b4afc328c705"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 23:57:02.528577 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:02.528552 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1aa39837-dbe0-4478-a514-b4afc328c705-kube-api-access-hhxkg" pod "1aa39837-dbe0-4478-a514-b4afc328c705" (UID: "1aa39837-dbe0-4478-a514-b4afc328c705"). InnerVolumeSpecName "kube-api-access-hhxkg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:57:02.627930 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:02.627890 4927 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1aa39837-dbe0-4478-a514-b4afc328c705-maas-api-openshift-ca\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 23:57:02.627930 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:02.627923 4927 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1aa39837-dbe0-4478-a514-b4afc328c705-maas-api-xks-ca\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 23:57:02.627930 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:02.627935 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hhxkg\" (UniqueName: \"kubernetes.io/projected/1aa39837-dbe0-4478-a514-b4afc328c705-kube-api-access-hhxkg\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 23:57:03.276277 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:03.276242 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20c2fd60-6jhdm" event={"ID":"1aa39837-dbe0-4478-a514-b4afc328c705","Type":"ContainerDied","Data":"9490e339e6e189dc258fb10d3e650c2a1c61725b44de2b732690662a629be1e5"} Oct 30 23:57:03.276277 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:03.276274 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20c2fd60-6jhdm" Oct 30 23:57:03.276717 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:03.276288 4927 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9490e339e6e189dc258fb10d3e650c2a1c61725b44de2b732690662a629be1e5" Oct 30 23:57:04.195837 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:04.195803 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:57:06.516616 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:06.516579 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-56af7078-tvhbk"] Oct 30 23:57:06.517465 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:06.517440 4927 cpu_manager.go:414] "containerMap: removing container" podUID="833799d2-5e63-4445-9e34-42bbb7e13359" containerName="revoke-keys" Oct 30 23:57:06.517465 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:06.517465 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="833799d2-5e63-4445-9e34-42bbb7e13359" containerName="revoke-keys" Oct 30 23:57:06.517661 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:06.517482 4927 cpu_manager.go:414] "containerMap: removing container" podUID="1c6b629f-f032-469a-a06f-399b6a1a19c8" containerName="revoke-keys" Oct 30 23:57:06.517661 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:06.517491 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c6b629f-f032-469a-a06f-399b6a1a19c8" containerName="revoke-keys" Oct 30 23:57:06.517661 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:06.517514 4927 cpu_manager.go:414] "containerMap: removing container" podUID="1aa39837-dbe0-4478-a514-b4afc328c705" containerName="revoke-keys" Oct 30 23:57:06.517661 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:06.517522 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1aa39837-dbe0-4478-a514-b4afc328c705" containerName="revoke-keys" Oct 30 23:57:06.517661 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:06.517650 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="833799d2-5e63-4445-9e34-42bbb7e13359" containerName="revoke-keys" Oct 30 23:57:06.517661 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:06.517660 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c6b629f-f032-469a-a06f-399b6a1a19c8" containerName="revoke-keys" Oct 30 23:57:06.517979 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:06.517671 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="1aa39837-dbe0-4478-a514-b4afc328c705" containerName="revoke-keys" Oct 30 23:57:06.523320 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:06.523285 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56af7078-tvhbk" Oct 30 23:57:06.576877 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:06.576840 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 23:57:06.639698 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:06.636437 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4fdzs\"" Oct 30 23:57:06.642583 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:06.642550 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-56af7078-tvhbk"] Oct 30 23:57:06.700691 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:06.700609 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1cb89f94-8f2a-4cd7-b1da-509f7d5f883b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-56af7078-tvhbk\" (UID: \"1cb89f94-8f2a-4cd7-b1da-509f7d5f883b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56af7078-tvhbk" Oct 30 23:57:06.700691 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:06.700645 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5t7ck\" (UniqueName: \"kubernetes.io/projected/1cb89f94-8f2a-4cd7-b1da-509f7d5f883b-kube-api-access-5t7ck\") pod \"maas-api-revoke-sub-56af7078-tvhbk\" (UID: \"1cb89f94-8f2a-4cd7-b1da-509f7d5f883b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56af7078-tvhbk" Oct 30 23:57:06.700891 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:06.700782 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1cb89f94-8f2a-4cd7-b1da-509f7d5f883b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-56af7078-tvhbk\" (UID: \"1cb89f94-8f2a-4cd7-b1da-509f7d5f883b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56af7078-tvhbk" Oct 30 23:57:06.801455 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:06.801365 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1cb89f94-8f2a-4cd7-b1da-509f7d5f883b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-56af7078-tvhbk\" (UID: \"1cb89f94-8f2a-4cd7-b1da-509f7d5f883b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56af7078-tvhbk" Oct 30 23:57:06.801455 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:06.801416 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5t7ck\" (UniqueName: \"kubernetes.io/projected/1cb89f94-8f2a-4cd7-b1da-509f7d5f883b-kube-api-access-5t7ck\") pod \"maas-api-revoke-sub-56af7078-tvhbk\" (UID: \"1cb89f94-8f2a-4cd7-b1da-509f7d5f883b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56af7078-tvhbk" Oct 30 23:57:06.801649 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:06.801494 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1cb89f94-8f2a-4cd7-b1da-509f7d5f883b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-56af7078-tvhbk\" (UID: \"1cb89f94-8f2a-4cd7-b1da-509f7d5f883b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56af7078-tvhbk" Oct 30 23:57:06.802089 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:06.802066 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1cb89f94-8f2a-4cd7-b1da-509f7d5f883b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-56af7078-tvhbk\" (UID: \"1cb89f94-8f2a-4cd7-b1da-509f7d5f883b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56af7078-tvhbk" Oct 30 23:57:06.803805 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:06.803784 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1cb89f94-8f2a-4cd7-b1da-509f7d5f883b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-56af7078-tvhbk\" (UID: \"1cb89f94-8f2a-4cd7-b1da-509f7d5f883b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56af7078-tvhbk" Oct 30 23:57:06.864865 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:06.864822 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5t7ck\" (UniqueName: \"kubernetes.io/projected/1cb89f94-8f2a-4cd7-b1da-509f7d5f883b-kube-api-access-5t7ck\") pod \"maas-api-revoke-sub-56af7078-tvhbk\" (UID: \"1cb89f94-8f2a-4cd7-b1da-509f7d5f883b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56af7078-tvhbk" Oct 30 23:57:07.140059 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:07.139975 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56af7078-tvhbk" Oct 30 23:57:07.528088 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:07.528055 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-56af7078-tvhbk"] Oct 30 23:57:07.528802 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:57:07.528767 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1cb89f94_8f2a_4cd7_b1da_509f7d5f883b.slice/crio-043991974bd827ee9f9e495342e15ef5b9ccb4aceb9df442d3f10cbfa3051a74 WatchSource:0}: Error finding container 043991974bd827ee9f9e495342e15ef5b9ccb4aceb9df442d3f10cbfa3051a74: Status 404 returned error can't find the container with id 043991974bd827ee9f9e495342e15ef5b9ccb4aceb9df442d3f10cbfa3051a74 Oct 30 23:57:08.345533 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:08.345495 4927 generic.go:356] "Generic (PLEG): container finished" podID="1cb89f94-8f2a-4cd7-b1da-509f7d5f883b" containerID="6dd79938b7269d8bd8ed7db80315d358fcef08e7223fec10844476c8687f3e58" exitCode=0 Oct 30 23:57:08.345676 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:08.345568 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56af7078-tvhbk" event={"ID":"1cb89f94-8f2a-4cd7-b1da-509f7d5f883b","Type":"ContainerDied","Data":"6dd79938b7269d8bd8ed7db80315d358fcef08e7223fec10844476c8687f3e58"} Oct 30 23:57:08.345676 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:08.345618 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56af7078-tvhbk" event={"ID":"1cb89f94-8f2a-4cd7-b1da-509f7d5f883b","Type":"ContainerStarted","Data":"043991974bd827ee9f9e495342e15ef5b9ccb4aceb9df442d3f10cbfa3051a74"} Oct 30 23:57:08.811971 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:08.811933 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:57:09.517091 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:09.517058 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56af7078-tvhbk" Oct 30 23:57:09.524529 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:09.524501 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1cb89f94-8f2a-4cd7-b1da-509f7d5f883b-kube-api-access-5t7ck\" (UniqueName: \"kubernetes.io/projected/1cb89f94-8f2a-4cd7-b1da-509f7d5f883b-kube-api-access-5t7ck\") pod \"1cb89f94-8f2a-4cd7-b1da-509f7d5f883b\" (UID: \"1cb89f94-8f2a-4cd7-b1da-509f7d5f883b\") " Oct 30 23:57:09.524641 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:09.524613 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1cb89f94-8f2a-4cd7-b1da-509f7d5f883b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1cb89f94-8f2a-4cd7-b1da-509f7d5f883b-maas-api-xks-ca\") pod \"1cb89f94-8f2a-4cd7-b1da-509f7d5f883b\" (UID: \"1cb89f94-8f2a-4cd7-b1da-509f7d5f883b\") " Oct 30 23:57:09.524685 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:09.524657 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1cb89f94-8f2a-4cd7-b1da-509f7d5f883b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1cb89f94-8f2a-4cd7-b1da-509f7d5f883b-maas-api-openshift-ca\") pod \"1cb89f94-8f2a-4cd7-b1da-509f7d5f883b\" (UID: \"1cb89f94-8f2a-4cd7-b1da-509f7d5f883b\") " Oct 30 23:57:09.525055 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:09.525028 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1cb89f94-8f2a-4cd7-b1da-509f7d5f883b-maas-api-openshift-ca" pod "1cb89f94-8f2a-4cd7-b1da-509f7d5f883b" (UID: "1cb89f94-8f2a-4cd7-b1da-509f7d5f883b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:57:09.527025 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:09.526996 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1cb89f94-8f2a-4cd7-b1da-509f7d5f883b-maas-api-xks-ca" pod "1cb89f94-8f2a-4cd7-b1da-509f7d5f883b" (UID: "1cb89f94-8f2a-4cd7-b1da-509f7d5f883b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 23:57:09.527025 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:09.527010 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1cb89f94-8f2a-4cd7-b1da-509f7d5f883b-kube-api-access-5t7ck" pod "1cb89f94-8f2a-4cd7-b1da-509f7d5f883b" (UID: "1cb89f94-8f2a-4cd7-b1da-509f7d5f883b"). InnerVolumeSpecName "kube-api-access-5t7ck". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:57:09.626160 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:09.626073 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5t7ck\" (UniqueName: \"kubernetes.io/projected/1cb89f94-8f2a-4cd7-b1da-509f7d5f883b-kube-api-access-5t7ck\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 23:57:09.626160 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:09.626113 4927 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1cb89f94-8f2a-4cd7-b1da-509f7d5f883b-maas-api-xks-ca\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 23:57:09.626160 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:09.626129 4927 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1cb89f94-8f2a-4cd7-b1da-509f7d5f883b-maas-api-openshift-ca\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 23:57:10.360856 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:10.360806 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56af7078-tvhbk" event={"ID":"1cb89f94-8f2a-4cd7-b1da-509f7d5f883b","Type":"ContainerDied","Data":"043991974bd827ee9f9e495342e15ef5b9ccb4aceb9df442d3f10cbfa3051a74"} Oct 30 23:57:10.360856 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:10.360841 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56af7078-tvhbk" Oct 30 23:57:10.361446 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:10.360865 4927 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="043991974bd827ee9f9e495342e15ef5b9ccb4aceb9df442d3f10cbfa3051a74" Oct 30 23:57:13.324029 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:13.323987 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:57:14.142747 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:14.142709 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-50b468fd-749rv"] Oct 30 23:57:14.143626 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:14.143603 4927 cpu_manager.go:414] "containerMap: removing container" podUID="1cb89f94-8f2a-4cd7-b1da-509f7d5f883b" containerName="revoke-keys" Oct 30 23:57:14.143704 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:14.143630 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1cb89f94-8f2a-4cd7-b1da-509f7d5f883b" containerName="revoke-keys" Oct 30 23:57:14.143803 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:14.143790 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="1cb89f94-8f2a-4cd7-b1da-509f7d5f883b" containerName="revoke-keys" Oct 30 23:57:14.147717 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:14.147693 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50b468fd-749rv" Oct 30 23:57:14.178809 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:14.178771 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15ba1fff-a83b-41e6-9e54-3ce2d7417014-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-50b468fd-749rv\" (UID: \"15ba1fff-a83b-41e6-9e54-3ce2d7417014\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50b468fd-749rv" Oct 30 23:57:14.178981 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:14.178879 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15ba1fff-a83b-41e6-9e54-3ce2d7417014-maas-api-xks-ca\") pod \"maas-api-revoke-sub-50b468fd-749rv\" (UID: \"15ba1fff-a83b-41e6-9e54-3ce2d7417014\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50b468fd-749rv" Oct 30 23:57:14.179056 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:14.179037 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ndjx4\" (UniqueName: \"kubernetes.io/projected/15ba1fff-a83b-41e6-9e54-3ce2d7417014-kube-api-access-ndjx4\") pod \"maas-api-revoke-sub-50b468fd-749rv\" (UID: \"15ba1fff-a83b-41e6-9e54-3ce2d7417014\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50b468fd-749rv" Oct 30 23:57:14.201682 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:14.201647 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-50b468fd-749rv"] Oct 30 23:57:14.203384 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:14.203359 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 23:57:14.247521 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:14.247480 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4fdzs\"" Oct 30 23:57:14.280283 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:14.280234 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndjx4\" (UniqueName: \"kubernetes.io/projected/15ba1fff-a83b-41e6-9e54-3ce2d7417014-kube-api-access-ndjx4\") pod \"maas-api-revoke-sub-50b468fd-749rv\" (UID: \"15ba1fff-a83b-41e6-9e54-3ce2d7417014\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50b468fd-749rv" Oct 30 23:57:14.280488 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:14.280359 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15ba1fff-a83b-41e6-9e54-3ce2d7417014-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-50b468fd-749rv\" (UID: \"15ba1fff-a83b-41e6-9e54-3ce2d7417014\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50b468fd-749rv" Oct 30 23:57:14.280488 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:14.280446 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15ba1fff-a83b-41e6-9e54-3ce2d7417014-maas-api-xks-ca\") pod \"maas-api-revoke-sub-50b468fd-749rv\" (UID: \"15ba1fff-a83b-41e6-9e54-3ce2d7417014\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50b468fd-749rv" Oct 30 23:57:14.281260 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:14.281232 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15ba1fff-a83b-41e6-9e54-3ce2d7417014-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-50b468fd-749rv\" (UID: \"15ba1fff-a83b-41e6-9e54-3ce2d7417014\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50b468fd-749rv" Oct 30 23:57:14.282973 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:14.282942 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15ba1fff-a83b-41e6-9e54-3ce2d7417014-maas-api-xks-ca\") pod \"maas-api-revoke-sub-50b468fd-749rv\" (UID: \"15ba1fff-a83b-41e6-9e54-3ce2d7417014\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50b468fd-749rv" Oct 30 23:57:14.343043 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:14.343011 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ndjx4\" (UniqueName: \"kubernetes.io/projected/15ba1fff-a83b-41e6-9e54-3ce2d7417014-kube-api-access-ndjx4\") pod \"maas-api-revoke-sub-50b468fd-749rv\" (UID: \"15ba1fff-a83b-41e6-9e54-3ce2d7417014\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50b468fd-749rv" Oct 30 23:57:14.463055 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:14.463014 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50b468fd-749rv" Oct 30 23:57:14.777159 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:14.777071 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-afa95927-2tlqf"] Oct 30 23:57:14.782200 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:14.782175 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-afa95927-2tlqf" Oct 30 23:57:14.840346 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:14.840294 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-afa95927-2tlqf"] Oct 30 23:57:14.887970 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:14.887313 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22c6669d-27fa-4841-a900-d4f70fbb9f2c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-afa95927-2tlqf\" (UID: \"22c6669d-27fa-4841-a900-d4f70fbb9f2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afa95927-2tlqf" Oct 30 23:57:14.887970 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:14.887383 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22c6669d-27fa-4841-a900-d4f70fbb9f2c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-afa95927-2tlqf\" (UID: \"22c6669d-27fa-4841-a900-d4f70fbb9f2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afa95927-2tlqf" Oct 30 23:57:14.887970 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:14.887436 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pg6r6\" (UniqueName: \"kubernetes.io/projected/22c6669d-27fa-4841-a900-d4f70fbb9f2c-kube-api-access-pg6r6\") pod \"maas-api-revoke-sub-afa95927-2tlqf\" (UID: \"22c6669d-27fa-4841-a900-d4f70fbb9f2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afa95927-2tlqf" Oct 30 23:57:14.893380 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:14.893349 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-50b468fd-749rv"] Oct 30 23:57:14.895391 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:57:14.895359 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15ba1fff_a83b_41e6_9e54_3ce2d7417014.slice/crio-5e9dbfc487d88f8bba60d6e0a48faf26e8a326993e6dc402a0a5c8b7e16cf1bb WatchSource:0}: Error finding container 5e9dbfc487d88f8bba60d6e0a48faf26e8a326993e6dc402a0a5c8b7e16cf1bb: Status 404 returned error can't find the container with id 5e9dbfc487d88f8bba60d6e0a48faf26e8a326993e6dc402a0a5c8b7e16cf1bb Oct 30 23:57:14.988813 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:14.988780 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22c6669d-27fa-4841-a900-d4f70fbb9f2c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-afa95927-2tlqf\" (UID: \"22c6669d-27fa-4841-a900-d4f70fbb9f2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afa95927-2tlqf" Oct 30 23:57:14.988813 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:14.988817 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22c6669d-27fa-4841-a900-d4f70fbb9f2c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-afa95927-2tlqf\" (UID: \"22c6669d-27fa-4841-a900-d4f70fbb9f2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afa95927-2tlqf" Oct 30 23:57:14.989062 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:14.988943 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pg6r6\" (UniqueName: \"kubernetes.io/projected/22c6669d-27fa-4841-a900-d4f70fbb9f2c-kube-api-access-pg6r6\") pod \"maas-api-revoke-sub-afa95927-2tlqf\" (UID: \"22c6669d-27fa-4841-a900-d4f70fbb9f2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afa95927-2tlqf" Oct 30 23:57:14.989524 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:14.989493 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22c6669d-27fa-4841-a900-d4f70fbb9f2c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-afa95927-2tlqf\" (UID: \"22c6669d-27fa-4841-a900-d4f70fbb9f2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afa95927-2tlqf" Oct 30 23:57:14.991153 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:14.991135 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22c6669d-27fa-4841-a900-d4f70fbb9f2c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-afa95927-2tlqf\" (UID: \"22c6669d-27fa-4841-a900-d4f70fbb9f2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afa95927-2tlqf" Oct 30 23:57:15.051445 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:15.051372 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pg6r6\" (UniqueName: \"kubernetes.io/projected/22c6669d-27fa-4841-a900-d4f70fbb9f2c-kube-api-access-pg6r6\") pod \"maas-api-revoke-sub-afa95927-2tlqf\" (UID: \"22c6669d-27fa-4841-a900-d4f70fbb9f2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afa95927-2tlqf" Oct 30 23:57:15.096532 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:15.096501 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-afa95927-2tlqf" Oct 30 23:57:15.419205 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:15.419166 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50b468fd-749rv" event={"ID":"15ba1fff-a83b-41e6-9e54-3ce2d7417014","Type":"ContainerStarted","Data":"4b3ed1ea4226b920b263173e2ecf9695600e86c822d53648628e23647825caab"} Oct 30 23:57:15.420045 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:15.419225 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50b468fd-749rv" event={"ID":"15ba1fff-a83b-41e6-9e54-3ce2d7417014","Type":"ContainerStarted","Data":"5e9dbfc487d88f8bba60d6e0a48faf26e8a326993e6dc402a0a5c8b7e16cf1bb"} Oct 30 23:57:15.486669 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:15.486641 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-afa95927-2tlqf"] Oct 30 23:57:15.487374 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:57:15.487343 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22c6669d_27fa_4841_a900_d4f70fbb9f2c.slice/crio-f0e5b3e22b64b211522900afbd0fad7bb824fcda583b5f21c7271b8bc31bddac WatchSource:0}: Error finding container f0e5b3e22b64b211522900afbd0fad7bb824fcda583b5f21c7271b8bc31bddac: Status 404 returned error can't find the container with id f0e5b3e22b64b211522900afbd0fad7bb824fcda583b5f21c7271b8bc31bddac Oct 30 23:57:15.538268 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:15.538211 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50b468fd-749rv" podStartSLOduration=1.538192539 podStartE2EDuration="1.538192539s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:57:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:57:15.537059003 +0000 UTC m=+948.573389826" watchObservedRunningTime="2026-10-30 23:57:15.538192539 +0000 UTC m=+948.574523361" Oct 30 23:57:16.428344 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:16.428291 4927 generic.go:356] "Generic (PLEG): container finished" podID="22c6669d-27fa-4841-a900-d4f70fbb9f2c" containerID="beb361bcaf638007b3141d2c5af0c4969c65f6cc9e687f0327387847362fdece" exitCode=0 Oct 30 23:57:16.428801 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:16.428370 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-afa95927-2tlqf" event={"ID":"22c6669d-27fa-4841-a900-d4f70fbb9f2c","Type":"ContainerDied","Data":"beb361bcaf638007b3141d2c5af0c4969c65f6cc9e687f0327387847362fdece"} Oct 30 23:57:16.428801 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:16.428436 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-afa95927-2tlqf" event={"ID":"22c6669d-27fa-4841-a900-d4f70fbb9f2c","Type":"ContainerStarted","Data":"f0e5b3e22b64b211522900afbd0fad7bb824fcda583b5f21c7271b8bc31bddac"} Oct 30 23:57:16.430828 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:16.430802 4927 generic.go:356] "Generic (PLEG): container finished" podID="15ba1fff-a83b-41e6-9e54-3ce2d7417014" containerID="4b3ed1ea4226b920b263173e2ecf9695600e86c822d53648628e23647825caab" exitCode=0 Oct 30 23:57:16.430932 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:16.430856 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50b468fd-749rv" event={"ID":"15ba1fff-a83b-41e6-9e54-3ce2d7417014","Type":"ContainerDied","Data":"4b3ed1ea4226b920b263173e2ecf9695600e86c822d53648628e23647825caab"} Oct 30 23:57:17.614558 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:17.614526 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50b468fd-749rv" Oct 30 23:57:17.661409 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:17.661386 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-afa95927-2tlqf" Oct 30 23:57:17.726591 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:17.726495 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22c6669d-27fa-4841-a900-d4f70fbb9f2c-kube-api-access-pg6r6\" (UniqueName: \"kubernetes.io/projected/22c6669d-27fa-4841-a900-d4f70fbb9f2c-kube-api-access-pg6r6\") pod \"22c6669d-27fa-4841-a900-d4f70fbb9f2c\" (UID: \"22c6669d-27fa-4841-a900-d4f70fbb9f2c\") " Oct 30 23:57:17.726769 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:17.726596 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/15ba1fff-a83b-41e6-9e54-3ce2d7417014-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15ba1fff-a83b-41e6-9e54-3ce2d7417014-maas-api-openshift-ca\") pod \"15ba1fff-a83b-41e6-9e54-3ce2d7417014\" (UID: \"15ba1fff-a83b-41e6-9e54-3ce2d7417014\") " Oct 30 23:57:17.726769 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:17.726716 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15ba1fff-a83b-41e6-9e54-3ce2d7417014-kube-api-access-ndjx4\" (UniqueName: \"kubernetes.io/projected/15ba1fff-a83b-41e6-9e54-3ce2d7417014-kube-api-access-ndjx4\") pod \"15ba1fff-a83b-41e6-9e54-3ce2d7417014\" (UID: \"15ba1fff-a83b-41e6-9e54-3ce2d7417014\") " Oct 30 23:57:17.726769 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:17.726754 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/22c6669d-27fa-4841-a900-d4f70fbb9f2c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22c6669d-27fa-4841-a900-d4f70fbb9f2c-maas-api-xks-ca\") pod \"22c6669d-27fa-4841-a900-d4f70fbb9f2c\" (UID: \"22c6669d-27fa-4841-a900-d4f70fbb9f2c\") " Oct 30 23:57:17.726946 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:17.726825 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/22c6669d-27fa-4841-a900-d4f70fbb9f2c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22c6669d-27fa-4841-a900-d4f70fbb9f2c-maas-api-openshift-ca\") pod \"22c6669d-27fa-4841-a900-d4f70fbb9f2c\" (UID: \"22c6669d-27fa-4841-a900-d4f70fbb9f2c\") " Oct 30 23:57:17.726946 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:17.726870 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/15ba1fff-a83b-41e6-9e54-3ce2d7417014-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15ba1fff-a83b-41e6-9e54-3ce2d7417014-maas-api-xks-ca\") pod \"15ba1fff-a83b-41e6-9e54-3ce2d7417014\" (UID: \"15ba1fff-a83b-41e6-9e54-3ce2d7417014\") " Oct 30 23:57:17.727093 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:17.727026 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15ba1fff-a83b-41e6-9e54-3ce2d7417014-maas-api-openshift-ca" pod "15ba1fff-a83b-41e6-9e54-3ce2d7417014" (UID: "15ba1fff-a83b-41e6-9e54-3ce2d7417014"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:57:17.727385 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:17.727360 4927 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15ba1fff-a83b-41e6-9e54-3ce2d7417014-maas-api-openshift-ca\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 23:57:17.727536 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:17.727431 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22c6669d-27fa-4841-a900-d4f70fbb9f2c-maas-api-openshift-ca" pod "22c6669d-27fa-4841-a900-d4f70fbb9f2c" (UID: "22c6669d-27fa-4841-a900-d4f70fbb9f2c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:57:17.729736 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:17.729710 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15ba1fff-a83b-41e6-9e54-3ce2d7417014-kube-api-access-ndjx4" pod "15ba1fff-a83b-41e6-9e54-3ce2d7417014" (UID: "15ba1fff-a83b-41e6-9e54-3ce2d7417014"). InnerVolumeSpecName "kube-api-access-ndjx4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:57:17.729990 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:17.729957 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/15ba1fff-a83b-41e6-9e54-3ce2d7417014-maas-api-xks-ca" pod "15ba1fff-a83b-41e6-9e54-3ce2d7417014" (UID: "15ba1fff-a83b-41e6-9e54-3ce2d7417014"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 23:57:17.729990 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:17.729949 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/22c6669d-27fa-4841-a900-d4f70fbb9f2c-maas-api-xks-ca" pod "22c6669d-27fa-4841-a900-d4f70fbb9f2c" (UID: "22c6669d-27fa-4841-a900-d4f70fbb9f2c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 23:57:17.729990 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:17.729958 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22c6669d-27fa-4841-a900-d4f70fbb9f2c-kube-api-access-pg6r6" pod "22c6669d-27fa-4841-a900-d4f70fbb9f2c" (UID: "22c6669d-27fa-4841-a900-d4f70fbb9f2c"). InnerVolumeSpecName "kube-api-access-pg6r6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:57:17.829278 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:17.829235 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ndjx4\" (UniqueName: \"kubernetes.io/projected/15ba1fff-a83b-41e6-9e54-3ce2d7417014-kube-api-access-ndjx4\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 23:57:17.829552 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:17.829521 4927 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22c6669d-27fa-4841-a900-d4f70fbb9f2c-maas-api-xks-ca\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 23:57:17.829552 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:17.829549 4927 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22c6669d-27fa-4841-a900-d4f70fbb9f2c-maas-api-openshift-ca\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 23:57:17.829753 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:17.829578 4927 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15ba1fff-a83b-41e6-9e54-3ce2d7417014-maas-api-xks-ca\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 23:57:17.829753 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:17.829593 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pg6r6\" (UniqueName: \"kubernetes.io/projected/22c6669d-27fa-4841-a900-d4f70fbb9f2c-kube-api-access-pg6r6\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 23:57:17.981670 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:17.981589 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:57:18.447365 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:18.447308 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-afa95927-2tlqf" Oct 30 23:57:18.447365 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:18.447348 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-afa95927-2tlqf" event={"ID":"22c6669d-27fa-4841-a900-d4f70fbb9f2c","Type":"ContainerDied","Data":"f0e5b3e22b64b211522900afbd0fad7bb824fcda583b5f21c7271b8bc31bddac"} Oct 30 23:57:18.447628 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:18.447399 4927 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f0e5b3e22b64b211522900afbd0fad7bb824fcda583b5f21c7271b8bc31bddac" Oct 30 23:57:18.448677 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:18.448654 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50b468fd-749rv" event={"ID":"15ba1fff-a83b-41e6-9e54-3ce2d7417014","Type":"ContainerDied","Data":"5e9dbfc487d88f8bba60d6e0a48faf26e8a326993e6dc402a0a5c8b7e16cf1bb"} Oct 30 23:57:18.448800 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:18.448688 4927 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5e9dbfc487d88f8bba60d6e0a48faf26e8a326993e6dc402a0a5c8b7e16cf1bb" Oct 30 23:57:18.448800 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:18.448716 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50b468fd-749rv" Oct 30 23:57:19.677872 ip-10-0-141-180 kubenswrapper[4927]: E1030 23:57:19.677837 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sqv69" containerName="extractor" Oct 30 23:57:21.165651 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:21.165613 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:57:22.401237 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:22.401206 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:57:22.596139 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:22.596093 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:57:22.939568 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:22.939532 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b9c40b0e-8lmzb"] Oct 30 23:57:22.940345 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:22.940311 4927 cpu_manager.go:414] "containerMap: removing container" podUID="15ba1fff-a83b-41e6-9e54-3ce2d7417014" containerName="revoke-keys" Oct 30 23:57:22.940449 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:22.940349 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15ba1fff-a83b-41e6-9e54-3ce2d7417014" containerName="revoke-keys" Oct 30 23:57:22.940449 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:22.940412 4927 cpu_manager.go:414] "containerMap: removing container" podUID="22c6669d-27fa-4841-a900-d4f70fbb9f2c" containerName="revoke-keys" Oct 30 23:57:22.940449 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:22.940420 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22c6669d-27fa-4841-a900-d4f70fbb9f2c" containerName="revoke-keys" Oct 30 23:57:22.940561 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:22.940549 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="22c6669d-27fa-4841-a900-d4f70fbb9f2c" containerName="revoke-keys" Oct 30 23:57:22.940596 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:22.940568 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="15ba1fff-a83b-41e6-9e54-3ce2d7417014" containerName="revoke-keys" Oct 30 23:57:22.944950 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:22.944933 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9c40b0e-8lmzb" Oct 30 23:57:22.995911 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:22.995879 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4spw9\" (UniqueName: \"kubernetes.io/projected/6f4e75ff-6c3e-4d35-a73c-4b11c2ec2b7d-kube-api-access-4spw9\") pod \"maas-api-revoke-sub-b9c40b0e-8lmzb\" (UID: \"6f4e75ff-6c3e-4d35-a73c-4b11c2ec2b7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9c40b0e-8lmzb" Oct 30 23:57:22.995911 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:22.995941 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f4e75ff-6c3e-4d35-a73c-4b11c2ec2b7d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b9c40b0e-8lmzb\" (UID: \"6f4e75ff-6c3e-4d35-a73c-4b11c2ec2b7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9c40b0e-8lmzb" Oct 30 23:57:22.996246 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:22.996050 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f4e75ff-6c3e-4d35-a73c-4b11c2ec2b7d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b9c40b0e-8lmzb\" (UID: \"6f4e75ff-6c3e-4d35-a73c-4b11c2ec2b7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9c40b0e-8lmzb" Oct 30 23:57:23.002060 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:23.002030 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b9c40b0e-8lmzb"] Oct 30 23:57:23.047759 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:23.047716 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 23:57:23.047952 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:23.047823 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4fdzs\"" Oct 30 23:57:23.097161 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:23.097123 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4spw9\" (UniqueName: \"kubernetes.io/projected/6f4e75ff-6c3e-4d35-a73c-4b11c2ec2b7d-kube-api-access-4spw9\") pod \"maas-api-revoke-sub-b9c40b0e-8lmzb\" (UID: \"6f4e75ff-6c3e-4d35-a73c-4b11c2ec2b7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9c40b0e-8lmzb" Oct 30 23:57:23.097392 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:23.097184 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f4e75ff-6c3e-4d35-a73c-4b11c2ec2b7d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b9c40b0e-8lmzb\" (UID: \"6f4e75ff-6c3e-4d35-a73c-4b11c2ec2b7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9c40b0e-8lmzb" Oct 30 23:57:23.097392 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:23.097235 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f4e75ff-6c3e-4d35-a73c-4b11c2ec2b7d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b9c40b0e-8lmzb\" (UID: \"6f4e75ff-6c3e-4d35-a73c-4b11c2ec2b7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9c40b0e-8lmzb" Oct 30 23:57:23.097872 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:23.097849 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f4e75ff-6c3e-4d35-a73c-4b11c2ec2b7d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b9c40b0e-8lmzb\" (UID: \"6f4e75ff-6c3e-4d35-a73c-4b11c2ec2b7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9c40b0e-8lmzb" Oct 30 23:57:23.099709 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:23.099683 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f4e75ff-6c3e-4d35-a73c-4b11c2ec2b7d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b9c40b0e-8lmzb\" (UID: \"6f4e75ff-6c3e-4d35-a73c-4b11c2ec2b7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9c40b0e-8lmzb" Oct 30 23:57:23.159852 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:23.159814 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4spw9\" (UniqueName: \"kubernetes.io/projected/6f4e75ff-6c3e-4d35-a73c-4b11c2ec2b7d-kube-api-access-4spw9\") pod \"maas-api-revoke-sub-b9c40b0e-8lmzb\" (UID: \"6f4e75ff-6c3e-4d35-a73c-4b11c2ec2b7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9c40b0e-8lmzb" Oct 30 23:57:23.260168 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:23.260090 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9c40b0e-8lmzb" Oct 30 23:57:23.646622 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:23.646587 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b9c40b0e-8lmzb"] Oct 30 23:57:23.647234 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:57:23.647209 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f4e75ff_6c3e_4d35_a73c_4b11c2ec2b7d.slice/crio-c7467cc6b712545f6a5638809d78f2096422d5b767248884a65e7edbf5d491d7 WatchSource:0}: Error finding container c7467cc6b712545f6a5638809d78f2096422d5b767248884a65e7edbf5d491d7: Status 404 returned error can't find the container with id c7467cc6b712545f6a5638809d78f2096422d5b767248884a65e7edbf5d491d7 Oct 30 23:57:24.490430 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:24.490396 4927 generic.go:356] "Generic (PLEG): container finished" podID="6f4e75ff-6c3e-4d35-a73c-4b11c2ec2b7d" containerID="4e0f6edfeb5185a6651f45cd23c9c0058c57e87df15da1cf5972af1226159634" exitCode=0 Oct 30 23:57:24.490589 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:24.490475 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9c40b0e-8lmzb" event={"ID":"6f4e75ff-6c3e-4d35-a73c-4b11c2ec2b7d","Type":"ContainerDied","Data":"4e0f6edfeb5185a6651f45cd23c9c0058c57e87df15da1cf5972af1226159634"} Oct 30 23:57:24.490589 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:24.490530 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9c40b0e-8lmzb" event={"ID":"6f4e75ff-6c3e-4d35-a73c-4b11c2ec2b7d","Type":"ContainerStarted","Data":"c7467cc6b712545f6a5638809d78f2096422d5b767248884a65e7edbf5d491d7"} Oct 30 23:57:25.548992 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:25.548954 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c802f331-n5s86"] Oct 30 23:57:25.553603 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:25.553578 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c802f331-n5s86" Oct 30 23:57:25.613859 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:25.613819 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c802f331-n5s86"] Oct 30 23:57:25.623140 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:25.622881 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d617fcff-5199-415f-9d97-2185aae73433-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c802f331-n5s86\" (UID: \"d617fcff-5199-415f-9d97-2185aae73433\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c802f331-n5s86" Oct 30 23:57:25.623140 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:25.622986 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4zdqg\" (UniqueName: \"kubernetes.io/projected/d617fcff-5199-415f-9d97-2185aae73433-kube-api-access-4zdqg\") pod \"maas-api-revoke-sub-c802f331-n5s86\" (UID: \"d617fcff-5199-415f-9d97-2185aae73433\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c802f331-n5s86" Oct 30 23:57:25.623140 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:25.623040 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d617fcff-5199-415f-9d97-2185aae73433-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c802f331-n5s86\" (UID: \"d617fcff-5199-415f-9d97-2185aae73433\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c802f331-n5s86" Oct 30 23:57:25.680393 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:25.680367 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9c40b0e-8lmzb" Oct 30 23:57:25.724464 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:25.724426 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zdqg\" (UniqueName: \"kubernetes.io/projected/d617fcff-5199-415f-9d97-2185aae73433-kube-api-access-4zdqg\") pod \"maas-api-revoke-sub-c802f331-n5s86\" (UID: \"d617fcff-5199-415f-9d97-2185aae73433\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c802f331-n5s86" Oct 30 23:57:25.724646 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:25.724477 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d617fcff-5199-415f-9d97-2185aae73433-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c802f331-n5s86\" (UID: \"d617fcff-5199-415f-9d97-2185aae73433\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c802f331-n5s86" Oct 30 23:57:25.724646 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:25.724557 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d617fcff-5199-415f-9d97-2185aae73433-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c802f331-n5s86\" (UID: \"d617fcff-5199-415f-9d97-2185aae73433\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c802f331-n5s86" Oct 30 23:57:25.725295 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:25.725267 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d617fcff-5199-415f-9d97-2185aae73433-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c802f331-n5s86\" (UID: \"d617fcff-5199-415f-9d97-2185aae73433\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c802f331-n5s86" Oct 30 23:57:25.726977 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:25.726954 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d617fcff-5199-415f-9d97-2185aae73433-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c802f331-n5s86\" (UID: \"d617fcff-5199-415f-9d97-2185aae73433\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c802f331-n5s86" Oct 30 23:57:25.787163 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:25.787085 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zdqg\" (UniqueName: \"kubernetes.io/projected/d617fcff-5199-415f-9d97-2185aae73433-kube-api-access-4zdqg\") pod \"maas-api-revoke-sub-c802f331-n5s86\" (UID: \"d617fcff-5199-415f-9d97-2185aae73433\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c802f331-n5s86" Oct 30 23:57:25.825549 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:25.825510 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f4e75ff-6c3e-4d35-a73c-4b11c2ec2b7d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f4e75ff-6c3e-4d35-a73c-4b11c2ec2b7d-maas-api-openshift-ca\") pod \"6f4e75ff-6c3e-4d35-a73c-4b11c2ec2b7d\" (UID: \"6f4e75ff-6c3e-4d35-a73c-4b11c2ec2b7d\") " Oct 30 23:57:25.825732 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:25.825656 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f4e75ff-6c3e-4d35-a73c-4b11c2ec2b7d-kube-api-access-4spw9\" (UniqueName: \"kubernetes.io/projected/6f4e75ff-6c3e-4d35-a73c-4b11c2ec2b7d-kube-api-access-4spw9\") pod \"6f4e75ff-6c3e-4d35-a73c-4b11c2ec2b7d\" (UID: \"6f4e75ff-6c3e-4d35-a73c-4b11c2ec2b7d\") " Oct 30 23:57:25.825732 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:25.825726 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6f4e75ff-6c3e-4d35-a73c-4b11c2ec2b7d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f4e75ff-6c3e-4d35-a73c-4b11c2ec2b7d-maas-api-xks-ca\") pod \"6f4e75ff-6c3e-4d35-a73c-4b11c2ec2b7d\" (UID: \"6f4e75ff-6c3e-4d35-a73c-4b11c2ec2b7d\") " Oct 30 23:57:25.825978 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:25.825947 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f4e75ff-6c3e-4d35-a73c-4b11c2ec2b7d-maas-api-openshift-ca" pod "6f4e75ff-6c3e-4d35-a73c-4b11c2ec2b7d" (UID: "6f4e75ff-6c3e-4d35-a73c-4b11c2ec2b7d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:57:25.826164 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:25.826144 4927 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f4e75ff-6c3e-4d35-a73c-4b11c2ec2b7d-maas-api-openshift-ca\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 23:57:25.828406 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:25.828302 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6f4e75ff-6c3e-4d35-a73c-4b11c2ec2b7d-maas-api-xks-ca" pod "6f4e75ff-6c3e-4d35-a73c-4b11c2ec2b7d" (UID: "6f4e75ff-6c3e-4d35-a73c-4b11c2ec2b7d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 23:57:25.828691 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:25.828660 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f4e75ff-6c3e-4d35-a73c-4b11c2ec2b7d-kube-api-access-4spw9" pod "6f4e75ff-6c3e-4d35-a73c-4b11c2ec2b7d" (UID: "6f4e75ff-6c3e-4d35-a73c-4b11c2ec2b7d"). InnerVolumeSpecName "kube-api-access-4spw9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:57:25.868886 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:25.868856 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c802f331-n5s86" Oct 30 23:57:25.927249 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:25.927185 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4spw9\" (UniqueName: \"kubernetes.io/projected/6f4e75ff-6c3e-4d35-a73c-4b11c2ec2b7d-kube-api-access-4spw9\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 23:57:25.927249 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:25.927223 4927 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f4e75ff-6c3e-4d35-a73c-4b11c2ec2b7d-maas-api-xks-ca\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 23:57:26.246376 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:26.246351 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c802f331-n5s86"] Oct 30 23:57:26.247900 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:57:26.247869 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd617fcff_5199_415f_9d97_2185aae73433.slice/crio-fd5fe5f48f2a6144258a38e391e247b993303860ca81caaf352b2558834227bb WatchSource:0}: Error finding container fd5fe5f48f2a6144258a38e391e247b993303860ca81caaf352b2558834227bb: Status 404 returned error can't find the container with id fd5fe5f48f2a6144258a38e391e247b993303860ca81caaf352b2558834227bb Oct 30 23:57:26.507916 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:26.507814 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c802f331-n5s86" event={"ID":"d617fcff-5199-415f-9d97-2185aae73433","Type":"ContainerStarted","Data":"fd5fe5f48f2a6144258a38e391e247b993303860ca81caaf352b2558834227bb"} Oct 30 23:57:26.509512 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:26.509481 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9c40b0e-8lmzb" event={"ID":"6f4e75ff-6c3e-4d35-a73c-4b11c2ec2b7d","Type":"ContainerDied","Data":"c7467cc6b712545f6a5638809d78f2096422d5b767248884a65e7edbf5d491d7"} Oct 30 23:57:26.509512 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:26.509492 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9c40b0e-8lmzb" Oct 30 23:57:26.509722 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:26.509524 4927 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c7467cc6b712545f6a5638809d78f2096422d5b767248884a65e7edbf5d491d7" Oct 30 23:57:26.987355 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:26.986597 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-6gwbm"] Oct 30 23:57:26.987834 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:26.987680 4927 cpu_manager.go:414] "containerMap: removing container" podUID="6f4e75ff-6c3e-4d35-a73c-4b11c2ec2b7d" containerName="revoke-keys" Oct 30 23:57:26.987834 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:26.987705 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f4e75ff-6c3e-4d35-a73c-4b11c2ec2b7d" containerName="revoke-keys" Oct 30 23:57:26.987967 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:26.987856 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f4e75ff-6c3e-4d35-a73c-4b11c2ec2b7d" containerName="revoke-keys" Oct 30 23:57:27.002353 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:27.000632 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-6gwbm" Oct 30 23:57:27.056488 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:27.056455 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-34f1ce-openshift-default-dockercfg-gfqnh\"" Oct 30 23:57:27.114990 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:27.114284 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-6gwbm"] Oct 30 23:57:27.165579 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:27.164949 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4095540-8f80-4704-9146-06c5d2cab638-workload-socket\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-6gwbm\" (UID: \"c4095540-8f80-4704-9146-06c5d2cab638\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-6gwbm" Oct 30 23:57:27.165579 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:27.164995 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4095540-8f80-4704-9146-06c5d2cab638-istio-token\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-6gwbm\" (UID: \"c4095540-8f80-4704-9146-06c5d2cab638\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-6gwbm" Oct 30 23:57:27.165579 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:27.165034 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4095540-8f80-4704-9146-06c5d2cab638-istiod-ca-cert\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-6gwbm\" (UID: \"c4095540-8f80-4704-9146-06c5d2cab638\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-6gwbm" Oct 30 23:57:27.165579 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:27.165101 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4095540-8f80-4704-9146-06c5d2cab638-workload-certs\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-6gwbm\" (UID: \"c4095540-8f80-4704-9146-06c5d2cab638\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-6gwbm" Oct 30 23:57:27.165579 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:27.165128 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4095540-8f80-4704-9146-06c5d2cab638-istio-envoy\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-6gwbm\" (UID: \"c4095540-8f80-4704-9146-06c5d2cab638\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-6gwbm" Oct 30 23:57:27.165579 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:27.165169 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqf8z\" (UniqueName: \"kubernetes.io/projected/c4095540-8f80-4704-9146-06c5d2cab638-kube-api-access-sqf8z\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-6gwbm\" (UID: \"c4095540-8f80-4704-9146-06c5d2cab638\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-6gwbm" Oct 30 23:57:27.165579 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:27.165215 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4095540-8f80-4704-9146-06c5d2cab638-credential-socket\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-6gwbm\" (UID: \"c4095540-8f80-4704-9146-06c5d2cab638\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-6gwbm" Oct 30 23:57:27.165579 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:27.165274 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4095540-8f80-4704-9146-06c5d2cab638-istio-data\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-6gwbm\" (UID: \"c4095540-8f80-4704-9146-06c5d2cab638\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-6gwbm" Oct 30 23:57:27.165579 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:27.165306 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4095540-8f80-4704-9146-06c5d2cab638-istio-podinfo\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-6gwbm\" (UID: \"c4095540-8f80-4704-9146-06c5d2cab638\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-6gwbm" Oct 30 23:57:27.247680 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:27.247581 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:57:27.271137 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:27.267950 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4095540-8f80-4704-9146-06c5d2cab638-istio-data\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-6gwbm\" (UID: \"c4095540-8f80-4704-9146-06c5d2cab638\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-6gwbm" Oct 30 23:57:27.271432 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:27.271389 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4095540-8f80-4704-9146-06c5d2cab638-istio-podinfo\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-6gwbm\" (UID: \"c4095540-8f80-4704-9146-06c5d2cab638\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-6gwbm" Oct 30 23:57:27.271571 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:27.271534 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4095540-8f80-4704-9146-06c5d2cab638-workload-socket\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-6gwbm\" (UID: \"c4095540-8f80-4704-9146-06c5d2cab638\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-6gwbm" Oct 30 23:57:27.271571 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:27.271563 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4095540-8f80-4704-9146-06c5d2cab638-istio-token\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-6gwbm\" (UID: \"c4095540-8f80-4704-9146-06c5d2cab638\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-6gwbm" Oct 30 23:57:27.271707 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:27.271605 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4095540-8f80-4704-9146-06c5d2cab638-istiod-ca-cert\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-6gwbm\" (UID: \"c4095540-8f80-4704-9146-06c5d2cab638\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-6gwbm" Oct 30 23:57:27.271707 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:27.271675 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4095540-8f80-4704-9146-06c5d2cab638-workload-certs\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-6gwbm\" (UID: \"c4095540-8f80-4704-9146-06c5d2cab638\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-6gwbm" Oct 30 23:57:27.271822 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:27.271707 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4095540-8f80-4704-9146-06c5d2cab638-istio-envoy\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-6gwbm\" (UID: \"c4095540-8f80-4704-9146-06c5d2cab638\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-6gwbm" Oct 30 23:57:27.271822 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:27.270894 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4095540-8f80-4704-9146-06c5d2cab638-istio-data\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-6gwbm\" (UID: \"c4095540-8f80-4704-9146-06c5d2cab638\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-6gwbm" Oct 30 23:57:27.271822 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:27.271740 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqf8z\" (UniqueName: \"kubernetes.io/projected/c4095540-8f80-4704-9146-06c5d2cab638-kube-api-access-sqf8z\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-6gwbm\" (UID: \"c4095540-8f80-4704-9146-06c5d2cab638\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-6gwbm" Oct 30 23:57:27.271984 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:27.271819 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4095540-8f80-4704-9146-06c5d2cab638-credential-socket\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-6gwbm\" (UID: \"c4095540-8f80-4704-9146-06c5d2cab638\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-6gwbm" Oct 30 23:57:27.272215 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:27.272161 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4095540-8f80-4704-9146-06c5d2cab638-credential-socket\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-6gwbm\" (UID: \"c4095540-8f80-4704-9146-06c5d2cab638\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-6gwbm" Oct 30 23:57:27.272927 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:27.272898 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4095540-8f80-4704-9146-06c5d2cab638-istiod-ca-cert\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-6gwbm\" (UID: \"c4095540-8f80-4704-9146-06c5d2cab638\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-6gwbm" Oct 30 23:57:27.275097 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:27.275079 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4095540-8f80-4704-9146-06c5d2cab638-workload-socket\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-6gwbm\" (UID: \"c4095540-8f80-4704-9146-06c5d2cab638\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-6gwbm" Oct 30 23:57:27.276784 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:27.276759 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4095540-8f80-4704-9146-06c5d2cab638-workload-certs\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-6gwbm\" (UID: \"c4095540-8f80-4704-9146-06c5d2cab638\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-6gwbm" Oct 30 23:57:27.283369 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:27.279318 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4095540-8f80-4704-9146-06c5d2cab638-istio-podinfo\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-6gwbm\" (UID: \"c4095540-8f80-4704-9146-06c5d2cab638\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-6gwbm" Oct 30 23:57:27.284178 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:27.284139 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4095540-8f80-4704-9146-06c5d2cab638-istio-envoy\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-6gwbm\" (UID: \"c4095540-8f80-4704-9146-06c5d2cab638\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-6gwbm" Oct 30 23:57:27.344196 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:27.344159 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqf8z\" (UniqueName: \"kubernetes.io/projected/c4095540-8f80-4704-9146-06c5d2cab638-kube-api-access-sqf8z\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-6gwbm\" (UID: \"c4095540-8f80-4704-9146-06c5d2cab638\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-6gwbm" Oct 30 23:57:27.347162 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:27.347101 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4095540-8f80-4704-9146-06c5d2cab638-istio-token\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-6gwbm\" (UID: \"c4095540-8f80-4704-9146-06c5d2cab638\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-6gwbm" Oct 30 23:57:27.519834 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:27.519740 4927 generic.go:356] "Generic (PLEG): container finished" podID="d617fcff-5199-415f-9d97-2185aae73433" containerID="3c50a1544fbd51c805356bf11bbd843333ac2484aa92df7735634b4ab1997ea4" exitCode=0 Oct 30 23:57:27.519995 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:27.519828 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c802f331-n5s86" event={"ID":"d617fcff-5199-415f-9d97-2185aae73433","Type":"ContainerDied","Data":"3c50a1544fbd51c805356bf11bbd843333ac2484aa92df7735634b4ab1997ea4"} Oct 30 23:57:27.645306 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:27.645270 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-6gwbm" Oct 30 23:57:28.176426 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:28.165378 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-6gwbm"] Oct 30 23:57:28.187424 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:57:28.187383 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4095540_8f80_4704_9146_06c5d2cab638.slice/crio-05a237873bcab7c8bc18b0da2ede80f9c02df0608b78caf27483364c939e15e7 WatchSource:0}: Error finding container 05a237873bcab7c8bc18b0da2ede80f9c02df0608b78caf27483364c939e15e7: Status 404 returned error can't find the container with id 05a237873bcab7c8bc18b0da2ede80f9c02df0608b78caf27483364c939e15e7 Oct 30 23:57:28.189728 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:28.189692 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 23:57:28.189837 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:28.189764 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 23:57:28.189837 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:28.189797 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 23:57:28.534692 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:28.534597 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-6gwbm" event={"ID":"c4095540-8f80-4704-9146-06c5d2cab638","Type":"ContainerStarted","Data":"b0ffaaa12da8f91486d229c1c0bb92ccbb26d83af61ba90d4486fa24a17e787a"} Oct 30 23:57:28.534692 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:28.534668 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-6gwbm" event={"ID":"c4095540-8f80-4704-9146-06c5d2cab638","Type":"ContainerStarted","Data":"05a237873bcab7c8bc18b0da2ede80f9c02df0608b78caf27483364c939e15e7"} Oct 30 23:57:28.645508 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:28.645475 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-6gwbm" Oct 30 23:57:28.667237 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:28.667183 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-6gwbm" podStartSLOduration=2.667166818 podStartE2EDuration="2.667166818s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:57:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:57:28.660464107 +0000 UTC m=+961.696794932" watchObservedRunningTime="2026-10-30 23:57:28.667166818 +0000 UTC m=+961.703497712" Oct 30 23:57:28.721964 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:28.721930 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-t762r"] Oct 30 23:57:28.726671 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:28.726641 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-t762r" Oct 30 23:57:28.748004 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:28.747981 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c802f331-n5s86" Oct 30 23:57:28.780946 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:28.780909 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-fe01fa-openshift-default-dockercfg-5mw89\"" Oct 30 23:57:28.783448 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:28.783190 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-t762r"] Oct 30 23:57:28.803547 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:28.803467 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8r7wf\" (UniqueName: \"kubernetes.io/projected/b9e9d9b1-22af-48d6-b1de-b1c28107ace1-kube-api-access-8r7wf\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-t762r\" (UID: \"b9e9d9b1-22af-48d6-b1de-b1c28107ace1\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-t762r" Oct 30 23:57:28.803670 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:28.803552 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9e9d9b1-22af-48d6-b1de-b1c28107ace1-istio-token\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-t762r\" (UID: \"b9e9d9b1-22af-48d6-b1de-b1c28107ace1\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-t762r" Oct 30 23:57:28.803670 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:28.803596 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9e9d9b1-22af-48d6-b1de-b1c28107ace1-istio-envoy\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-t762r\" (UID: \"b9e9d9b1-22af-48d6-b1de-b1c28107ace1\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-t762r" Oct 30 23:57:28.803670 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:28.803659 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9e9d9b1-22af-48d6-b1de-b1c28107ace1-istio-podinfo\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-t762r\" (UID: \"b9e9d9b1-22af-48d6-b1de-b1c28107ace1\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-t762r" Oct 30 23:57:28.803783 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:28.803745 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9e9d9b1-22af-48d6-b1de-b1c28107ace1-workload-certs\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-t762r\" (UID: \"b9e9d9b1-22af-48d6-b1de-b1c28107ace1\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-t762r" Oct 30 23:57:28.803783 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:28.803769 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9e9d9b1-22af-48d6-b1de-b1c28107ace1-istiod-ca-cert\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-t762r\" (UID: \"b9e9d9b1-22af-48d6-b1de-b1c28107ace1\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-t762r" Oct 30 23:57:28.803901 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:28.803867 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9e9d9b1-22af-48d6-b1de-b1c28107ace1-credential-socket\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-t762r\" (UID: \"b9e9d9b1-22af-48d6-b1de-b1c28107ace1\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-t762r" Oct 30 23:57:28.803933 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:28.803907 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9e9d9b1-22af-48d6-b1de-b1c28107ace1-workload-socket\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-t762r\" (UID: \"b9e9d9b1-22af-48d6-b1de-b1c28107ace1\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-t762r" Oct 30 23:57:28.804378 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:28.804089 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9e9d9b1-22af-48d6-b1de-b1c28107ace1-istio-data\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-t762r\" (UID: \"b9e9d9b1-22af-48d6-b1de-b1c28107ace1\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-t762r" Oct 30 23:57:28.905581 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:28.905538 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d617fcff-5199-415f-9d97-2185aae73433-kube-api-access-4zdqg\" (UniqueName: \"kubernetes.io/projected/d617fcff-5199-415f-9d97-2185aae73433-kube-api-access-4zdqg\") pod \"d617fcff-5199-415f-9d97-2185aae73433\" (UID: \"d617fcff-5199-415f-9d97-2185aae73433\") " Oct 30 23:57:28.905743 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:28.905720 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d617fcff-5199-415f-9d97-2185aae73433-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d617fcff-5199-415f-9d97-2185aae73433-maas-api-openshift-ca\") pod \"d617fcff-5199-415f-9d97-2185aae73433\" (UID: \"d617fcff-5199-415f-9d97-2185aae73433\") " Oct 30 23:57:28.905803 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:28.905785 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d617fcff-5199-415f-9d97-2185aae73433-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d617fcff-5199-415f-9d97-2185aae73433-maas-api-xks-ca\") pod \"d617fcff-5199-415f-9d97-2185aae73433\" (UID: \"d617fcff-5199-415f-9d97-2185aae73433\") " Oct 30 23:57:28.906261 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:28.906010 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9e9d9b1-22af-48d6-b1de-b1c28107ace1-istio-token\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-t762r\" (UID: \"b9e9d9b1-22af-48d6-b1de-b1c28107ace1\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-t762r" Oct 30 23:57:28.906261 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:28.906056 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9e9d9b1-22af-48d6-b1de-b1c28107ace1-istio-envoy\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-t762r\" (UID: \"b9e9d9b1-22af-48d6-b1de-b1c28107ace1\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-t762r" Oct 30 23:57:28.906261 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:28.906118 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9e9d9b1-22af-48d6-b1de-b1c28107ace1-istio-podinfo\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-t762r\" (UID: \"b9e9d9b1-22af-48d6-b1de-b1c28107ace1\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-t762r" Oct 30 23:57:28.906261 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:28.906124 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d617fcff-5199-415f-9d97-2185aae73433-maas-api-openshift-ca" pod "d617fcff-5199-415f-9d97-2185aae73433" (UID: "d617fcff-5199-415f-9d97-2185aae73433"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:57:28.906261 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:28.906225 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9e9d9b1-22af-48d6-b1de-b1c28107ace1-workload-certs\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-t762r\" (UID: \"b9e9d9b1-22af-48d6-b1de-b1c28107ace1\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-t762r" Oct 30 23:57:28.906723 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:28.906272 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9e9d9b1-22af-48d6-b1de-b1c28107ace1-istiod-ca-cert\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-t762r\" (UID: \"b9e9d9b1-22af-48d6-b1de-b1c28107ace1\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-t762r" Oct 30 23:57:28.906723 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:28.906583 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9e9d9b1-22af-48d6-b1de-b1c28107ace1-workload-certs\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-t762r\" (UID: \"b9e9d9b1-22af-48d6-b1de-b1c28107ace1\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-t762r" Oct 30 23:57:28.907284 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:28.906911 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9e9d9b1-22af-48d6-b1de-b1c28107ace1-credential-socket\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-t762r\" (UID: \"b9e9d9b1-22af-48d6-b1de-b1c28107ace1\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-t762r" Oct 30 23:57:28.907284 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:28.906964 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9e9d9b1-22af-48d6-b1de-b1c28107ace1-workload-socket\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-t762r\" (UID: \"b9e9d9b1-22af-48d6-b1de-b1c28107ace1\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-t762r" Oct 30 23:57:28.907284 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:28.906989 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9e9d9b1-22af-48d6-b1de-b1c28107ace1-istiod-ca-cert\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-t762r\" (UID: \"b9e9d9b1-22af-48d6-b1de-b1c28107ace1\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-t762r" Oct 30 23:57:28.907284 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:28.907003 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9e9d9b1-22af-48d6-b1de-b1c28107ace1-istio-data\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-t762r\" (UID: \"b9e9d9b1-22af-48d6-b1de-b1c28107ace1\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-t762r" Oct 30 23:57:28.907284 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:28.907062 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8r7wf\" (UniqueName: \"kubernetes.io/projected/b9e9d9b1-22af-48d6-b1de-b1c28107ace1-kube-api-access-8r7wf\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-t762r\" (UID: \"b9e9d9b1-22af-48d6-b1de-b1c28107ace1\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-t762r" Oct 30 23:57:28.907284 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:28.907150 4927 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d617fcff-5199-415f-9d97-2185aae73433-maas-api-openshift-ca\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 23:57:28.907284 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:28.907175 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9e9d9b1-22af-48d6-b1de-b1c28107ace1-credential-socket\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-t762r\" (UID: \"b9e9d9b1-22af-48d6-b1de-b1c28107ace1\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-t762r" Oct 30 23:57:28.907284 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:28.907240 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9e9d9b1-22af-48d6-b1de-b1c28107ace1-istio-data\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-t762r\" (UID: \"b9e9d9b1-22af-48d6-b1de-b1c28107ace1\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-t762r" Oct 30 23:57:28.907793 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:28.907456 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9e9d9b1-22af-48d6-b1de-b1c28107ace1-workload-socket\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-t762r\" (UID: \"b9e9d9b1-22af-48d6-b1de-b1c28107ace1\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-t762r" Oct 30 23:57:28.936357 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:28.912628 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d617fcff-5199-415f-9d97-2185aae73433-maas-api-xks-ca" pod "d617fcff-5199-415f-9d97-2185aae73433" (UID: "d617fcff-5199-415f-9d97-2185aae73433"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 23:57:28.936357 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:28.922496 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d617fcff-5199-415f-9d97-2185aae73433-kube-api-access-4zdqg" pod "d617fcff-5199-415f-9d97-2185aae73433" (UID: "d617fcff-5199-415f-9d97-2185aae73433"). InnerVolumeSpecName "kube-api-access-4zdqg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:57:28.936357 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:28.923610 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9e9d9b1-22af-48d6-b1de-b1c28107ace1-istio-envoy\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-t762r\" (UID: \"b9e9d9b1-22af-48d6-b1de-b1c28107ace1\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-t762r" Oct 30 23:57:28.960356 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:28.946970 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9e9d9b1-22af-48d6-b1de-b1c28107ace1-istio-podinfo\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-t762r\" (UID: \"b9e9d9b1-22af-48d6-b1de-b1c28107ace1\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-t762r" Oct 30 23:57:28.976368 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:28.971269 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9e9d9b1-22af-48d6-b1de-b1c28107ace1-istio-token\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-t762r\" (UID: \"b9e9d9b1-22af-48d6-b1de-b1c28107ace1\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-t762r" Oct 30 23:57:28.992379 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:28.992082 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8r7wf\" (UniqueName: \"kubernetes.io/projected/b9e9d9b1-22af-48d6-b1de-b1c28107ace1-kube-api-access-8r7wf\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-t762r\" (UID: \"b9e9d9b1-22af-48d6-b1de-b1c28107ace1\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-t762r" Oct 30 23:57:29.011357 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:29.008581 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4zdqg\" (UniqueName: \"kubernetes.io/projected/d617fcff-5199-415f-9d97-2185aae73433-kube-api-access-4zdqg\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 23:57:29.011357 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:29.008618 4927 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d617fcff-5199-415f-9d97-2185aae73433-maas-api-xks-ca\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 23:57:29.066409 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:29.065527 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-t762r" Oct 30 23:57:29.564350 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:29.564297 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c802f331-n5s86" Oct 30 23:57:29.566070 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:29.565839 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c802f331-n5s86" event={"ID":"d617fcff-5199-415f-9d97-2185aae73433","Type":"ContainerDied","Data":"fd5fe5f48f2a6144258a38e391e247b993303860ca81caaf352b2558834227bb"} Oct 30 23:57:29.566070 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:29.565886 4927 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fd5fe5f48f2a6144258a38e391e247b993303860ca81caaf352b2558834227bb" Oct 30 23:57:29.618363 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:29.618231 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-t762r"] Oct 30 23:57:29.639896 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:29.639853 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 23:57:29.640101 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:29.640080 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 23:57:29.640273 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:29.640251 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 23:57:29.666029 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:29.665949 4927 patch_prober.go:29] interesting pod/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-6gwbm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Oct 30 23:57:29.666029 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:29.666004 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-6gwbm" podUID="c4095540-8f80-4704-9146-06c5d2cab638" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Oct 30 23:57:30.054024 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:30.053990 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b1844f1a-t6tp6"] Oct 30 23:57:30.055117 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:30.055092 4927 cpu_manager.go:414] "containerMap: removing container" podUID="d617fcff-5199-415f-9d97-2185aae73433" containerName="revoke-keys" Oct 30 23:57:30.055270 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:30.055255 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d617fcff-5199-415f-9d97-2185aae73433" containerName="revoke-keys" Oct 30 23:57:30.055618 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:30.055604 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="d617fcff-5199-415f-9d97-2185aae73433" containerName="revoke-keys" Oct 30 23:57:30.059837 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:30.059819 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1844f1a-t6tp6" Oct 30 23:57:30.142693 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:30.141998 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4fdzs\"" Oct 30 23:57:30.142693 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:30.142080 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 23:57:30.159326 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:30.159285 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b1844f1a-t6tp6"] Oct 30 23:57:30.171602 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:30.168049 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af1791d5-2960-42bd-95a8-6f6b0ef78e93-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b1844f1a-t6tp6\" (UID: \"af1791d5-2960-42bd-95a8-6f6b0ef78e93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1844f1a-t6tp6" Oct 30 23:57:30.171602 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:30.168173 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af1791d5-2960-42bd-95a8-6f6b0ef78e93-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b1844f1a-t6tp6\" (UID: \"af1791d5-2960-42bd-95a8-6f6b0ef78e93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1844f1a-t6tp6" Oct 30 23:57:30.171602 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:30.168299 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7nk9j\" (UniqueName: \"kubernetes.io/projected/af1791d5-2960-42bd-95a8-6f6b0ef78e93-kube-api-access-7nk9j\") pod \"maas-api-revoke-sub-b1844f1a-t6tp6\" (UID: \"af1791d5-2960-42bd-95a8-6f6b0ef78e93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1844f1a-t6tp6" Oct 30 23:57:30.177509 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:30.176758 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-944gh"] Oct 30 23:57:30.195183 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:30.195148 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-944gh" Oct 30 23:57:30.261400 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:30.261368 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-944gh"] Oct 30 23:57:30.269045 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:30.269015 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af1791d5-2960-42bd-95a8-6f6b0ef78e93-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b1844f1a-t6tp6\" (UID: \"af1791d5-2960-42bd-95a8-6f6b0ef78e93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1844f1a-t6tp6" Oct 30 23:57:30.269348 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:30.269315 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7nk9j\" (UniqueName: \"kubernetes.io/projected/af1791d5-2960-42bd-95a8-6f6b0ef78e93-kube-api-access-7nk9j\") pod \"maas-api-revoke-sub-b1844f1a-t6tp6\" (UID: \"af1791d5-2960-42bd-95a8-6f6b0ef78e93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1844f1a-t6tp6" Oct 30 23:57:30.269831 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:30.269807 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af1791d5-2960-42bd-95a8-6f6b0ef78e93-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b1844f1a-t6tp6\" (UID: \"af1791d5-2960-42bd-95a8-6f6b0ef78e93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1844f1a-t6tp6" Oct 30 23:57:30.270505 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:30.270484 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af1791d5-2960-42bd-95a8-6f6b0ef78e93-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b1844f1a-t6tp6\" (UID: \"af1791d5-2960-42bd-95a8-6f6b0ef78e93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1844f1a-t6tp6" Oct 30 23:57:30.276677 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:30.276657 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af1791d5-2960-42bd-95a8-6f6b0ef78e93-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b1844f1a-t6tp6\" (UID: \"af1791d5-2960-42bd-95a8-6f6b0ef78e93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1844f1a-t6tp6" Oct 30 23:57:30.343507 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:30.343428 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7nk9j\" (UniqueName: \"kubernetes.io/projected/af1791d5-2960-42bd-95a8-6f6b0ef78e93-kube-api-access-7nk9j\") pod \"maas-api-revoke-sub-b1844f1a-t6tp6\" (UID: \"af1791d5-2960-42bd-95a8-6f6b0ef78e93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1844f1a-t6tp6" Oct 30 23:57:30.371759 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:30.371728 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91f11663-5773-418a-bde2-532ead3c628c-istio-data\") pod \"maas-default-gateway-openshift-default-8f6f87f59-944gh\" (UID: \"91f11663-5773-418a-bde2-532ead3c628c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-944gh" Oct 30 23:57:30.371931 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:30.371775 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91f11663-5773-418a-bde2-532ead3c628c-workload-certs\") pod \"maas-default-gateway-openshift-default-8f6f87f59-944gh\" (UID: \"91f11663-5773-418a-bde2-532ead3c628c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-944gh" Oct 30 23:57:30.371931 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:30.371791 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91f11663-5773-418a-bde2-532ead3c628c-istio-envoy\") pod \"maas-default-gateway-openshift-default-8f6f87f59-944gh\" (UID: \"91f11663-5773-418a-bde2-532ead3c628c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-944gh" Oct 30 23:57:30.372023 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:30.371945 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91f11663-5773-418a-bde2-532ead3c628c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8f6f87f59-944gh\" (UID: \"91f11663-5773-418a-bde2-532ead3c628c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-944gh" Oct 30 23:57:30.372023 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:30.371988 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91f11663-5773-418a-bde2-532ead3c628c-istio-token\") pod \"maas-default-gateway-openshift-default-8f6f87f59-944gh\" (UID: \"91f11663-5773-418a-bde2-532ead3c628c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-944gh" Oct 30 23:57:30.372096 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:30.372051 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91f11663-5773-418a-bde2-532ead3c628c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8f6f87f59-944gh\" (UID: \"91f11663-5773-418a-bde2-532ead3c628c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-944gh" Oct 30 23:57:30.372143 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:30.372097 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91f11663-5773-418a-bde2-532ead3c628c-workload-socket\") pod \"maas-default-gateway-openshift-default-8f6f87f59-944gh\" (UID: \"91f11663-5773-418a-bde2-532ead3c628c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-944gh" Oct 30 23:57:30.372179 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:30.372143 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5fb6\" (UniqueName: \"kubernetes.io/projected/91f11663-5773-418a-bde2-532ead3c628c-kube-api-access-k5fb6\") pod \"maas-default-gateway-openshift-default-8f6f87f59-944gh\" (UID: \"91f11663-5773-418a-bde2-532ead3c628c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-944gh" Oct 30 23:57:30.372179 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:30.372167 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91f11663-5773-418a-bde2-532ead3c628c-credential-socket\") pod \"maas-default-gateway-openshift-default-8f6f87f59-944gh\" (UID: \"91f11663-5773-418a-bde2-532ead3c628c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-944gh" Oct 30 23:57:30.378485 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:30.378462 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1844f1a-t6tp6" Oct 30 23:57:30.473034 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:30.472993 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91f11663-5773-418a-bde2-532ead3c628c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8f6f87f59-944gh\" (UID: \"91f11663-5773-418a-bde2-532ead3c628c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-944gh" Oct 30 23:57:30.473155 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:30.473067 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91f11663-5773-418a-bde2-532ead3c628c-istio-token\") pod \"maas-default-gateway-openshift-default-8f6f87f59-944gh\" (UID: \"91f11663-5773-418a-bde2-532ead3c628c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-944gh" Oct 30 23:57:30.473155 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:30.473094 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91f11663-5773-418a-bde2-532ead3c628c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8f6f87f59-944gh\" (UID: \"91f11663-5773-418a-bde2-532ead3c628c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-944gh" Oct 30 23:57:30.473155 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:30.473134 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91f11663-5773-418a-bde2-532ead3c628c-workload-socket\") pod \"maas-default-gateway-openshift-default-8f6f87f59-944gh\" (UID: \"91f11663-5773-418a-bde2-532ead3c628c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-944gh" Oct 30 23:57:30.473405 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:30.473168 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5fb6\" (UniqueName: \"kubernetes.io/projected/91f11663-5773-418a-bde2-532ead3c628c-kube-api-access-k5fb6\") pod \"maas-default-gateway-openshift-default-8f6f87f59-944gh\" (UID: \"91f11663-5773-418a-bde2-532ead3c628c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-944gh" Oct 30 23:57:30.473405 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:30.473190 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91f11663-5773-418a-bde2-532ead3c628c-credential-socket\") pod \"maas-default-gateway-openshift-default-8f6f87f59-944gh\" (UID: \"91f11663-5773-418a-bde2-532ead3c628c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-944gh" Oct 30 23:57:30.473405 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:30.473260 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91f11663-5773-418a-bde2-532ead3c628c-istio-data\") pod \"maas-default-gateway-openshift-default-8f6f87f59-944gh\" (UID: \"91f11663-5773-418a-bde2-532ead3c628c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-944gh" Oct 30 23:57:30.473405 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:30.473289 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91f11663-5773-418a-bde2-532ead3c628c-workload-certs\") pod \"maas-default-gateway-openshift-default-8f6f87f59-944gh\" (UID: \"91f11663-5773-418a-bde2-532ead3c628c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-944gh" Oct 30 23:57:30.473405 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:30.473304 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91f11663-5773-418a-bde2-532ead3c628c-istio-envoy\") pod \"maas-default-gateway-openshift-default-8f6f87f59-944gh\" (UID: \"91f11663-5773-418a-bde2-532ead3c628c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-944gh" Oct 30 23:57:30.481037 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:30.480154 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91f11663-5773-418a-bde2-532ead3c628c-workload-socket\") pod \"maas-default-gateway-openshift-default-8f6f87f59-944gh\" (UID: \"91f11663-5773-418a-bde2-532ead3c628c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-944gh" Oct 30 23:57:30.481037 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:30.480559 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91f11663-5773-418a-bde2-532ead3c628c-credential-socket\") pod \"maas-default-gateway-openshift-default-8f6f87f59-944gh\" (UID: \"91f11663-5773-418a-bde2-532ead3c628c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-944gh" Oct 30 23:57:30.481037 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:30.480958 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91f11663-5773-418a-bde2-532ead3c628c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8f6f87f59-944gh\" (UID: \"91f11663-5773-418a-bde2-532ead3c628c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-944gh" Oct 30 23:57:30.481365 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:30.481239 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91f11663-5773-418a-bde2-532ead3c628c-istio-data\") pod \"maas-default-gateway-openshift-default-8f6f87f59-944gh\" (UID: \"91f11663-5773-418a-bde2-532ead3c628c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-944gh" Oct 30 23:57:30.483845 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:30.483802 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91f11663-5773-418a-bde2-532ead3c628c-workload-certs\") pod \"maas-default-gateway-openshift-default-8f6f87f59-944gh\" (UID: \"91f11663-5773-418a-bde2-532ead3c628c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-944gh" Oct 30 23:57:30.484312 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:30.484273 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91f11663-5773-418a-bde2-532ead3c628c-istio-envoy\") pod \"maas-default-gateway-openshift-default-8f6f87f59-944gh\" (UID: \"91f11663-5773-418a-bde2-532ead3c628c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-944gh" Oct 30 23:57:30.486754 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:30.486732 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91f11663-5773-418a-bde2-532ead3c628c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8f6f87f59-944gh\" (UID: \"91f11663-5773-418a-bde2-532ead3c628c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-944gh" Oct 30 23:57:30.593378 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:30.592835 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91f11663-5773-418a-bde2-532ead3c628c-istio-token\") pod \"maas-default-gateway-openshift-default-8f6f87f59-944gh\" (UID: \"91f11663-5773-418a-bde2-532ead3c628c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-944gh" Oct 30 23:57:30.593378 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:30.593152 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5fb6\" (UniqueName: \"kubernetes.io/projected/91f11663-5773-418a-bde2-532ead3c628c-kube-api-access-k5fb6\") pod \"maas-default-gateway-openshift-default-8f6f87f59-944gh\" (UID: \"91f11663-5773-418a-bde2-532ead3c628c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-944gh" Oct 30 23:57:30.595884 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:30.595795 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-t762r" event={"ID":"b9e9d9b1-22af-48d6-b1de-b1c28107ace1","Type":"ContainerStarted","Data":"0548d73c9bd7e0ba046a631af25ddd93a5a5321d0e09857824458683c21f6a8e"} Oct 30 23:57:30.595884 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:30.595856 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-t762r" event={"ID":"b9e9d9b1-22af-48d6-b1de-b1c28107ace1","Type":"ContainerStarted","Data":"02a281219231a3ac0f94f7781dced895007d23ab7c69f7a337780855815f5605"} Oct 30 23:57:30.653366 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:30.646759 4927 patch_prober.go:29] interesting pod/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-6gwbm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Oct 30 23:57:30.653366 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:30.646825 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-6gwbm" podUID="c4095540-8f80-4704-9146-06c5d2cab638" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Oct 30 23:57:30.812852 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:30.812586 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-944gh" Oct 30 23:57:30.867752 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:30.867684 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-t762r" podStartSLOduration=2.86765822 podStartE2EDuration="2.86765822s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:57:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:57:30.739700382 +0000 UTC m=+963.776031237" watchObservedRunningTime="2026-10-30 23:57:30.86765822 +0000 UTC m=+963.903989044" Oct 30 23:57:30.869233 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:30.869200 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b1844f1a-t6tp6"] Oct 30 23:57:30.881035 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:57:30.880975 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf1791d5_2960_42bd_95a8_6f6b0ef78e93.slice/crio-626a731fe954ae75fb31622dc6704e9aa02d8d86d79776577d64d72f3b445d16 WatchSource:0}: Error finding container 626a731fe954ae75fb31622dc6704e9aa02d8d86d79776577d64d72f3b445d16: Status 404 returned error can't find the container with id 626a731fe954ae75fb31622dc6704e9aa02d8d86d79776577d64d72f3b445d16 Oct 30 23:57:31.075514 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:31.075435 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-t762r" Oct 30 23:57:31.080519 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:31.078476 4927 patch_prober.go:29] interesting pod/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-t762r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Oct 30 23:57:31.080519 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:31.078533 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-t762r" podUID="b9e9d9b1-22af-48d6-b1de-b1c28107ace1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Oct 30 23:57:31.390136 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:57:31.390100 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91f11663_5773_418a_bde2_532ead3c628c.slice/crio-7ad3ee7cfa501b72fc8bbceadfd70465a874089f15bb06774958c31f2a2577e8 WatchSource:0}: Error finding container 7ad3ee7cfa501b72fc8bbceadfd70465a874089f15bb06774958c31f2a2577e8: Status 404 returned error can't find the container with id 7ad3ee7cfa501b72fc8bbceadfd70465a874089f15bb06774958c31f2a2577e8 Oct 30 23:57:31.390791 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:31.390665 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-944gh"] Oct 30 23:57:31.397322 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:31.397150 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 23:57:31.397322 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:31.397241 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 23:57:31.397322 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:31.397285 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 23:57:31.605011 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:31.604979 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1844f1a-t6tp6" event={"ID":"af1791d5-2960-42bd-95a8-6f6b0ef78e93","Type":"ContainerStarted","Data":"5cd99cb82d13da4c5d9598682dff7706c9c170d579d3d39f0344434b0b9d383d"} Oct 30 23:57:31.605422 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:31.605041 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1844f1a-t6tp6" event={"ID":"af1791d5-2960-42bd-95a8-6f6b0ef78e93","Type":"ContainerStarted","Data":"626a731fe954ae75fb31622dc6704e9aa02d8d86d79776577d64d72f3b445d16"} Oct 30 23:57:31.606651 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:31.606626 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-944gh" event={"ID":"91f11663-5773-418a-bde2-532ead3c628c","Type":"ContainerStarted","Data":"7ad3ee7cfa501b72fc8bbceadfd70465a874089f15bb06774958c31f2a2577e8"} Oct 30 23:57:31.652358 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:31.651772 4927 patch_prober.go:29] interesting pod/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-6gwbm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Oct 30 23:57:31.652358 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:31.651839 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-6gwbm" podUID="c4095540-8f80-4704-9146-06c5d2cab638" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Oct 30 23:57:31.747414 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:31.740679 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1844f1a-t6tp6" podStartSLOduration=2.740659558 podStartE2EDuration="2.740659558s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:57:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:57:31.739750318 +0000 UTC m=+964.776081179" watchObservedRunningTime="2026-10-30 23:57:31.740659558 +0000 UTC m=+964.776990380" Oct 30 23:57:32.067406 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:32.066750 4927 patch_prober.go:29] interesting pod/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-t762r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Oct 30 23:57:32.067406 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:32.066816 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-t762r" podUID="b9e9d9b1-22af-48d6-b1de-b1c28107ace1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Oct 30 23:57:32.096269 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:32.096237 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:57:32.636353 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:32.636230 4927 generic.go:356] "Generic (PLEG): container finished" podID="af1791d5-2960-42bd-95a8-6f6b0ef78e93" containerID="5cd99cb82d13da4c5d9598682dff7706c9c170d579d3d39f0344434b0b9d383d" exitCode=0 Oct 30 23:57:32.636845 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:32.636394 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1844f1a-t6tp6" event={"ID":"af1791d5-2960-42bd-95a8-6f6b0ef78e93","Type":"ContainerDied","Data":"5cd99cb82d13da4c5d9598682dff7706c9c170d579d3d39f0344434b0b9d383d"} Oct 30 23:57:32.639481 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:32.639453 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-944gh" event={"ID":"91f11663-5773-418a-bde2-532ead3c628c","Type":"ContainerStarted","Data":"0d9a7b1bae0ff7417b190cdc75722d41470432042fdc361ecee591c45a550339"} Oct 30 23:57:32.647734 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:32.647703 4927 patch_prober.go:29] interesting pod/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-6gwbm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Oct 30 23:57:32.648956 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:32.648925 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-6gwbm" podUID="c4095540-8f80-4704-9146-06c5d2cab638" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Oct 30 23:57:32.833645 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:32.822854 4927 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-944gh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Oct 30 23:57:32.833645 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:32.823008 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-944gh" podUID="91f11663-5773-418a-bde2-532ead3c628c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Oct 30 23:57:32.833645 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:32.827554 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-944gh" Oct 30 23:57:32.984386 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:32.983938 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-944gh" podStartSLOduration=2.9839141639999998 podStartE2EDuration="2.983914164s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:57:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:57:32.935175774 +0000 UTC m=+965.971506600" watchObservedRunningTime="2026-10-30 23:57:32.983914164 +0000 UTC m=+966.020244987" Oct 30 23:57:33.070356 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:33.069743 4927 patch_prober.go:29] interesting pod/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-t762r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Oct 30 23:57:33.070356 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:33.069804 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-t762r" podUID="b9e9d9b1-22af-48d6-b1de-b1c28107ace1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Oct 30 23:57:33.654362 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:33.653935 4927 patch_prober.go:29] interesting pod/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-6gwbm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Oct 30 23:57:33.654362 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:33.654004 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-6gwbm" podUID="c4095540-8f80-4704-9146-06c5d2cab638" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Oct 30 23:57:33.816401 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:33.816364 4927 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-944gh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Oct 30 23:57:33.817304 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:33.817270 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-944gh" podUID="91f11663-5773-418a-bde2-532ead3c628c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Oct 30 23:57:34.207480 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:34.207443 4927 patch_prober.go:29] interesting pod/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-t762r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Oct 30 23:57:34.211353 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:34.207693 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-t762r" podUID="b9e9d9b1-22af-48d6-b1de-b1c28107ace1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Oct 30 23:57:34.405629 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:34.405600 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1844f1a-t6tp6" Oct 30 23:57:34.496444 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:34.496307 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af1791d5-2960-42bd-95a8-6f6b0ef78e93-kube-api-access-7nk9j\" (UniqueName: \"kubernetes.io/projected/af1791d5-2960-42bd-95a8-6f6b0ef78e93-kube-api-access-7nk9j\") pod \"af1791d5-2960-42bd-95a8-6f6b0ef78e93\" (UID: \"af1791d5-2960-42bd-95a8-6f6b0ef78e93\") " Oct 30 23:57:34.496444 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:34.496407 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/af1791d5-2960-42bd-95a8-6f6b0ef78e93-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af1791d5-2960-42bd-95a8-6f6b0ef78e93-maas-api-xks-ca\") pod \"af1791d5-2960-42bd-95a8-6f6b0ef78e93\" (UID: \"af1791d5-2960-42bd-95a8-6f6b0ef78e93\") " Oct 30 23:57:34.496636 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:34.496476 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af1791d5-2960-42bd-95a8-6f6b0ef78e93-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af1791d5-2960-42bd-95a8-6f6b0ef78e93-maas-api-openshift-ca\") pod \"af1791d5-2960-42bd-95a8-6f6b0ef78e93\" (UID: \"af1791d5-2960-42bd-95a8-6f6b0ef78e93\") " Oct 30 23:57:34.496888 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:34.496861 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af1791d5-2960-42bd-95a8-6f6b0ef78e93-maas-api-openshift-ca" pod "af1791d5-2960-42bd-95a8-6f6b0ef78e93" (UID: "af1791d5-2960-42bd-95a8-6f6b0ef78e93"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:57:34.497010 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:34.496991 4927 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af1791d5-2960-42bd-95a8-6f6b0ef78e93-maas-api-openshift-ca\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 23:57:34.498703 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:34.498679 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af1791d5-2960-42bd-95a8-6f6b0ef78e93-kube-api-access-7nk9j" pod "af1791d5-2960-42bd-95a8-6f6b0ef78e93" (UID: "af1791d5-2960-42bd-95a8-6f6b0ef78e93"). InnerVolumeSpecName "kube-api-access-7nk9j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:57:34.498818 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:34.498743 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af1791d5-2960-42bd-95a8-6f6b0ef78e93-maas-api-xks-ca" pod "af1791d5-2960-42bd-95a8-6f6b0ef78e93" (UID: "af1791d5-2960-42bd-95a8-6f6b0ef78e93"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 23:57:34.598446 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:34.598406 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7nk9j\" (UniqueName: \"kubernetes.io/projected/af1791d5-2960-42bd-95a8-6f6b0ef78e93-kube-api-access-7nk9j\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 23:57:34.598446 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:34.598444 4927 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af1791d5-2960-42bd-95a8-6f6b0ef78e93-maas-api-xks-ca\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 23:57:34.660353 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:34.659745 4927 patch_prober.go:29] interesting pod/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-6gwbm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Oct 30 23:57:34.660353 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:34.659809 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-6gwbm" podUID="c4095540-8f80-4704-9146-06c5d2cab638" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Oct 30 23:57:34.673368 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:34.673325 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1844f1a-t6tp6" Oct 30 23:57:34.673368 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:34.673360 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1844f1a-t6tp6" event={"ID":"af1791d5-2960-42bd-95a8-6f6b0ef78e93","Type":"ContainerDied","Data":"626a731fe954ae75fb31622dc6704e9aa02d8d86d79776577d64d72f3b445d16"} Oct 30 23:57:34.673564 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:34.673410 4927 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="626a731fe954ae75fb31622dc6704e9aa02d8d86d79776577d64d72f3b445d16" Oct 30 23:57:34.827414 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:34.820312 4927 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-944gh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Oct 30 23:57:34.827414 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:34.820402 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-944gh" podUID="91f11663-5773-418a-bde2-532ead3c628c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Oct 30 23:57:35.068357 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:35.066774 4927 patch_prober.go:29] interesting pod/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-t762r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Oct 30 23:57:35.068357 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:35.066840 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-t762r" podUID="b9e9d9b1-22af-48d6-b1de-b1c28107ace1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Oct 30 23:57:35.658614 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:35.658580 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-6gwbm" Oct 30 23:57:35.659472 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:35.659419 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-6gwbm" Oct 30 23:57:35.660835 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:35.660813 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-6gwbm" Oct 30 23:57:35.816773 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:35.816737 4927 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-944gh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Oct 30 23:57:35.817041 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:35.817012 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-944gh" podUID="91f11663-5773-418a-bde2-532ead3c628c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Oct 30 23:57:36.072846 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:36.072750 4927 patch_prober.go:29] interesting pod/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-t762r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Oct 30 23:57:36.073020 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:36.072871 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-t762r" podUID="b9e9d9b1-22af-48d6-b1de-b1c28107ace1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Oct 30 23:57:36.820356 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:36.817640 4927 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-944gh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Oct 30 23:57:36.820356 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:36.817706 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-944gh" podUID="91f11663-5773-418a-bde2-532ead3c628c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Oct 30 23:57:36.952856 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:36.952821 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:57:37.159768 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:37.159736 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-t762r" Oct 30 23:57:37.160530 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:37.160507 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-t762r" Oct 30 23:57:37.161533 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:37.161506 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-t762r" Oct 30 23:57:37.829935 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:37.829893 4927 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-944gh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Oct 30 23:57:37.830513 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:37.830479 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-944gh" podUID="91f11663-5773-418a-bde2-532ead3c628c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Oct 30 23:57:38.813903 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:38.813862 4927 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-944gh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Oct 30 23:57:38.814086 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:38.813932 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-944gh" podUID="91f11663-5773-418a-bde2-532ead3c628c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Oct 30 23:57:39.813678 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:39.813642 4927 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-944gh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Oct 30 23:57:39.814074 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:39.813712 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-944gh" podUID="91f11663-5773-418a-bde2-532ead3c628c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Oct 30 23:57:40.818036 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:40.818001 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-944gh" Oct 30 23:57:40.818558 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:40.818461 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-944gh" Oct 30 23:57:40.819226 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:40.819205 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-944gh" Oct 30 23:57:41.585405 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:41.585370 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:57:41.894235 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:41.894158 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3e0479e-2vfsd"] Oct 30 23:57:41.894721 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:41.894706 4927 cpu_manager.go:414] "containerMap: removing container" podUID="af1791d5-2960-42bd-95a8-6f6b0ef78e93" containerName="revoke-keys" Oct 30 23:57:41.894761 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:41.894723 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af1791d5-2960-42bd-95a8-6f6b0ef78e93" containerName="revoke-keys" Oct 30 23:57:41.896375 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:41.895559 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="af1791d5-2960-42bd-95a8-6f6b0ef78e93" containerName="revoke-keys" Oct 30 23:57:41.901950 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:41.901926 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3e0479e-2vfsd" Oct 30 23:57:41.917746 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:41.917716 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1354a346-145b-495d-84ea-c9d92967d477-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3e0479e-2vfsd\" (UID: \"1354a346-145b-495d-84ea-c9d92967d477\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3e0479e-2vfsd" Oct 30 23:57:41.917926 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:41.917904 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1354a346-145b-495d-84ea-c9d92967d477-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3e0479e-2vfsd\" (UID: \"1354a346-145b-495d-84ea-c9d92967d477\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3e0479e-2vfsd" Oct 30 23:57:41.917980 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:41.917968 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v7nxq\" (UniqueName: \"kubernetes.io/projected/1354a346-145b-495d-84ea-c9d92967d477-kube-api-access-v7nxq\") pod \"maas-api-revoke-sub-d3e0479e-2vfsd\" (UID: \"1354a346-145b-495d-84ea-c9d92967d477\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3e0479e-2vfsd" Oct 30 23:57:41.956754 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:41.956676 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4fdzs\"" Oct 30 23:57:41.956754 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:41.956725 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 23:57:41.957025 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:41.956999 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3e0479e-2vfsd"] Oct 30 23:57:42.019941 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:42.019897 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1354a346-145b-495d-84ea-c9d92967d477-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3e0479e-2vfsd\" (UID: \"1354a346-145b-495d-84ea-c9d92967d477\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3e0479e-2vfsd" Oct 30 23:57:42.020118 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:42.019999 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7nxq\" (UniqueName: \"kubernetes.io/projected/1354a346-145b-495d-84ea-c9d92967d477-kube-api-access-v7nxq\") pod \"maas-api-revoke-sub-d3e0479e-2vfsd\" (UID: \"1354a346-145b-495d-84ea-c9d92967d477\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3e0479e-2vfsd" Oct 30 23:57:42.020118 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:42.020056 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1354a346-145b-495d-84ea-c9d92967d477-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3e0479e-2vfsd\" (UID: \"1354a346-145b-495d-84ea-c9d92967d477\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3e0479e-2vfsd" Oct 30 23:57:42.020770 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:42.020734 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1354a346-145b-495d-84ea-c9d92967d477-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3e0479e-2vfsd\" (UID: \"1354a346-145b-495d-84ea-c9d92967d477\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3e0479e-2vfsd" Oct 30 23:57:42.022555 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:42.022535 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1354a346-145b-495d-84ea-c9d92967d477-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3e0479e-2vfsd\" (UID: \"1354a346-145b-495d-84ea-c9d92967d477\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3e0479e-2vfsd" Oct 30 23:57:42.083401 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:42.083364 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7nxq\" (UniqueName: \"kubernetes.io/projected/1354a346-145b-495d-84ea-c9d92967d477-kube-api-access-v7nxq\") pod \"maas-api-revoke-sub-d3e0479e-2vfsd\" (UID: \"1354a346-145b-495d-84ea-c9d92967d477\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3e0479e-2vfsd" Oct 30 23:57:42.218200 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:42.218161 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3e0479e-2vfsd" Oct 30 23:57:42.601097 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:57:42.601055 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1354a346_145b_495d_84ea_c9d92967d477.slice/crio-976e7db6aa9a5bb40480864ceec2b35ce20da0adf9efd53e805af0e71ac46aab WatchSource:0}: Error finding container 976e7db6aa9a5bb40480864ceec2b35ce20da0adf9efd53e805af0e71ac46aab: Status 404 returned error can't find the container with id 976e7db6aa9a5bb40480864ceec2b35ce20da0adf9efd53e805af0e71ac46aab Oct 30 23:57:42.601227 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:42.601191 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3e0479e-2vfsd"] Oct 30 23:57:42.751730 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:42.751694 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3e0479e-2vfsd" event={"ID":"1354a346-145b-495d-84ea-c9d92967d477","Type":"ContainerStarted","Data":"976e7db6aa9a5bb40480864ceec2b35ce20da0adf9efd53e805af0e71ac46aab"} Oct 30 23:57:43.163467 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:43.163432 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-nr4qz"] Oct 30 23:57:43.167604 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:43.167559 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-nr4qz" Oct 30 23:57:43.233150 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:43.233111 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/333d9301-4a9b-4867-b3d9-88d3766d1acb-credential-socket\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-nr4qz\" (UID: \"333d9301-4a9b-4867-b3d9-88d3766d1acb\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-nr4qz" Oct 30 23:57:43.233311 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:43.233213 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/333d9301-4a9b-4867-b3d9-88d3766d1acb-istio-podinfo\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-nr4qz\" (UID: \"333d9301-4a9b-4867-b3d9-88d3766d1acb\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-nr4qz" Oct 30 23:57:43.233311 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:43.233250 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/333d9301-4a9b-4867-b3d9-88d3766d1acb-istio-data\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-nr4qz\" (UID: \"333d9301-4a9b-4867-b3d9-88d3766d1acb\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-nr4qz" Oct 30 23:57:43.233465 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:43.233366 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/333d9301-4a9b-4867-b3d9-88d3766d1acb-workload-certs\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-nr4qz\" (UID: \"333d9301-4a9b-4867-b3d9-88d3766d1acb\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-nr4qz" Oct 30 23:57:43.233465 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:43.233457 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hbr6r\" (UniqueName: \"kubernetes.io/projected/333d9301-4a9b-4867-b3d9-88d3766d1acb-kube-api-access-hbr6r\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-nr4qz\" (UID: \"333d9301-4a9b-4867-b3d9-88d3766d1acb\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-nr4qz" Oct 30 23:57:43.233546 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:43.233484 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/333d9301-4a9b-4867-b3d9-88d3766d1acb-workload-socket\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-nr4qz\" (UID: \"333d9301-4a9b-4867-b3d9-88d3766d1acb\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-nr4qz" Oct 30 23:57:43.233546 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:43.233500 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/333d9301-4a9b-4867-b3d9-88d3766d1acb-istio-token\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-nr4qz\" (UID: \"333d9301-4a9b-4867-b3d9-88d3766d1acb\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-nr4qz" Oct 30 23:57:43.233661 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:43.233645 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/333d9301-4a9b-4867-b3d9-88d3766d1acb-istio-envoy\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-nr4qz\" (UID: \"333d9301-4a9b-4867-b3d9-88d3766d1acb\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-nr4qz" Oct 30 23:57:43.233695 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:43.233687 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/333d9301-4a9b-4867-b3d9-88d3766d1acb-istiod-ca-cert\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-nr4qz\" (UID: \"333d9301-4a9b-4867-b3d9-88d3766d1acb\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-nr4qz" Oct 30 23:57:43.277068 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:43.277035 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-nr4qz"] Oct 30 23:57:43.335315 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:43.335276 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbr6r\" (UniqueName: \"kubernetes.io/projected/333d9301-4a9b-4867-b3d9-88d3766d1acb-kube-api-access-hbr6r\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-nr4qz\" (UID: \"333d9301-4a9b-4867-b3d9-88d3766d1acb\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-nr4qz" Oct 30 23:57:43.335549 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:43.335326 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/333d9301-4a9b-4867-b3d9-88d3766d1acb-workload-socket\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-nr4qz\" (UID: \"333d9301-4a9b-4867-b3d9-88d3766d1acb\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-nr4qz" Oct 30 23:57:43.335549 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:43.335388 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/333d9301-4a9b-4867-b3d9-88d3766d1acb-istio-token\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-nr4qz\" (UID: \"333d9301-4a9b-4867-b3d9-88d3766d1acb\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-nr4qz" Oct 30 23:57:43.335549 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:43.335492 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/333d9301-4a9b-4867-b3d9-88d3766d1acb-istio-envoy\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-nr4qz\" (UID: \"333d9301-4a9b-4867-b3d9-88d3766d1acb\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-nr4qz" Oct 30 23:57:43.335725 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:43.335579 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/333d9301-4a9b-4867-b3d9-88d3766d1acb-istiod-ca-cert\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-nr4qz\" (UID: \"333d9301-4a9b-4867-b3d9-88d3766d1acb\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-nr4qz" Oct 30 23:57:43.335725 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:43.335619 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/333d9301-4a9b-4867-b3d9-88d3766d1acb-credential-socket\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-nr4qz\" (UID: \"333d9301-4a9b-4867-b3d9-88d3766d1acb\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-nr4qz" Oct 30 23:57:43.335725 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:43.335686 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/333d9301-4a9b-4867-b3d9-88d3766d1acb-istio-podinfo\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-nr4qz\" (UID: \"333d9301-4a9b-4867-b3d9-88d3766d1acb\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-nr4qz" Oct 30 23:57:43.335725 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:43.335719 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/333d9301-4a9b-4867-b3d9-88d3766d1acb-istio-data\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-nr4qz\" (UID: \"333d9301-4a9b-4867-b3d9-88d3766d1acb\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-nr4qz" Oct 30 23:57:43.335928 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:43.335747 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/333d9301-4a9b-4867-b3d9-88d3766d1acb-workload-certs\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-nr4qz\" (UID: \"333d9301-4a9b-4867-b3d9-88d3766d1acb\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-nr4qz" Oct 30 23:57:43.335928 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:43.335797 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/333d9301-4a9b-4867-b3d9-88d3766d1acb-workload-socket\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-nr4qz\" (UID: \"333d9301-4a9b-4867-b3d9-88d3766d1acb\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-nr4qz" Oct 30 23:57:43.336029 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:43.335989 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/333d9301-4a9b-4867-b3d9-88d3766d1acb-credential-socket\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-nr4qz\" (UID: \"333d9301-4a9b-4867-b3d9-88d3766d1acb\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-nr4qz" Oct 30 23:57:43.336114 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:43.336057 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/333d9301-4a9b-4867-b3d9-88d3766d1acb-istio-data\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-nr4qz\" (UID: \"333d9301-4a9b-4867-b3d9-88d3766d1acb\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-nr4qz" Oct 30 23:57:43.336176 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:43.336137 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/333d9301-4a9b-4867-b3d9-88d3766d1acb-workload-certs\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-nr4qz\" (UID: \"333d9301-4a9b-4867-b3d9-88d3766d1acb\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-nr4qz" Oct 30 23:57:43.336297 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:43.336277 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/333d9301-4a9b-4867-b3d9-88d3766d1acb-istiod-ca-cert\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-nr4qz\" (UID: \"333d9301-4a9b-4867-b3d9-88d3766d1acb\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-nr4qz" Oct 30 23:57:43.337837 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:43.337805 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/333d9301-4a9b-4867-b3d9-88d3766d1acb-istio-envoy\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-nr4qz\" (UID: \"333d9301-4a9b-4867-b3d9-88d3766d1acb\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-nr4qz" Oct 30 23:57:43.338098 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:43.338081 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/333d9301-4a9b-4867-b3d9-88d3766d1acb-istio-podinfo\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-nr4qz\" (UID: \"333d9301-4a9b-4867-b3d9-88d3766d1acb\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-nr4qz" Oct 30 23:57:43.397722 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:43.397684 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/333d9301-4a9b-4867-b3d9-88d3766d1acb-istio-token\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-nr4qz\" (UID: \"333d9301-4a9b-4867-b3d9-88d3766d1acb\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-nr4qz" Oct 30 23:57:43.397878 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:43.397786 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbr6r\" (UniqueName: \"kubernetes.io/projected/333d9301-4a9b-4867-b3d9-88d3766d1acb-kube-api-access-hbr6r\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-nr4qz\" (UID: \"333d9301-4a9b-4867-b3d9-88d3766d1acb\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-nr4qz" Oct 30 23:57:43.482066 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:43.481986 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-nr4qz" Oct 30 23:57:43.760501 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:43.760427 4927 generic.go:356] "Generic (PLEG): container finished" podID="1354a346-145b-495d-84ea-c9d92967d477" containerID="eb893e33419dee156c59d8629ad9f208247a92d5f7462f9fd06899b9ebae03b7" exitCode=0 Oct 30 23:57:43.760640 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:43.760511 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3e0479e-2vfsd" event={"ID":"1354a346-145b-495d-84ea-c9d92967d477","Type":"ContainerDied","Data":"eb893e33419dee156c59d8629ad9f208247a92d5f7462f9fd06899b9ebae03b7"} Oct 30 23:57:43.920363 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:43.920317 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-nr4qz"] Oct 30 23:57:43.920661 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:57:43.920640 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod333d9301_4a9b_4867_b3d9_88d3766d1acb.slice/crio-0f90f2bb757a8342bd6e605c2931a2c8f35924148215d274f4cce376b9fee438 WatchSource:0}: Error finding container 0f90f2bb757a8342bd6e605c2931a2c8f35924148215d274f4cce376b9fee438: Status 404 returned error can't find the container with id 0f90f2bb757a8342bd6e605c2931a2c8f35924148215d274f4cce376b9fee438 Oct 30 23:57:43.923061 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:43.923031 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 23:57:43.923119 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:43.923089 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 23:57:43.923159 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:43.923148 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 23:57:44.770947 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:44.770905 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-nr4qz" event={"ID":"333d9301-4a9b-4867-b3d9-88d3766d1acb","Type":"ContainerStarted","Data":"2cbee7072e17e752090d8a76ddbb931486f62ce02a86fb77ec088ce1899fa412"} Oct 30 23:57:44.771424 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:44.770970 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-nr4qz" event={"ID":"333d9301-4a9b-4867-b3d9-88d3766d1acb","Type":"ContainerStarted","Data":"0f90f2bb757a8342bd6e605c2931a2c8f35924148215d274f4cce376b9fee438"} Oct 30 23:57:44.896961 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:44.895679 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-nr4qz" podStartSLOduration=1.895659918 podStartE2EDuration="1.895659918s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:57:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:57:44.892743553 +0000 UTC m=+977.929074378" watchObservedRunningTime="2026-10-30 23:57:44.895659918 +0000 UTC m=+977.931990744" Oct 30 23:57:45.044948 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:45.044869 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3e0479e-2vfsd" Oct 30 23:57:45.148501 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:45.148467 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-xsgv7"] Oct 30 23:57:45.149159 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:45.149118 4927 cpu_manager.go:414] "containerMap: removing container" podUID="1354a346-145b-495d-84ea-c9d92967d477" containerName="revoke-keys" Oct 30 23:57:45.149159 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:45.149149 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1354a346-145b-495d-84ea-c9d92967d477" containerName="revoke-keys" Oct 30 23:57:45.149367 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:45.149249 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="1354a346-145b-495d-84ea-c9d92967d477" containerName="revoke-keys" Oct 30 23:57:45.152860 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:45.152833 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-xsgv7" Oct 30 23:57:45.161851 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:45.161787 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1354a346-145b-495d-84ea-c9d92967d477-kube-api-access-v7nxq\" (UniqueName: \"kubernetes.io/projected/1354a346-145b-495d-84ea-c9d92967d477-kube-api-access-v7nxq\") pod \"1354a346-145b-495d-84ea-c9d92967d477\" (UID: \"1354a346-145b-495d-84ea-c9d92967d477\") " Oct 30 23:57:45.162261 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:45.162236 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1354a346-145b-495d-84ea-c9d92967d477-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1354a346-145b-495d-84ea-c9d92967d477-maas-api-openshift-ca\") pod \"1354a346-145b-495d-84ea-c9d92967d477\" (UID: \"1354a346-145b-495d-84ea-c9d92967d477\") " Oct 30 23:57:45.162596 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:45.162572 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1354a346-145b-495d-84ea-c9d92967d477-maas-api-openshift-ca" pod "1354a346-145b-495d-84ea-c9d92967d477" (UID: "1354a346-145b-495d-84ea-c9d92967d477"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:57:45.162744 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:45.162722 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1354a346-145b-495d-84ea-c9d92967d477-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1354a346-145b-495d-84ea-c9d92967d477-maas-api-xks-ca\") pod \"1354a346-145b-495d-84ea-c9d92967d477\" (UID: \"1354a346-145b-495d-84ea-c9d92967d477\") " Oct 30 23:57:45.163535 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:45.163503 4927 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1354a346-145b-495d-84ea-c9d92967d477-maas-api-openshift-ca\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 23:57:45.164087 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:45.164064 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1354a346-145b-495d-84ea-c9d92967d477-kube-api-access-v7nxq" pod "1354a346-145b-495d-84ea-c9d92967d477" (UID: "1354a346-145b-495d-84ea-c9d92967d477"). InnerVolumeSpecName "kube-api-access-v7nxq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:57:45.164273 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:45.164253 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1354a346-145b-495d-84ea-c9d92967d477-maas-api-xks-ca" pod "1354a346-145b-495d-84ea-c9d92967d477" (UID: "1354a346-145b-495d-84ea-c9d92967d477"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 23:57:45.265514 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:45.264930 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/647695af-bfbe-41f4-b395-4d1ff1c0286f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8f6f87f59-xsgv7\" (UID: \"647695af-bfbe-41f4-b395-4d1ff1c0286f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-xsgv7" Oct 30 23:57:45.265514 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:45.265007 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/647695af-bfbe-41f4-b395-4d1ff1c0286f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8f6f87f59-xsgv7\" (UID: \"647695af-bfbe-41f4-b395-4d1ff1c0286f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-xsgv7" Oct 30 23:57:45.265514 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:45.265042 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9k6jb\" (UniqueName: \"kubernetes.io/projected/647695af-bfbe-41f4-b395-4d1ff1c0286f-kube-api-access-9k6jb\") pod \"maas-default-gateway-openshift-default-8f6f87f59-xsgv7\" (UID: \"647695af-bfbe-41f4-b395-4d1ff1c0286f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-xsgv7" Oct 30 23:57:45.265514 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:45.265083 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/647695af-bfbe-41f4-b395-4d1ff1c0286f-workload-certs\") pod \"maas-default-gateway-openshift-default-8f6f87f59-xsgv7\" (UID: \"647695af-bfbe-41f4-b395-4d1ff1c0286f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-xsgv7" Oct 30 23:57:45.265514 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:45.265195 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/647695af-bfbe-41f4-b395-4d1ff1c0286f-credential-socket\") pod \"maas-default-gateway-openshift-default-8f6f87f59-xsgv7\" (UID: \"647695af-bfbe-41f4-b395-4d1ff1c0286f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-xsgv7" Oct 30 23:57:45.265514 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:45.265226 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/647695af-bfbe-41f4-b395-4d1ff1c0286f-istio-envoy\") pod \"maas-default-gateway-openshift-default-8f6f87f59-xsgv7\" (UID: \"647695af-bfbe-41f4-b395-4d1ff1c0286f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-xsgv7" Oct 30 23:57:45.265514 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:45.265277 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/647695af-bfbe-41f4-b395-4d1ff1c0286f-istio-data\") pod \"maas-default-gateway-openshift-default-8f6f87f59-xsgv7\" (UID: \"647695af-bfbe-41f4-b395-4d1ff1c0286f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-xsgv7" Oct 30 23:57:45.265514 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:45.265310 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/647695af-bfbe-41f4-b395-4d1ff1c0286f-istio-token\") pod \"maas-default-gateway-openshift-default-8f6f87f59-xsgv7\" (UID: \"647695af-bfbe-41f4-b395-4d1ff1c0286f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-xsgv7" Oct 30 23:57:45.265514 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:45.265379 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/647695af-bfbe-41f4-b395-4d1ff1c0286f-workload-socket\") pod \"maas-default-gateway-openshift-default-8f6f87f59-xsgv7\" (UID: \"647695af-bfbe-41f4-b395-4d1ff1c0286f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-xsgv7" Oct 30 23:57:45.265514 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:45.265477 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v7nxq\" (UniqueName: \"kubernetes.io/projected/1354a346-145b-495d-84ea-c9d92967d477-kube-api-access-v7nxq\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 23:57:45.265514 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:45.265496 4927 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1354a346-145b-495d-84ea-c9d92967d477-maas-api-xks-ca\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 23:57:45.283821 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:45.283784 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-xsgv7"] Oct 30 23:57:45.345518 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:45.345432 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-wftsq"] Oct 30 23:57:45.350727 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:45.350703 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-wftsq" Oct 30 23:57:45.366659 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:45.366628 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/647695af-bfbe-41f4-b395-4d1ff1c0286f-credential-socket\") pod \"maas-default-gateway-openshift-default-8f6f87f59-xsgv7\" (UID: \"647695af-bfbe-41f4-b395-4d1ff1c0286f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-xsgv7" Oct 30 23:57:45.366659 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:45.366661 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/647695af-bfbe-41f4-b395-4d1ff1c0286f-istio-envoy\") pod \"maas-default-gateway-openshift-default-8f6f87f59-xsgv7\" (UID: \"647695af-bfbe-41f4-b395-4d1ff1c0286f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-xsgv7" Oct 30 23:57:45.366841 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:45.366686 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/647695af-bfbe-41f4-b395-4d1ff1c0286f-istio-data\") pod \"maas-default-gateway-openshift-default-8f6f87f59-xsgv7\" (UID: \"647695af-bfbe-41f4-b395-4d1ff1c0286f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-xsgv7" Oct 30 23:57:45.366841 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:45.366707 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/647695af-bfbe-41f4-b395-4d1ff1c0286f-istio-token\") pod \"maas-default-gateway-openshift-default-8f6f87f59-xsgv7\" (UID: \"647695af-bfbe-41f4-b395-4d1ff1c0286f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-xsgv7" Oct 30 23:57:45.366841 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:45.366749 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/647695af-bfbe-41f4-b395-4d1ff1c0286f-workload-socket\") pod \"maas-default-gateway-openshift-default-8f6f87f59-xsgv7\" (UID: \"647695af-bfbe-41f4-b395-4d1ff1c0286f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-xsgv7" Oct 30 23:57:45.366999 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:45.366844 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/647695af-bfbe-41f4-b395-4d1ff1c0286f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8f6f87f59-xsgv7\" (UID: \"647695af-bfbe-41f4-b395-4d1ff1c0286f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-xsgv7" Oct 30 23:57:45.366999 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:45.366887 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/647695af-bfbe-41f4-b395-4d1ff1c0286f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8f6f87f59-xsgv7\" (UID: \"647695af-bfbe-41f4-b395-4d1ff1c0286f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-xsgv7" Oct 30 23:57:45.366999 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:45.366909 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9k6jb\" (UniqueName: \"kubernetes.io/projected/647695af-bfbe-41f4-b395-4d1ff1c0286f-kube-api-access-9k6jb\") pod \"maas-default-gateway-openshift-default-8f6f87f59-xsgv7\" (UID: \"647695af-bfbe-41f4-b395-4d1ff1c0286f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-xsgv7" Oct 30 23:57:45.366999 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:45.366943 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/647695af-bfbe-41f4-b395-4d1ff1c0286f-workload-certs\") pod \"maas-default-gateway-openshift-default-8f6f87f59-xsgv7\" (UID: \"647695af-bfbe-41f4-b395-4d1ff1c0286f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-xsgv7" Oct 30 23:57:45.367196 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:45.367057 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/647695af-bfbe-41f4-b395-4d1ff1c0286f-istio-data\") pod \"maas-default-gateway-openshift-default-8f6f87f59-xsgv7\" (UID: \"647695af-bfbe-41f4-b395-4d1ff1c0286f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-xsgv7" Oct 30 23:57:45.367196 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:45.367058 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/647695af-bfbe-41f4-b395-4d1ff1c0286f-credential-socket\") pod \"maas-default-gateway-openshift-default-8f6f87f59-xsgv7\" (UID: \"647695af-bfbe-41f4-b395-4d1ff1c0286f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-xsgv7" Oct 30 23:57:45.367305 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:45.367280 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/647695af-bfbe-41f4-b395-4d1ff1c0286f-workload-certs\") pod \"maas-default-gateway-openshift-default-8f6f87f59-xsgv7\" (UID: \"647695af-bfbe-41f4-b395-4d1ff1c0286f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-xsgv7" Oct 30 23:57:45.367663 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:45.367637 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/647695af-bfbe-41f4-b395-4d1ff1c0286f-workload-socket\") pod \"maas-default-gateway-openshift-default-8f6f87f59-xsgv7\" (UID: \"647695af-bfbe-41f4-b395-4d1ff1c0286f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-xsgv7" Oct 30 23:57:45.367754 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:45.367737 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/647695af-bfbe-41f4-b395-4d1ff1c0286f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8f6f87f59-xsgv7\" (UID: \"647695af-bfbe-41f4-b395-4d1ff1c0286f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-xsgv7" Oct 30 23:57:45.369382 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:45.369362 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/647695af-bfbe-41f4-b395-4d1ff1c0286f-istio-envoy\") pod \"maas-default-gateway-openshift-default-8f6f87f59-xsgv7\" (UID: \"647695af-bfbe-41f4-b395-4d1ff1c0286f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-xsgv7" Oct 30 23:57:45.369489 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:45.369457 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/647695af-bfbe-41f4-b395-4d1ff1c0286f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8f6f87f59-xsgv7\" (UID: \"647695af-bfbe-41f4-b395-4d1ff1c0286f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-xsgv7" Oct 30 23:57:45.437864 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:45.419429 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-wftsq"] Oct 30 23:57:45.441082 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:45.441050 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9k6jb\" (UniqueName: \"kubernetes.io/projected/647695af-bfbe-41f4-b395-4d1ff1c0286f-kube-api-access-9k6jb\") pod \"maas-default-gateway-openshift-default-8f6f87f59-xsgv7\" (UID: \"647695af-bfbe-41f4-b395-4d1ff1c0286f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-xsgv7" Oct 30 23:57:45.445360 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:45.442076 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/647695af-bfbe-41f4-b395-4d1ff1c0286f-istio-token\") pod \"maas-default-gateway-openshift-default-8f6f87f59-xsgv7\" (UID: \"647695af-bfbe-41f4-b395-4d1ff1c0286f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-xsgv7" Oct 30 23:57:45.468742 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:45.468705 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/222f0528-7bb2-4803-91ed-cda98b5541dc-istio-envoy\") pod \"maas-default-gateway-openshift-default-8f6f87f59-wftsq\" (UID: \"222f0528-7bb2-4803-91ed-cda98b5541dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-wftsq" Oct 30 23:57:45.468882 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:45.468810 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/222f0528-7bb2-4803-91ed-cda98b5541dc-workload-certs\") pod \"maas-default-gateway-openshift-default-8f6f87f59-wftsq\" (UID: \"222f0528-7bb2-4803-91ed-cda98b5541dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-wftsq" Oct 30 23:57:45.468932 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:45.468885 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/222f0528-7bb2-4803-91ed-cda98b5541dc-istio-token\") pod \"maas-default-gateway-openshift-default-8f6f87f59-wftsq\" (UID: \"222f0528-7bb2-4803-91ed-cda98b5541dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-wftsq" Oct 30 23:57:45.468997 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:45.468976 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/222f0528-7bb2-4803-91ed-cda98b5541dc-workload-socket\") pod \"maas-default-gateway-openshift-default-8f6f87f59-wftsq\" (UID: \"222f0528-7bb2-4803-91ed-cda98b5541dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-wftsq" Oct 30 23:57:45.469051 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:45.469033 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9mk79\" (UniqueName: \"kubernetes.io/projected/222f0528-7bb2-4803-91ed-cda98b5541dc-kube-api-access-9mk79\") pod \"maas-default-gateway-openshift-default-8f6f87f59-wftsq\" (UID: \"222f0528-7bb2-4803-91ed-cda98b5541dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-wftsq" Oct 30 23:57:45.469094 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:45.469080 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/222f0528-7bb2-4803-91ed-cda98b5541dc-istio-data\") pod \"maas-default-gateway-openshift-default-8f6f87f59-wftsq\" (UID: \"222f0528-7bb2-4803-91ed-cda98b5541dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-wftsq" Oct 30 23:57:45.469146 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:45.469116 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/222f0528-7bb2-4803-91ed-cda98b5541dc-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8f6f87f59-wftsq\" (UID: \"222f0528-7bb2-4803-91ed-cda98b5541dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-wftsq" Oct 30 23:57:45.469180 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:45.469160 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/222f0528-7bb2-4803-91ed-cda98b5541dc-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8f6f87f59-wftsq\" (UID: \"222f0528-7bb2-4803-91ed-cda98b5541dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-wftsq" Oct 30 23:57:45.469246 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:45.469230 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/222f0528-7bb2-4803-91ed-cda98b5541dc-credential-socket\") pod \"maas-default-gateway-openshift-default-8f6f87f59-wftsq\" (UID: \"222f0528-7bb2-4803-91ed-cda98b5541dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-wftsq" Oct 30 23:57:45.470480 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:45.470459 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-xsgv7" Oct 30 23:57:45.482692 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:45.482674 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-nr4qz" Oct 30 23:57:45.483818 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:45.483795 4927 patch_prober.go:29] interesting pod/e2e-worker-w6-35a187-openshift-default-5b54c65f75-nr4qz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Oct 30 23:57:45.483909 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:45.483838 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-nr4qz" podUID="333d9301-4a9b-4867-b3d9-88d3766d1acb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Oct 30 23:57:45.503900 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:45.503867 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5012722b-8hqgd"] Oct 30 23:57:45.516678 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:45.516650 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5012722b-8hqgd" Oct 30 23:57:45.571920 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:45.568928 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5012722b-8hqgd"] Oct 30 23:57:45.571920 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:45.570456 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/222f0528-7bb2-4803-91ed-cda98b5541dc-workload-socket\") pod \"maas-default-gateway-openshift-default-8f6f87f59-wftsq\" (UID: \"222f0528-7bb2-4803-91ed-cda98b5541dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-wftsq" Oct 30 23:57:45.571920 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:45.570514 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mk79\" (UniqueName: \"kubernetes.io/projected/222f0528-7bb2-4803-91ed-cda98b5541dc-kube-api-access-9mk79\") pod \"maas-default-gateway-openshift-default-8f6f87f59-wftsq\" (UID: \"222f0528-7bb2-4803-91ed-cda98b5541dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-wftsq" Oct 30 23:57:45.571920 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:45.570566 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/222f0528-7bb2-4803-91ed-cda98b5541dc-istio-data\") pod \"maas-default-gateway-openshift-default-8f6f87f59-wftsq\" (UID: \"222f0528-7bb2-4803-91ed-cda98b5541dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-wftsq" Oct 30 23:57:45.571920 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:45.570599 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/222f0528-7bb2-4803-91ed-cda98b5541dc-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8f6f87f59-wftsq\" (UID: \"222f0528-7bb2-4803-91ed-cda98b5541dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-wftsq" Oct 30 23:57:45.571920 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:45.570630 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fgzkn\" (UniqueName: \"kubernetes.io/projected/5bda55b5-e785-43af-874c-de9d032489a5-kube-api-access-fgzkn\") pod \"maas-api-revoke-sub-5012722b-8hqgd\" (UID: \"5bda55b5-e785-43af-874c-de9d032489a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5012722b-8hqgd" Oct 30 23:57:45.571920 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:45.570669 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5bda55b5-e785-43af-874c-de9d032489a5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5012722b-8hqgd\" (UID: \"5bda55b5-e785-43af-874c-de9d032489a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5012722b-8hqgd" Oct 30 23:57:45.571920 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:45.570696 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/222f0528-7bb2-4803-91ed-cda98b5541dc-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8f6f87f59-wftsq\" (UID: \"222f0528-7bb2-4803-91ed-cda98b5541dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-wftsq" Oct 30 23:57:45.571920 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:45.570724 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/222f0528-7bb2-4803-91ed-cda98b5541dc-credential-socket\") pod \"maas-default-gateway-openshift-default-8f6f87f59-wftsq\" (UID: \"222f0528-7bb2-4803-91ed-cda98b5541dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-wftsq" Oct 30 23:57:45.571920 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:45.570957 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/222f0528-7bb2-4803-91ed-cda98b5541dc-istio-envoy\") pod \"maas-default-gateway-openshift-default-8f6f87f59-wftsq\" (UID: \"222f0528-7bb2-4803-91ed-cda98b5541dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-wftsq" Oct 30 23:57:45.571920 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:45.571021 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/222f0528-7bb2-4803-91ed-cda98b5541dc-workload-certs\") pod \"maas-default-gateway-openshift-default-8f6f87f59-wftsq\" (UID: \"222f0528-7bb2-4803-91ed-cda98b5541dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-wftsq" Oct 30 23:57:45.571920 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:45.571055 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5bda55b5-e785-43af-874c-de9d032489a5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5012722b-8hqgd\" (UID: \"5bda55b5-e785-43af-874c-de9d032489a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5012722b-8hqgd" Oct 30 23:57:45.571920 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:45.571176 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/222f0528-7bb2-4803-91ed-cda98b5541dc-istio-token\") pod \"maas-default-gateway-openshift-default-8f6f87f59-wftsq\" (UID: \"222f0528-7bb2-4803-91ed-cda98b5541dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-wftsq" Oct 30 23:57:45.571920 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:45.571246 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/222f0528-7bb2-4803-91ed-cda98b5541dc-credential-socket\") pod \"maas-default-gateway-openshift-default-8f6f87f59-wftsq\" (UID: \"222f0528-7bb2-4803-91ed-cda98b5541dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-wftsq" Oct 30 23:57:45.571920 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:45.571542 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/222f0528-7bb2-4803-91ed-cda98b5541dc-workload-socket\") pod \"maas-default-gateway-openshift-default-8f6f87f59-wftsq\" (UID: \"222f0528-7bb2-4803-91ed-cda98b5541dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-wftsq" Oct 30 23:57:45.572859 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:45.572056 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/222f0528-7bb2-4803-91ed-cda98b5541dc-istio-data\") pod \"maas-default-gateway-openshift-default-8f6f87f59-wftsq\" (UID: \"222f0528-7bb2-4803-91ed-cda98b5541dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-wftsq" Oct 30 23:57:45.572859 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:45.572816 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/222f0528-7bb2-4803-91ed-cda98b5541dc-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8f6f87f59-wftsq\" (UID: \"222f0528-7bb2-4803-91ed-cda98b5541dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-wftsq" Oct 30 23:57:45.579967 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:45.575819 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/222f0528-7bb2-4803-91ed-cda98b5541dc-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8f6f87f59-wftsq\" (UID: \"222f0528-7bb2-4803-91ed-cda98b5541dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-wftsq" Oct 30 23:57:45.579967 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:45.576008 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/222f0528-7bb2-4803-91ed-cda98b5541dc-workload-certs\") pod \"maas-default-gateway-openshift-default-8f6f87f59-wftsq\" (UID: \"222f0528-7bb2-4803-91ed-cda98b5541dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-wftsq" Oct 30 23:57:45.584530 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:45.584503 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/222f0528-7bb2-4803-91ed-cda98b5541dc-istio-envoy\") pod \"maas-default-gateway-openshift-default-8f6f87f59-wftsq\" (UID: \"222f0528-7bb2-4803-91ed-cda98b5541dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-wftsq" Oct 30 23:57:45.638141 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:45.638032 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/222f0528-7bb2-4803-91ed-cda98b5541dc-istio-token\") pod \"maas-default-gateway-openshift-default-8f6f87f59-wftsq\" (UID: \"222f0528-7bb2-4803-91ed-cda98b5541dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-wftsq" Oct 30 23:57:45.638141 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:45.638094 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mk79\" (UniqueName: \"kubernetes.io/projected/222f0528-7bb2-4803-91ed-cda98b5541dc-kube-api-access-9mk79\") pod \"maas-default-gateway-openshift-default-8f6f87f59-wftsq\" (UID: \"222f0528-7bb2-4803-91ed-cda98b5541dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-wftsq" Oct 30 23:57:45.664056 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:45.664025 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-wftsq" Oct 30 23:57:45.672774 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:45.672749 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5bda55b5-e785-43af-874c-de9d032489a5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5012722b-8hqgd\" (UID: \"5bda55b5-e785-43af-874c-de9d032489a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5012722b-8hqgd" Oct 30 23:57:45.672910 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:45.672891 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgzkn\" (UniqueName: \"kubernetes.io/projected/5bda55b5-e785-43af-874c-de9d032489a5-kube-api-access-fgzkn\") pod \"maas-api-revoke-sub-5012722b-8hqgd\" (UID: \"5bda55b5-e785-43af-874c-de9d032489a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5012722b-8hqgd" Oct 30 23:57:45.672956 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:45.672932 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5bda55b5-e785-43af-874c-de9d032489a5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5012722b-8hqgd\" (UID: \"5bda55b5-e785-43af-874c-de9d032489a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5012722b-8hqgd" Oct 30 23:57:45.673672 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:45.673652 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5bda55b5-e785-43af-874c-de9d032489a5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5012722b-8hqgd\" (UID: \"5bda55b5-e785-43af-874c-de9d032489a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5012722b-8hqgd" Oct 30 23:57:45.675668 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:45.675644 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5bda55b5-e785-43af-874c-de9d032489a5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5012722b-8hqgd\" (UID: \"5bda55b5-e785-43af-874c-de9d032489a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5012722b-8hqgd" Oct 30 23:57:45.738370 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:45.737935 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fgzkn\" (UniqueName: \"kubernetes.io/projected/5bda55b5-e785-43af-874c-de9d032489a5-kube-api-access-fgzkn\") pod \"maas-api-revoke-sub-5012722b-8hqgd\" (UID: \"5bda55b5-e785-43af-874c-de9d032489a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5012722b-8hqgd" Oct 30 23:57:45.781076 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:45.781047 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3e0479e-2vfsd" Oct 30 23:57:45.781501 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:45.781082 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3e0479e-2vfsd" event={"ID":"1354a346-145b-495d-84ea-c9d92967d477","Type":"ContainerDied","Data":"976e7db6aa9a5bb40480864ceec2b35ce20da0adf9efd53e805af0e71ac46aab"} Oct 30 23:57:45.781501 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:45.781119 4927 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="976e7db6aa9a5bb40480864ceec2b35ce20da0adf9efd53e805af0e71ac46aab" Oct 30 23:57:45.848615 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:45.848583 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5012722b-8hqgd" Oct 30 23:57:45.946201 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:45.946171 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-xsgv7"] Oct 30 23:57:45.950022 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:45.949981 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 23:57:45.950520 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:45.950444 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 23:57:45.950520 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:45.950506 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 23:57:46.082999 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:46.082930 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-wftsq"] Oct 30 23:57:46.086488 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:46.086454 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 23:57:46.086581 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:46.086519 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 23:57:46.086581 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:46.086548 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 23:57:46.242850 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:57:46.242800 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5bda55b5_e785_43af_874c_de9d032489a5.slice/crio-64fdb6b17467da65c5be714742f5fad8d619284ccb2af6e646177c650f5ccad2 WatchSource:0}: Error finding container 64fdb6b17467da65c5be714742f5fad8d619284ccb2af6e646177c650f5ccad2: Status 404 returned error can't find the container with id 64fdb6b17467da65c5be714742f5fad8d619284ccb2af6e646177c650f5ccad2 Oct 30 23:57:46.246743 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:46.246719 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5012722b-8hqgd"] Oct 30 23:57:46.289229 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:46.289197 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:57:46.482781 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:46.482752 4927 patch_prober.go:29] interesting pod/e2e-worker-w6-35a187-openshift-default-5b54c65f75-nr4qz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Oct 30 23:57:46.482976 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:46.482957 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-nr4qz" podUID="333d9301-4a9b-4867-b3d9-88d3766d1acb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Oct 30 23:57:46.798061 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:46.797965 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-wftsq" event={"ID":"222f0528-7bb2-4803-91ed-cda98b5541dc","Type":"ContainerStarted","Data":"ce6d91159de4fcc3a26120c66b762524ee35b794fc7ce06ba5449b64309722f9"} Oct 30 23:57:46.798665 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:46.798636 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-wftsq" event={"ID":"222f0528-7bb2-4803-91ed-cda98b5541dc","Type":"ContainerStarted","Data":"1a4bea5f2e143edd36f624a9eed6f3a3e162e0291946363b17632d0366d9cf5e"} Oct 30 23:57:46.803728 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:46.803697 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5012722b-8hqgd" event={"ID":"5bda55b5-e785-43af-874c-de9d032489a5","Type":"ContainerStarted","Data":"edb199ea0829c5898e4eef8b946c683f4e0673fe9a0b509d4287dfb35f56d9a7"} Oct 30 23:57:46.804004 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:46.803969 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5012722b-8hqgd" event={"ID":"5bda55b5-e785-43af-874c-de9d032489a5","Type":"ContainerStarted","Data":"64fdb6b17467da65c5be714742f5fad8d619284ccb2af6e646177c650f5ccad2"} Oct 30 23:57:46.888173 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:46.887665 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-xsgv7" event={"ID":"647695af-bfbe-41f4-b395-4d1ff1c0286f","Type":"ContainerStarted","Data":"e8af9c72a5830de22e6214c1545562b68ddf01f2d153b236bc34644acce466d6"} Oct 30 23:57:46.888173 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:46.887732 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-xsgv7" event={"ID":"647695af-bfbe-41f4-b395-4d1ff1c0286f","Type":"ContainerStarted","Data":"48c868f2a578b03210c7d2ddf4fc25bd077dcd1ae368fa648d2fb2ba1c403a04"} Oct 30 23:57:46.970649 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:46.955667 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-wftsq" podStartSLOduration=1.955646576 podStartE2EDuration="1.955646576s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:57:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:57:46.953836139 +0000 UTC m=+979.990166977" watchObservedRunningTime="2026-10-30 23:57:46.955646576 +0000 UTC m=+979.991977400" Oct 30 23:57:47.098462 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:47.096106 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-xsgv7" podStartSLOduration=2.09608487 podStartE2EDuration="2.09608487s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:57:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:57:47.084841521 +0000 UTC m=+980.121172345" watchObservedRunningTime="2026-10-30 23:57:47.09608487 +0000 UTC m=+980.132415714" Oct 30 23:57:47.205068 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:47.205004 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5012722b-8hqgd" podStartSLOduration=2.204991704 podStartE2EDuration="2.204991704s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:57:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:57:47.203529973 +0000 UTC m=+980.239860795" watchObservedRunningTime="2026-10-30 23:57:47.204991704 +0000 UTC m=+980.241322525" Oct 30 23:57:47.475966 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:47.475493 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-xsgv7" Oct 30 23:57:47.475966 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:47.475623 4927 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-xsgv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Oct 30 23:57:47.475966 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:47.475665 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-xsgv7" podUID="647695af-bfbe-41f4-b395-4d1ff1c0286f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Oct 30 23:57:47.482819 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:47.482699 4927 patch_prober.go:29] interesting pod/e2e-worker-w6-35a187-openshift-default-5b54c65f75-nr4qz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Oct 30 23:57:47.482819 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:47.482758 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-nr4qz" podUID="333d9301-4a9b-4867-b3d9-88d3766d1acb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Oct 30 23:57:47.664515 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:47.664481 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-wftsq" Oct 30 23:57:47.676356 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:47.667352 4927 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-wftsq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Oct 30 23:57:47.676356 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:47.667413 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-wftsq" podUID="222f0528-7bb2-4803-91ed-cda98b5541dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Oct 30 23:57:47.903449 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:47.903412 4927 generic.go:356] "Generic (PLEG): container finished" podID="5bda55b5-e785-43af-874c-de9d032489a5" containerID="edb199ea0829c5898e4eef8b946c683f4e0673fe9a0b509d4287dfb35f56d9a7" exitCode=0 Oct 30 23:57:47.903950 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:47.903502 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5012722b-8hqgd" event={"ID":"5bda55b5-e785-43af-874c-de9d032489a5","Type":"ContainerDied","Data":"edb199ea0829c5898e4eef8b946c683f4e0673fe9a0b509d4287dfb35f56d9a7"} Oct 30 23:57:48.490357 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:48.480861 4927 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-xsgv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Oct 30 23:57:48.490357 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:48.480928 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-xsgv7" podUID="647695af-bfbe-41f4-b395-4d1ff1c0286f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Oct 30 23:57:48.490357 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:48.486755 4927 patch_prober.go:29] interesting pod/e2e-worker-w6-35a187-openshift-default-5b54c65f75-nr4qz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Oct 30 23:57:48.490357 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:48.486811 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-nr4qz" podUID="333d9301-4a9b-4867-b3d9-88d3766d1acb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Oct 30 23:57:48.665850 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:48.665724 4927 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-wftsq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Oct 30 23:57:48.665850 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:48.665790 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-wftsq" podUID="222f0528-7bb2-4803-91ed-cda98b5541dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Oct 30 23:57:49.547407 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:49.533613 4927 patch_prober.go:29] interesting pod/e2e-worker-w6-35a187-openshift-default-5b54c65f75-nr4qz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Oct 30 23:57:49.547407 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:49.533701 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-nr4qz" podUID="333d9301-4a9b-4867-b3d9-88d3766d1acb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Oct 30 23:57:49.547407 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:49.544172 4927 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-xsgv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Oct 30 23:57:49.547407 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:49.544228 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-xsgv7" podUID="647695af-bfbe-41f4-b395-4d1ff1c0286f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Oct 30 23:57:49.669491 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:49.669323 4927 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-wftsq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Oct 30 23:57:49.669491 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:49.669419 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-wftsq" podUID="222f0528-7bb2-4803-91ed-cda98b5541dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Oct 30 23:57:49.773098 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:49.766670 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5012722b-8hqgd" Oct 30 23:57:49.871443 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:49.861866 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5bda55b5-e785-43af-874c-de9d032489a5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5bda55b5-e785-43af-874c-de9d032489a5-maas-api-openshift-ca\") pod \"5bda55b5-e785-43af-874c-de9d032489a5\" (UID: \"5bda55b5-e785-43af-874c-de9d032489a5\") " Oct 30 23:57:49.871443 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:49.862036 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5bda55b5-e785-43af-874c-de9d032489a5-kube-api-access-fgzkn\" (UniqueName: \"kubernetes.io/projected/5bda55b5-e785-43af-874c-de9d032489a5-kube-api-access-fgzkn\") pod \"5bda55b5-e785-43af-874c-de9d032489a5\" (UID: \"5bda55b5-e785-43af-874c-de9d032489a5\") " Oct 30 23:57:49.871443 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:49.862179 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5bda55b5-e785-43af-874c-de9d032489a5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5bda55b5-e785-43af-874c-de9d032489a5-maas-api-xks-ca\") pod \"5bda55b5-e785-43af-874c-de9d032489a5\" (UID: \"5bda55b5-e785-43af-874c-de9d032489a5\") " Oct 30 23:57:49.871443 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:49.867888 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5bda55b5-e785-43af-874c-de9d032489a5-maas-api-openshift-ca" pod "5bda55b5-e785-43af-874c-de9d032489a5" (UID: "5bda55b5-e785-43af-874c-de9d032489a5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:57:49.871443 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:49.868432 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5bda55b5-e785-43af-874c-de9d032489a5-maas-api-xks-ca" pod "5bda55b5-e785-43af-874c-de9d032489a5" (UID: "5bda55b5-e785-43af-874c-de9d032489a5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 23:57:49.871443 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:49.870589 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5bda55b5-e785-43af-874c-de9d032489a5-kube-api-access-fgzkn" pod "5bda55b5-e785-43af-874c-de9d032489a5" (UID: "5bda55b5-e785-43af-874c-de9d032489a5"). InnerVolumeSpecName "kube-api-access-fgzkn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:57:49.970358 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:49.967679 4927 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5bda55b5-e785-43af-874c-de9d032489a5-maas-api-xks-ca\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 23:57:49.970358 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:49.967714 4927 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5bda55b5-e785-43af-874c-de9d032489a5-maas-api-openshift-ca\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 23:57:49.970358 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:49.967732 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fgzkn\" (UniqueName: \"kubernetes.io/projected/5bda55b5-e785-43af-874c-de9d032489a5-kube-api-access-fgzkn\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 23:57:49.996355 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:49.993900 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5012722b-8hqgd" event={"ID":"5bda55b5-e785-43af-874c-de9d032489a5","Type":"ContainerDied","Data":"64fdb6b17467da65c5be714742f5fad8d619284ccb2af6e646177c650f5ccad2"} Oct 30 23:57:49.996355 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:49.993963 4927 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="64fdb6b17467da65c5be714742f5fad8d619284ccb2af6e646177c650f5ccad2" Oct 30 23:57:49.996355 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:49.994033 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5012722b-8hqgd" Oct 30 23:57:50.092364 ip-10-0-141-180 kubenswrapper[4927]: E1030 23:57:50.083778 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sqv69" containerName="extractor" Oct 30 23:57:50.104883 ip-10-0-141-180 kubenswrapper[4927]: E1030 23:57:50.104854 4927 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5bda55b5_e785_43af_874c_de9d032489a5.slice\": RecentStats: unable to find data in memory cache]" Oct 30 23:57:50.179363 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:50.175763 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:57:50.474416 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:50.473525 4927 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-xsgv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Oct 30 23:57:50.474416 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:50.473592 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-xsgv7" podUID="647695af-bfbe-41f4-b395-4d1ff1c0286f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Oct 30 23:57:50.483646 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:50.483601 4927 patch_prober.go:29] interesting pod/e2e-worker-w6-35a187-openshift-default-5b54c65f75-nr4qz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Oct 30 23:57:50.483801 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:50.483686 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-nr4qz" podUID="333d9301-4a9b-4867-b3d9-88d3766d1acb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Oct 30 23:57:50.668382 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:50.667418 4927 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-wftsq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Oct 30 23:57:50.668382 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:50.667488 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-wftsq" podUID="222f0528-7bb2-4803-91ed-cda98b5541dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Oct 30 23:57:50.946377 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:50.940087 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:57:51.488358 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:51.476326 4927 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-xsgv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Oct 30 23:57:51.488358 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:51.476417 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-xsgv7" podUID="647695af-bfbe-41f4-b395-4d1ff1c0286f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Oct 30 23:57:51.503305 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:51.503267 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-nr4qz" Oct 30 23:57:51.504244 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:51.504222 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-nr4qz" Oct 30 23:57:51.505135 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:51.505110 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-nr4qz" Oct 30 23:57:51.670594 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:51.670554 4927 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-wftsq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Oct 30 23:57:51.671103 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:51.670617 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-wftsq" podUID="222f0528-7bb2-4803-91ed-cda98b5541dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Oct 30 23:57:52.471843 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:52.471793 4927 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-xsgv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Oct 30 23:57:52.472062 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:52.471862 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-xsgv7" podUID="647695af-bfbe-41f4-b395-4d1ff1c0286f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Oct 30 23:57:52.657487 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:52.657451 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-18e00928-dv47f"] Oct 30 23:57:52.658087 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:52.658069 4927 cpu_manager.go:414] "containerMap: removing container" podUID="5bda55b5-e785-43af-874c-de9d032489a5" containerName="revoke-keys" Oct 30 23:57:52.658087 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:52.658086 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5bda55b5-e785-43af-874c-de9d032489a5" containerName="revoke-keys" Oct 30 23:57:52.658255 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:52.658209 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="5bda55b5-e785-43af-874c-de9d032489a5" containerName="revoke-keys" Oct 30 23:57:52.662134 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:52.662111 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18e00928-dv47f" Oct 30 23:57:52.664696 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:52.664666 4927 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-wftsq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Oct 30 23:57:52.664811 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:52.664722 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-wftsq" podUID="222f0528-7bb2-4803-91ed-cda98b5541dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Oct 30 23:57:52.696758 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:52.696722 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gpkxq\" (UniqueName: \"kubernetes.io/projected/5f5417b9-0f02-4589-9953-aff04e83f410-kube-api-access-gpkxq\") pod \"maas-api-revoke-sub-18e00928-dv47f\" (UID: \"5f5417b9-0f02-4589-9953-aff04e83f410\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18e00928-dv47f" Oct 30 23:57:52.697187 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:52.696891 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f5417b9-0f02-4589-9953-aff04e83f410-maas-api-xks-ca\") pod \"maas-api-revoke-sub-18e00928-dv47f\" (UID: \"5f5417b9-0f02-4589-9953-aff04e83f410\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18e00928-dv47f" Oct 30 23:57:52.697187 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:52.697077 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f5417b9-0f02-4589-9953-aff04e83f410-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-18e00928-dv47f\" (UID: \"5f5417b9-0f02-4589-9953-aff04e83f410\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18e00928-dv47f" Oct 30 23:57:52.726466 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:52.726071 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-18e00928-dv47f"] Oct 30 23:57:52.766077 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:52.766039 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4fdzs\"" Oct 30 23:57:52.766077 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:52.766041 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 23:57:52.798522 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:52.798474 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f5417b9-0f02-4589-9953-aff04e83f410-maas-api-xks-ca\") pod \"maas-api-revoke-sub-18e00928-dv47f\" (UID: \"5f5417b9-0f02-4589-9953-aff04e83f410\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18e00928-dv47f" Oct 30 23:57:52.798704 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:52.798640 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f5417b9-0f02-4589-9953-aff04e83f410-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-18e00928-dv47f\" (UID: \"5f5417b9-0f02-4589-9953-aff04e83f410\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18e00928-dv47f" Oct 30 23:57:52.798767 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:52.798731 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gpkxq\" (UniqueName: \"kubernetes.io/projected/5f5417b9-0f02-4589-9953-aff04e83f410-kube-api-access-gpkxq\") pod \"maas-api-revoke-sub-18e00928-dv47f\" (UID: \"5f5417b9-0f02-4589-9953-aff04e83f410\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18e00928-dv47f" Oct 30 23:57:52.799400 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:52.799378 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f5417b9-0f02-4589-9953-aff04e83f410-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-18e00928-dv47f\" (UID: \"5f5417b9-0f02-4589-9953-aff04e83f410\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18e00928-dv47f" Oct 30 23:57:52.800960 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:52.800939 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f5417b9-0f02-4589-9953-aff04e83f410-maas-api-xks-ca\") pod \"maas-api-revoke-sub-18e00928-dv47f\" (UID: \"5f5417b9-0f02-4589-9953-aff04e83f410\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18e00928-dv47f" Oct 30 23:57:52.862067 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:52.862025 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gpkxq\" (UniqueName: \"kubernetes.io/projected/5f5417b9-0f02-4589-9953-aff04e83f410-kube-api-access-gpkxq\") pod \"maas-api-revoke-sub-18e00928-dv47f\" (UID: \"5f5417b9-0f02-4589-9953-aff04e83f410\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18e00928-dv47f" Oct 30 23:57:52.980783 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:52.980698 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18e00928-dv47f" Oct 30 23:57:53.407694 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:53.407665 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-18e00928-dv47f"] Oct 30 23:57:53.480526 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:53.472720 4927 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-xsgv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Oct 30 23:57:53.480526 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:53.472780 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-xsgv7" podUID="647695af-bfbe-41f4-b395-4d1ff1c0286f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Oct 30 23:57:53.665123 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:53.665071 4927 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-wftsq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Oct 30 23:57:53.665322 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:53.665155 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-wftsq" podUID="222f0528-7bb2-4803-91ed-cda98b5541dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Oct 30 23:57:54.252110 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:54.252057 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18e00928-dv47f" event={"ID":"5f5417b9-0f02-4589-9953-aff04e83f410","Type":"ContainerStarted","Data":"814de666ae9679c71902461fe5fc2dbfd66068bea2574d8c8731cd408dc111f5"} Oct 30 23:57:54.252721 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:54.252676 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18e00928-dv47f" event={"ID":"5f5417b9-0f02-4589-9953-aff04e83f410","Type":"ContainerStarted","Data":"b542295703577101f6432be53534cb9fd71af67a85b52a2891e908828da51bf7"} Oct 30 23:57:54.403357 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:54.402689 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18e00928-dv47f" podStartSLOduration=2.40267116 podStartE2EDuration="2.40267116s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:57:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:57:54.391854684 +0000 UTC m=+987.428185507" watchObservedRunningTime="2026-10-30 23:57:54.40267116 +0000 UTC m=+987.439001984" Oct 30 23:57:54.485353 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:54.475719 4927 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-xsgv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Oct 30 23:57:54.485353 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:54.475799 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-xsgv7" podUID="647695af-bfbe-41f4-b395-4d1ff1c0286f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Oct 30 23:57:54.675397 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:54.672134 4927 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-wftsq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Oct 30 23:57:54.675397 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:54.672194 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-wftsq" podUID="222f0528-7bb2-4803-91ed-cda98b5541dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Oct 30 23:57:55.313824 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:55.313439 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5c3374/premium-w1-5c3374-kserve-6c56d4dcd-llwcl" podUID="dc813f02-e077-4251-8c1e-fea0986b0f36" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:55.635878 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:55.635763 4927 generic.go:356] "Generic (PLEG): container finished" podID="5f5417b9-0f02-4589-9953-aff04e83f410" containerID="814de666ae9679c71902461fe5fc2dbfd66068bea2574d8c8731cd408dc111f5" exitCode=0 Oct 30 23:57:55.636064 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:55.635920 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18e00928-dv47f" event={"ID":"5f5417b9-0f02-4589-9953-aff04e83f410","Type":"ContainerDied","Data":"814de666ae9679c71902461fe5fc2dbfd66068bea2574d8c8731cd408dc111f5"} Oct 30 23:57:55.727357 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:55.723994 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:57:56.500517 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:56.500469 4927 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-xsgv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 23:57:56.503089 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:56.500549 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-xsgv7" podUID="647695af-bfbe-41f4-b395-4d1ff1c0286f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:56.544241 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:56.544199 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-xsgv7" Oct 30 23:57:56.545271 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:56.545244 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-xsgv7" Oct 30 23:57:56.546031 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:56.546009 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-xsgv7" Oct 30 23:57:56.704656 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:56.704616 4927 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-wftsq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 23:57:56.704842 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:56.704678 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-wftsq" podUID="222f0528-7bb2-4803-91ed-cda98b5541dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:56.764396 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:56.763903 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-wftsq" Oct 30 23:57:56.767371 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:56.765132 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-wftsq" Oct 30 23:57:56.767371 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:56.766658 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-wftsq" Oct 30 23:57:56.863956 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:56.863928 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18e00928-dv47f" Oct 30 23:57:56.987470 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:56.987442 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f5417b9-0f02-4589-9953-aff04e83f410-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f5417b9-0f02-4589-9953-aff04e83f410-maas-api-openshift-ca\") pod \"5f5417b9-0f02-4589-9953-aff04e83f410\" (UID: \"5f5417b9-0f02-4589-9953-aff04e83f410\") " Oct 30 23:57:56.987675 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:56.987543 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5f5417b9-0f02-4589-9953-aff04e83f410-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f5417b9-0f02-4589-9953-aff04e83f410-maas-api-xks-ca\") pod \"5f5417b9-0f02-4589-9953-aff04e83f410\" (UID: \"5f5417b9-0f02-4589-9953-aff04e83f410\") " Oct 30 23:57:56.987675 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:56.987635 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f5417b9-0f02-4589-9953-aff04e83f410-kube-api-access-gpkxq\" (UniqueName: \"kubernetes.io/projected/5f5417b9-0f02-4589-9953-aff04e83f410-kube-api-access-gpkxq\") pod \"5f5417b9-0f02-4589-9953-aff04e83f410\" (UID: \"5f5417b9-0f02-4589-9953-aff04e83f410\") " Oct 30 23:57:56.987880 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:56.987855 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f5417b9-0f02-4589-9953-aff04e83f410-maas-api-openshift-ca" pod "5f5417b9-0f02-4589-9953-aff04e83f410" (UID: "5f5417b9-0f02-4589-9953-aff04e83f410"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:57:56.988064 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:56.988040 4927 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f5417b9-0f02-4589-9953-aff04e83f410-maas-api-openshift-ca\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 23:57:56.989660 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:56.989635 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f5417b9-0f02-4589-9953-aff04e83f410-maas-api-xks-ca" pod "5f5417b9-0f02-4589-9953-aff04e83f410" (UID: "5f5417b9-0f02-4589-9953-aff04e83f410"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 23:57:56.989767 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:56.989730 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f5417b9-0f02-4589-9953-aff04e83f410-kube-api-access-gpkxq" pod "5f5417b9-0f02-4589-9953-aff04e83f410" (UID: "5f5417b9-0f02-4589-9953-aff04e83f410"). InnerVolumeSpecName "kube-api-access-gpkxq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:57:57.089042 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:57.088967 4927 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f5417b9-0f02-4589-9953-aff04e83f410-maas-api-xks-ca\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 23:57:57.089042 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:57.089003 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gpkxq\" (UniqueName: \"kubernetes.io/projected/5f5417b9-0f02-4589-9953-aff04e83f410-kube-api-access-gpkxq\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 23:57:57.656921 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:57.656888 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18e00928-dv47f" Oct 30 23:57:57.657429 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:57.656921 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18e00928-dv47f" event={"ID":"5f5417b9-0f02-4589-9953-aff04e83f410","Type":"ContainerDied","Data":"b542295703577101f6432be53534cb9fd71af67a85b52a2891e908828da51bf7"} Oct 30 23:57:57.657429 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:57:57.656973 4927 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b542295703577101f6432be53534cb9fd71af67a85b52a2891e908828da51bf7" Oct 30 23:58:00.302196 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:58:00.302164 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:58:04.892647 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:58:04.892607 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:58:09.497564 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:58:09.497522 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:58:12.889026 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:58:12.888988 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:58:12.986728 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:58:12.986689 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-2qnns"] Oct 30 23:58:12.988146 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:58:12.988116 4927 cpu_manager.go:414] "containerMap: removing container" podUID="5f5417b9-0f02-4589-9953-aff04e83f410" containerName="revoke-keys" Oct 30 23:58:12.988409 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:58:12.988305 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f5417b9-0f02-4589-9953-aff04e83f410" containerName="revoke-keys" Oct 30 23:58:12.988720 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:58:12.988701 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f5417b9-0f02-4589-9953-aff04e83f410" containerName="revoke-keys" Oct 30 23:58:13.011355 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:58:12.995574 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-2qnns" Oct 30 23:58:13.102918 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:58:13.102884 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7194267-cfef-4252-ae16-a194eccd2a1f-credential-socket\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-2qnns\" (UID: \"b7194267-cfef-4252-ae16-a194eccd2a1f\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-2qnns" Oct 30 23:58:13.102918 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:58:13.102931 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7194267-cfef-4252-ae16-a194eccd2a1f-istio-data\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-2qnns\" (UID: \"b7194267-cfef-4252-ae16-a194eccd2a1f\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-2qnns" Oct 30 23:58:13.103193 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:58:13.103022 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7194267-cfef-4252-ae16-a194eccd2a1f-istio-envoy\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-2qnns\" (UID: \"b7194267-cfef-4252-ae16-a194eccd2a1f\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-2qnns" Oct 30 23:58:13.103193 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:58:13.103043 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7194267-cfef-4252-ae16-a194eccd2a1f-istiod-ca-cert\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-2qnns\" (UID: \"b7194267-cfef-4252-ae16-a194eccd2a1f\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-2qnns" Oct 30 23:58:13.103193 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:58:13.103078 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7194267-cfef-4252-ae16-a194eccd2a1f-workload-certs\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-2qnns\" (UID: \"b7194267-cfef-4252-ae16-a194eccd2a1f\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-2qnns" Oct 30 23:58:13.103375 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:58:13.103225 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7194267-cfef-4252-ae16-a194eccd2a1f-workload-socket\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-2qnns\" (UID: \"b7194267-cfef-4252-ae16-a194eccd2a1f\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-2qnns" Oct 30 23:58:13.103375 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:58:13.103307 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fp7p6\" (UniqueName: \"kubernetes.io/projected/b7194267-cfef-4252-ae16-a194eccd2a1f-kube-api-access-fp7p6\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-2qnns\" (UID: \"b7194267-cfef-4252-ae16-a194eccd2a1f\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-2qnns" Oct 30 23:58:13.103496 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:58:13.103388 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7194267-cfef-4252-ae16-a194eccd2a1f-istio-token\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-2qnns\" (UID: \"b7194267-cfef-4252-ae16-a194eccd2a1f\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-2qnns" Oct 30 23:58:13.103496 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:58:13.103415 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7194267-cfef-4252-ae16-a194eccd2a1f-istio-podinfo\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-2qnns\" (UID: \"b7194267-cfef-4252-ae16-a194eccd2a1f\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-2qnns" Oct 30 23:58:13.107759 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:58:13.107733 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-2qnns"] Oct 30 23:58:13.217111 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:58:13.217071 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7194267-cfef-4252-ae16-a194eccd2a1f-credential-socket\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-2qnns\" (UID: \"b7194267-cfef-4252-ae16-a194eccd2a1f\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-2qnns" Oct 30 23:58:13.217879 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:58:13.217853 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7194267-cfef-4252-ae16-a194eccd2a1f-istio-data\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-2qnns\" (UID: \"b7194267-cfef-4252-ae16-a194eccd2a1f\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-2qnns" Oct 30 23:58:13.218090 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:58:13.218070 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7194267-cfef-4252-ae16-a194eccd2a1f-istio-envoy\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-2qnns\" (UID: \"b7194267-cfef-4252-ae16-a194eccd2a1f\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-2qnns" Oct 30 23:58:13.218211 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:58:13.218197 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7194267-cfef-4252-ae16-a194eccd2a1f-istiod-ca-cert\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-2qnns\" (UID: \"b7194267-cfef-4252-ae16-a194eccd2a1f\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-2qnns" Oct 30 23:58:13.218377 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:58:13.218361 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7194267-cfef-4252-ae16-a194eccd2a1f-workload-certs\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-2qnns\" (UID: \"b7194267-cfef-4252-ae16-a194eccd2a1f\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-2qnns" Oct 30 23:58:13.218625 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:58:13.218606 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7194267-cfef-4252-ae16-a194eccd2a1f-workload-socket\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-2qnns\" (UID: \"b7194267-cfef-4252-ae16-a194eccd2a1f\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-2qnns" Oct 30 23:58:13.218787 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:58:13.218771 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fp7p6\" (UniqueName: \"kubernetes.io/projected/b7194267-cfef-4252-ae16-a194eccd2a1f-kube-api-access-fp7p6\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-2qnns\" (UID: \"b7194267-cfef-4252-ae16-a194eccd2a1f\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-2qnns" Oct 30 23:58:13.218970 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:58:13.218953 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7194267-cfef-4252-ae16-a194eccd2a1f-istio-token\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-2qnns\" (UID: \"b7194267-cfef-4252-ae16-a194eccd2a1f\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-2qnns" Oct 30 23:58:13.219110 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:58:13.219094 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7194267-cfef-4252-ae16-a194eccd2a1f-istio-podinfo\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-2qnns\" (UID: \"b7194267-cfef-4252-ae16-a194eccd2a1f\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-2qnns" Oct 30 23:58:13.219358 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:58:13.219315 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7194267-cfef-4252-ae16-a194eccd2a1f-istiod-ca-cert\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-2qnns\" (UID: \"b7194267-cfef-4252-ae16-a194eccd2a1f\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-2qnns" Oct 30 23:58:13.220721 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:58:13.218679 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7194267-cfef-4252-ae16-a194eccd2a1f-istio-data\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-2qnns\" (UID: \"b7194267-cfef-4252-ae16-a194eccd2a1f\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-2qnns" Oct 30 23:58:13.220721 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:58:13.219907 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7194267-cfef-4252-ae16-a194eccd2a1f-workload-certs\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-2qnns\" (UID: \"b7194267-cfef-4252-ae16-a194eccd2a1f\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-2qnns" Oct 30 23:58:13.220721 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:58:13.220162 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7194267-cfef-4252-ae16-a194eccd2a1f-workload-socket\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-2qnns\" (UID: \"b7194267-cfef-4252-ae16-a194eccd2a1f\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-2qnns" Oct 30 23:58:13.220721 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:58:13.217766 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7194267-cfef-4252-ae16-a194eccd2a1f-credential-socket\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-2qnns\" (UID: \"b7194267-cfef-4252-ae16-a194eccd2a1f\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-2qnns" Oct 30 23:58:13.232359 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:58:13.230262 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7194267-cfef-4252-ae16-a194eccd2a1f-istio-podinfo\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-2qnns\" (UID: \"b7194267-cfef-4252-ae16-a194eccd2a1f\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-2qnns" Oct 30 23:58:13.248503 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:58:13.247135 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7194267-cfef-4252-ae16-a194eccd2a1f-istio-envoy\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-2qnns\" (UID: \"b7194267-cfef-4252-ae16-a194eccd2a1f\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-2qnns" Oct 30 23:58:13.287660 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:58:13.286866 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7194267-cfef-4252-ae16-a194eccd2a1f-istio-token\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-2qnns\" (UID: \"b7194267-cfef-4252-ae16-a194eccd2a1f\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-2qnns" Oct 30 23:58:13.324357 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:58:13.322771 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fp7p6\" (UniqueName: \"kubernetes.io/projected/b7194267-cfef-4252-ae16-a194eccd2a1f-kube-api-access-fp7p6\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-2qnns\" (UID: \"b7194267-cfef-4252-ae16-a194eccd2a1f\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-2qnns" Oct 30 23:58:13.329586 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:58:13.329552 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-2qnns" Oct 30 23:58:13.783588 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:58:13.783489 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-ccdzj"] Oct 30 23:58:13.790169 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:58:13.790142 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-ccdzj" Oct 30 23:58:13.901406 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:58:13.901163 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-ccdzj"] Oct 30 23:58:13.904059 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:58:13.902749 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/afc963ee-cfd4-441b-9185-a6ae83276e19-istio-podinfo\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-ccdzj\" (UID: \"afc963ee-cfd4-441b-9185-a6ae83276e19\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-ccdzj" Oct 30 23:58:13.904059 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:58:13.902803 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/afc963ee-cfd4-441b-9185-a6ae83276e19-workload-certs\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-ccdzj\" (UID: \"afc963ee-cfd4-441b-9185-a6ae83276e19\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-ccdzj" Oct 30 23:58:13.904059 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:58:13.902840 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/afc963ee-cfd4-441b-9185-a6ae83276e19-istio-data\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-ccdzj\" (UID: \"afc963ee-cfd4-441b-9185-a6ae83276e19\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-ccdzj" Oct 30 23:58:13.904059 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:58:13.902885 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/afc963ee-cfd4-441b-9185-a6ae83276e19-istio-token\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-ccdzj\" (UID: \"afc963ee-cfd4-441b-9185-a6ae83276e19\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-ccdzj" Oct 30 23:58:13.904059 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:58:13.902957 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/afc963ee-cfd4-441b-9185-a6ae83276e19-istio-envoy\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-ccdzj\" (UID: \"afc963ee-cfd4-441b-9185-a6ae83276e19\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-ccdzj" Oct 30 23:58:13.904059 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:58:13.903007 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/afc963ee-cfd4-441b-9185-a6ae83276e19-credential-socket\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-ccdzj\" (UID: \"afc963ee-cfd4-441b-9185-a6ae83276e19\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-ccdzj" Oct 30 23:58:13.904059 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:58:13.903112 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/afc963ee-cfd4-441b-9185-a6ae83276e19-workload-socket\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-ccdzj\" (UID: \"afc963ee-cfd4-441b-9185-a6ae83276e19\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-ccdzj" Oct 30 23:58:13.904059 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:58:13.903151 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/afc963ee-cfd4-441b-9185-a6ae83276e19-istiod-ca-cert\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-ccdzj\" (UID: \"afc963ee-cfd4-441b-9185-a6ae83276e19\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-ccdzj" Oct 30 23:58:13.904059 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:58:13.903216 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zr7sv\" (UniqueName: \"kubernetes.io/projected/afc963ee-cfd4-441b-9185-a6ae83276e19-kube-api-access-zr7sv\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-ccdzj\" (UID: \"afc963ee-cfd4-441b-9185-a6ae83276e19\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-ccdzj" Oct 30 23:58:14.013504 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:58:14.012173 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zr7sv\" (UniqueName: \"kubernetes.io/projected/afc963ee-cfd4-441b-9185-a6ae83276e19-kube-api-access-zr7sv\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-ccdzj\" (UID: \"afc963ee-cfd4-441b-9185-a6ae83276e19\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-ccdzj" Oct 30 23:58:14.013504 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:58:14.012298 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/afc963ee-cfd4-441b-9185-a6ae83276e19-istio-podinfo\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-ccdzj\" (UID: \"afc963ee-cfd4-441b-9185-a6ae83276e19\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-ccdzj" Oct 30 23:58:14.013504 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:58:14.012345 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/afc963ee-cfd4-441b-9185-a6ae83276e19-workload-certs\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-ccdzj\" (UID: \"afc963ee-cfd4-441b-9185-a6ae83276e19\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-ccdzj" Oct 30 23:58:14.013504 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:58:14.012375 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/afc963ee-cfd4-441b-9185-a6ae83276e19-istio-data\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-ccdzj\" (UID: \"afc963ee-cfd4-441b-9185-a6ae83276e19\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-ccdzj" Oct 30 23:58:14.013504 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:58:14.012420 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/afc963ee-cfd4-441b-9185-a6ae83276e19-istio-token\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-ccdzj\" (UID: \"afc963ee-cfd4-441b-9185-a6ae83276e19\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-ccdzj" Oct 30 23:58:14.013504 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:58:14.012491 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/afc963ee-cfd4-441b-9185-a6ae83276e19-istio-envoy\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-ccdzj\" (UID: \"afc963ee-cfd4-441b-9185-a6ae83276e19\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-ccdzj" Oct 30 23:58:14.013504 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:58:14.012533 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/afc963ee-cfd4-441b-9185-a6ae83276e19-credential-socket\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-ccdzj\" (UID: \"afc963ee-cfd4-441b-9185-a6ae83276e19\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-ccdzj" Oct 30 23:58:14.013504 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:58:14.012637 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/afc963ee-cfd4-441b-9185-a6ae83276e19-workload-socket\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-ccdzj\" (UID: \"afc963ee-cfd4-441b-9185-a6ae83276e19\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-ccdzj" Oct 30 23:58:14.013504 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:58:14.012671 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/afc963ee-cfd4-441b-9185-a6ae83276e19-istiod-ca-cert\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-ccdzj\" (UID: \"afc963ee-cfd4-441b-9185-a6ae83276e19\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-ccdzj" Oct 30 23:58:14.015270 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:58:14.015227 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/afc963ee-cfd4-441b-9185-a6ae83276e19-istiod-ca-cert\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-ccdzj\" (UID: \"afc963ee-cfd4-441b-9185-a6ae83276e19\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-ccdzj" Oct 30 23:58:14.015470 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:58:14.015416 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/afc963ee-cfd4-441b-9185-a6ae83276e19-workload-certs\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-ccdzj\" (UID: \"afc963ee-cfd4-441b-9185-a6ae83276e19\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-ccdzj" Oct 30 23:58:14.015788 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:58:14.015678 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/afc963ee-cfd4-441b-9185-a6ae83276e19-istio-data\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-ccdzj\" (UID: \"afc963ee-cfd4-441b-9185-a6ae83276e19\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-ccdzj" Oct 30 23:58:14.016058 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:58:14.016017 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/afc963ee-cfd4-441b-9185-a6ae83276e19-credential-socket\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-ccdzj\" (UID: \"afc963ee-cfd4-441b-9185-a6ae83276e19\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-ccdzj" Oct 30 23:58:14.017022 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:58:14.016988 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/afc963ee-cfd4-441b-9185-a6ae83276e19-workload-socket\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-ccdzj\" (UID: \"afc963ee-cfd4-441b-9185-a6ae83276e19\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-ccdzj" Oct 30 23:58:14.019521 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:58:14.019469 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 23:58:14.020378 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:58:14.020292 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/afc963ee-cfd4-441b-9185-a6ae83276e19-istio-podinfo\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-ccdzj\" (UID: \"afc963ee-cfd4-441b-9185-a6ae83276e19\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-ccdzj" Oct 30 23:58:14.020541 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:58:14.019774 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 23:58:14.020774 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:58:14.020754 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 23:58:14.022976 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:58:14.022912 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/afc963ee-cfd4-441b-9185-a6ae83276e19-istio-envoy\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-ccdzj\" (UID: \"afc963ee-cfd4-441b-9185-a6ae83276e19\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-ccdzj" Oct 30 23:58:14.031025 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:58:14.031005 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-2qnns"] Oct 30 23:58:14.077220 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:58:14.077190 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/afc963ee-cfd4-441b-9185-a6ae83276e19-istio-token\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-ccdzj\" (UID: \"afc963ee-cfd4-441b-9185-a6ae83276e19\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-ccdzj" Oct 30 23:58:14.077387 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:58:14.077369 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zr7sv\" (UniqueName: \"kubernetes.io/projected/afc963ee-cfd4-441b-9185-a6ae83276e19-kube-api-access-zr7sv\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-ccdzj\" (UID: \"afc963ee-cfd4-441b-9185-a6ae83276e19\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-ccdzj" Oct 30 23:58:14.137625 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:58:14.137580 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-ccdzj" Oct 30 23:58:14.239579 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:58:14.239549 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:58:14.652509 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:58:14.652480 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podafc963ee_cfd4_441b_9185_a6ae83276e19.slice/crio-a310c24e096de9af436960158880c9c0cba8034fb08aa825387af6a8df5be438 WatchSource:0}: Error finding container a310c24e096de9af436960158880c9c0cba8034fb08aa825387af6a8df5be438: Status 404 returned error can't find the container with id a310c24e096de9af436960158880c9c0cba8034fb08aa825387af6a8df5be438 Oct 30 23:58:14.655478 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:58:14.655442 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-ccdzj"] Oct 30 23:58:14.656764 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:58:14.656713 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 23:58:14.656855 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:58:14.656825 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 23:58:14.656896 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:58:14.656873 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 23:58:14.839845 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:58:14.839811 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-ccdzj" event={"ID":"afc963ee-cfd4-441b-9185-a6ae83276e19","Type":"ContainerStarted","Data":"51b115498114d0ec225e6a94c5934f19833fa1d6d10f859b7f49762a5b8d07df"} Oct 30 23:58:14.840105 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:58:14.840077 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-ccdzj" event={"ID":"afc963ee-cfd4-441b-9185-a6ae83276e19","Type":"ContainerStarted","Data":"a310c24e096de9af436960158880c9c0cba8034fb08aa825387af6a8df5be438"} Oct 30 23:58:14.847773 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:58:14.847742 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-2qnns" event={"ID":"b7194267-cfef-4252-ae16-a194eccd2a1f","Type":"ContainerStarted","Data":"6616bfd1ea4a480ce02466ce388380eb31ea8e8030af93e453c8d0c7a09224b2"} Oct 30 23:58:14.847773 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:58:14.847786 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-2qnns" event={"ID":"b7194267-cfef-4252-ae16-a194eccd2a1f","Type":"ContainerStarted","Data":"606def46e4d828e7bc4e8fa5db08c5ea941805999bf5b50f32be59dcc6dc51c4"} Oct 30 23:58:14.966596 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:58:14.966528 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-ccdzj" podStartSLOduration=1.966502113 podStartE2EDuration="1.966502113s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:58:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:58:14.962100301 +0000 UTC m=+1007.998431145" watchObservedRunningTime="2026-10-30 23:58:14.966502113 +0000 UTC m=+1008.002832935" Oct 30 23:58:15.101908 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:58:15.101844 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-2qnns" podStartSLOduration=3.101829702 podStartE2EDuration="3.101829702s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:58:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:58:15.097518497 +0000 UTC m=+1008.133849320" watchObservedRunningTime="2026-10-30 23:58:15.101829702 +0000 UTC m=+1008.138160523" Oct 30 23:58:15.142473 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:58:15.142445 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-ccdzj" Oct 30 23:58:15.142736 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:58:15.142720 4927 patch_prober.go:29] interesting pod/e2e-worker-w0-46b586-openshift-default-86c598c96c-ccdzj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Oct 30 23:58:15.142838 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:58:15.142822 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-ccdzj" podUID="afc963ee-cfd4-441b-9185-a6ae83276e19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Oct 30 23:58:15.335670 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:58:15.335525 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-2qnns" Oct 30 23:58:15.335962 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:58:15.335940 4927 patch_prober.go:29] interesting pod/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-2qnns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body= Oct 30 23:58:15.336105 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:58:15.336083 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-2qnns" podUID="b7194267-cfef-4252-ae16-a194eccd2a1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" Oct 30 23:58:16.142356 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:58:16.141861 4927 patch_prober.go:29] interesting pod/e2e-worker-w0-46b586-openshift-default-86c598c96c-ccdzj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Oct 30 23:58:16.142356 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:58:16.141942 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-ccdzj" podUID="afc963ee-cfd4-441b-9185-a6ae83276e19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Oct 30 23:58:16.360406 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:58:16.351771 4927 patch_prober.go:29] interesting pod/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-2qnns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body= Oct 30 23:58:16.360406 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:58:16.351833 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-2qnns" podUID="b7194267-cfef-4252-ae16-a194eccd2a1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" Oct 30 23:58:17.141792 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:58:17.141752 4927 patch_prober.go:29] interesting pod/e2e-worker-w0-46b586-openshift-default-86c598c96c-ccdzj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Oct 30 23:58:17.142709 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:58:17.142678 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-ccdzj" podUID="afc963ee-cfd4-441b-9185-a6ae83276e19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Oct 30 23:58:17.337352 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:58:17.336705 4927 patch_prober.go:29] interesting pod/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-2qnns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body= Oct 30 23:58:17.337352 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:58:17.336767 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-2qnns" podUID="b7194267-cfef-4252-ae16-a194eccd2a1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" Oct 30 23:58:18.149216 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:58:18.148744 4927 patch_prober.go:29] interesting pod/e2e-worker-w0-46b586-openshift-default-86c598c96c-ccdzj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Oct 30 23:58:18.149216 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:58:18.148811 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-ccdzj" podUID="afc963ee-cfd4-441b-9185-a6ae83276e19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Oct 30 23:58:18.334738 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:58:18.334701 4927 patch_prober.go:29] interesting pod/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-2qnns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body= Oct 30 23:58:18.335022 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:58:18.334994 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-2qnns" podUID="b7194267-cfef-4252-ae16-a194eccd2a1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" Oct 30 23:58:18.506658 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:58:18.506620 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:58:18.926359 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:58:18.920276 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:58:19.171475 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:58:19.171438 4927 patch_prober.go:29] interesting pod/e2e-worker-w0-46b586-openshift-default-86c598c96c-ccdzj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Oct 30 23:58:19.172066 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:58:19.172034 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-ccdzj" podUID="afc963ee-cfd4-441b-9185-a6ae83276e19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Oct 30 23:58:19.338717 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:58:19.338622 4927 patch_prober.go:29] interesting pod/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-2qnns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body= Oct 30 23:58:19.338964 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:58:19.338936 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-2qnns" podUID="b7194267-cfef-4252-ae16-a194eccd2a1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" Oct 30 23:58:19.778431 ip-10-0-141-180 kubenswrapper[4927]: E1030 23:58:19.778388 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sqv69" containerName="extractor" Oct 30 23:58:20.148823 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:58:20.148730 4927 patch_prober.go:29] interesting pod/e2e-worker-w0-46b586-openshift-default-86c598c96c-ccdzj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Oct 30 23:58:20.149089 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:58:20.149062 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-ccdzj" podUID="afc963ee-cfd4-441b-9185-a6ae83276e19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Oct 30 23:58:20.346579 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:58:20.346515 4927 patch_prober.go:29] interesting pod/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-2qnns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body= Oct 30 23:58:20.347206 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:58:20.347156 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-2qnns" podUID="b7194267-cfef-4252-ae16-a194eccd2a1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" Oct 30 23:58:21.141414 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:58:21.141368 4927 patch_prober.go:29] interesting pod/e2e-worker-w0-46b586-openshift-default-86c598c96c-ccdzj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Oct 30 23:58:21.141623 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:58:21.141434 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-ccdzj" podUID="afc963ee-cfd4-441b-9185-a6ae83276e19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Oct 30 23:58:21.338635 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:58:21.338604 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-2qnns" Oct 30 23:58:21.338951 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:58:21.338932 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-2qnns" Oct 30 23:58:21.339695 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:58:21.339678 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-2qnns" Oct 30 23:58:22.146008 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:58:22.145975 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-ccdzj" Oct 30 23:58:22.146510 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:58:22.146468 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-ccdzj" Oct 30 23:58:22.147150 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:58:22.147129 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-ccdzj" Oct 30 23:58:22.764069 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:58:22.764027 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:58:23.508812 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:58:23.508770 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:58:24.279390 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:58:24.279312 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:58:28.090323 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:58:28.090271 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:58:32.349946 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:58:32.349910 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:58:32.820129 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:58:32.820092 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:58:34.110443 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:58:34.110404 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:58:34.261972 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:58:34.261929 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:58:37.390394 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:58:37.390287 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:58:42.006723 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:58:42.006239 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:58:43.746681 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:58:43.746641 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:58:46.757981 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:58:46.757943 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:58:49.678404 ip-10-0-141-180 kubenswrapper[4927]: E1030 23:58:49.678367 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sqv69" containerName="extractor" Oct 30 23:58:51.297645 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:58:51.297609 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:58:55.886165 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:58:55.886125 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:58:59.418201 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:58:59.418161 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-d44pq"] Oct 30 23:58:59.424077 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:58:59.424054 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-d44pq" Oct 30 23:58:59.512433 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:58:59.512386 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-d44pq"] Oct 30 23:58:59.535714 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:58:59.535682 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f08dd946-14b1-4546-8fae-01564f3c6908-credential-socket\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-d44pq\" (UID: \"f08dd946-14b1-4546-8fae-01564f3c6908\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-d44pq" Oct 30 23:58:59.535889 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:58:59.535744 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f08dd946-14b1-4546-8fae-01564f3c6908-istiod-ca-cert\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-d44pq\" (UID: \"f08dd946-14b1-4546-8fae-01564f3c6908\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-d44pq" Oct 30 23:58:59.535889 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:58:59.535871 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f08dd946-14b1-4546-8fae-01564f3c6908-istio-envoy\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-d44pq\" (UID: \"f08dd946-14b1-4546-8fae-01564f3c6908\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-d44pq" Oct 30 23:58:59.535970 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:58:59.535929 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f08dd946-14b1-4546-8fae-01564f3c6908-istio-token\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-d44pq\" (UID: \"f08dd946-14b1-4546-8fae-01564f3c6908\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-d44pq" Oct 30 23:58:59.536009 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:58:59.535973 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f08dd946-14b1-4546-8fae-01564f3c6908-workload-socket\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-d44pq\" (UID: \"f08dd946-14b1-4546-8fae-01564f3c6908\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-d44pq" Oct 30 23:58:59.536078 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:58:59.536060 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f08dd946-14b1-4546-8fae-01564f3c6908-istio-podinfo\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-d44pq\" (UID: \"f08dd946-14b1-4546-8fae-01564f3c6908\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-d44pq" Oct 30 23:58:59.536134 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:58:59.536106 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f08dd946-14b1-4546-8fae-01564f3c6908-workload-certs\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-d44pq\" (UID: \"f08dd946-14b1-4546-8fae-01564f3c6908\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-d44pq" Oct 30 23:58:59.536191 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:58:59.536131 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jf9bt\" (UniqueName: \"kubernetes.io/projected/f08dd946-14b1-4546-8fae-01564f3c6908-kube-api-access-jf9bt\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-d44pq\" (UID: \"f08dd946-14b1-4546-8fae-01564f3c6908\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-d44pq" Oct 30 23:58:59.536410 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:58:59.536391 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f08dd946-14b1-4546-8fae-01564f3c6908-istio-data\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-d44pq\" (UID: \"f08dd946-14b1-4546-8fae-01564f3c6908\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-d44pq" Oct 30 23:58:59.637640 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:58:59.637597 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f08dd946-14b1-4546-8fae-01564f3c6908-credential-socket\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-d44pq\" (UID: \"f08dd946-14b1-4546-8fae-01564f3c6908\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-d44pq" Oct 30 23:58:59.637835 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:58:59.637672 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f08dd946-14b1-4546-8fae-01564f3c6908-istiod-ca-cert\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-d44pq\" (UID: \"f08dd946-14b1-4546-8fae-01564f3c6908\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-d44pq" Oct 30 23:58:59.637835 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:58:59.637716 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f08dd946-14b1-4546-8fae-01564f3c6908-istio-envoy\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-d44pq\" (UID: \"f08dd946-14b1-4546-8fae-01564f3c6908\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-d44pq" Oct 30 23:58:59.637835 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:58:59.637762 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f08dd946-14b1-4546-8fae-01564f3c6908-istio-token\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-d44pq\" (UID: \"f08dd946-14b1-4546-8fae-01564f3c6908\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-d44pq" Oct 30 23:58:59.637835 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:58:59.637799 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f08dd946-14b1-4546-8fae-01564f3c6908-workload-socket\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-d44pq\" (UID: \"f08dd946-14b1-4546-8fae-01564f3c6908\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-d44pq" Oct 30 23:58:59.638126 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:58:59.637885 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f08dd946-14b1-4546-8fae-01564f3c6908-istio-podinfo\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-d44pq\" (UID: \"f08dd946-14b1-4546-8fae-01564f3c6908\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-d44pq" Oct 30 23:58:59.638126 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:58:59.637922 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f08dd946-14b1-4546-8fae-01564f3c6908-workload-certs\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-d44pq\" (UID: \"f08dd946-14b1-4546-8fae-01564f3c6908\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-d44pq" Oct 30 23:58:59.638126 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:58:59.637948 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jf9bt\" (UniqueName: \"kubernetes.io/projected/f08dd946-14b1-4546-8fae-01564f3c6908-kube-api-access-jf9bt\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-d44pq\" (UID: \"f08dd946-14b1-4546-8fae-01564f3c6908\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-d44pq" Oct 30 23:58:59.638126 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:58:59.638041 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f08dd946-14b1-4546-8fae-01564f3c6908-istio-data\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-d44pq\" (UID: \"f08dd946-14b1-4546-8fae-01564f3c6908\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-d44pq" Oct 30 23:58:59.638126 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:58:59.638082 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f08dd946-14b1-4546-8fae-01564f3c6908-credential-socket\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-d44pq\" (UID: \"f08dd946-14b1-4546-8fae-01564f3c6908\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-d44pq" Oct 30 23:58:59.638438 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:58:59.638181 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f08dd946-14b1-4546-8fae-01564f3c6908-workload-socket\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-d44pq\" (UID: \"f08dd946-14b1-4546-8fae-01564f3c6908\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-d44pq" Oct 30 23:58:59.638438 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:58:59.638400 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f08dd946-14b1-4546-8fae-01564f3c6908-istio-data\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-d44pq\" (UID: \"f08dd946-14b1-4546-8fae-01564f3c6908\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-d44pq" Oct 30 23:58:59.638438 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:58:59.638419 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f08dd946-14b1-4546-8fae-01564f3c6908-workload-certs\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-d44pq\" (UID: \"f08dd946-14b1-4546-8fae-01564f3c6908\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-d44pq" Oct 30 23:58:59.638616 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:58:59.638593 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f08dd946-14b1-4546-8fae-01564f3c6908-istiod-ca-cert\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-d44pq\" (UID: \"f08dd946-14b1-4546-8fae-01564f3c6908\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-d44pq" Oct 30 23:58:59.640476 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:58:59.640447 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f08dd946-14b1-4546-8fae-01564f3c6908-istio-envoy\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-d44pq\" (UID: \"f08dd946-14b1-4546-8fae-01564f3c6908\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-d44pq" Oct 30 23:58:59.640596 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:58:59.640580 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f08dd946-14b1-4546-8fae-01564f3c6908-istio-podinfo\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-d44pq\" (UID: \"f08dd946-14b1-4546-8fae-01564f3c6908\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-d44pq" Oct 30 23:58:59.702489 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:58:59.702409 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f08dd946-14b1-4546-8fae-01564f3c6908-istio-token\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-d44pq\" (UID: \"f08dd946-14b1-4546-8fae-01564f3c6908\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-d44pq" Oct 30 23:58:59.702669 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:58:59.702647 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jf9bt\" (UniqueName: \"kubernetes.io/projected/f08dd946-14b1-4546-8fae-01564f3c6908-kube-api-access-jf9bt\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-d44pq\" (UID: \"f08dd946-14b1-4546-8fae-01564f3c6908\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-d44pq" Oct 30 23:58:59.741304 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:58:59.741270 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-d44pq" Oct 30 23:59:00.186658 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:59:00.186625 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf08dd946_14b1_4546_8fae_01564f3c6908.slice/crio-5538873b02928199d71fb3aa254fb440d83f678402b4489fcbc65810ce66b646 WatchSource:0}: Error finding container 5538873b02928199d71fb3aa254fb440d83f678402b4489fcbc65810ce66b646: Status 404 returned error can't find the container with id 5538873b02928199d71fb3aa254fb440d83f678402b4489fcbc65810ce66b646 Oct 30 23:59:00.186858 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:59:00.186835 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-d44pq"] Oct 30 23:59:00.189147 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:59:00.189110 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 23:59:00.189232 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:59:00.189175 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 23:59:00.189232 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:59:00.189208 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 23:59:00.366818 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:59:00.366773 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-d44pq" event={"ID":"f08dd946-14b1-4546-8fae-01564f3c6908","Type":"ContainerStarted","Data":"e3b3bb0f5eb795718da9b14e5db0c35736c935970bc957b86792314b0c63c301"} Oct 30 23:59:00.367002 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:59:00.366849 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-d44pq" event={"ID":"f08dd946-14b1-4546-8fae-01564f3c6908","Type":"ContainerStarted","Data":"5538873b02928199d71fb3aa254fb440d83f678402b4489fcbc65810ce66b646"} Oct 30 23:59:00.488455 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:59:00.488367 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:59:00.494294 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:59:00.494247 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-d44pq" podStartSLOduration=1.494233509 podStartE2EDuration="1.494233509s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:58:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:59:00.49065578 +0000 UTC m=+1053.526986601" watchObservedRunningTime="2026-10-30 23:59:00.494233509 +0000 UTC m=+1053.530564331" Oct 30 23:59:00.741886 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:59:00.741784 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-d44pq" Oct 30 23:59:01.141084 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:59:01.140997 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:59:01.743452 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:59:01.743416 4927 patch_prober.go:29] interesting pod/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-d44pq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Oct 30 23:59:01.743979 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:59:01.743475 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-d44pq" podUID="f08dd946-14b1-4546-8fae-01564f3c6908" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Oct 30 23:59:02.742029 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:59:02.741985 4927 patch_prober.go:29] interesting pod/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-d44pq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Oct 30 23:59:02.742229 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:59:02.742058 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-d44pq" podUID="f08dd946-14b1-4546-8fae-01564f3c6908" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Oct 30 23:59:03.742526 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:59:03.742473 4927 patch_prober.go:29] interesting pod/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-d44pq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Oct 30 23:59:03.743080 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:59:03.742549 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-d44pq" podUID="f08dd946-14b1-4546-8fae-01564f3c6908" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Oct 30 23:59:04.741581 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:59:04.741546 4927 patch_prober.go:29] interesting pod/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-d44pq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Oct 30 23:59:04.741774 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:59:04.741608 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-d44pq" podUID="f08dd946-14b1-4546-8fae-01564f3c6908" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Oct 30 23:59:05.120411 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:59:05.120253 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:59:05.741871 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:59:05.741830 4927 patch_prober.go:29] interesting pod/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-d44pq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Oct 30 23:59:05.742039 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:59:05.741892 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-d44pq" podUID="f08dd946-14b1-4546-8fae-01564f3c6908" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Oct 30 23:59:06.742548 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:59:06.742500 4927 patch_prober.go:29] interesting pod/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-d44pq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Oct 30 23:59:06.742948 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:59:06.742577 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-d44pq" podUID="f08dd946-14b1-4546-8fae-01564f3c6908" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Oct 30 23:59:07.915356 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:59:07.913756 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-d44pq" Oct 30 23:59:07.915356 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:59:07.915348 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-d44pq" Oct 30 23:59:07.923625 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:59:07.923599 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-d44pq" Oct 30 23:59:09.790136 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:59:09.790101 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:59:10.003130 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:59:10.003095 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-27e82b/distinct-w2-27e82b-kserve-75f475fcd9-mccsz"] Oct 30 23:59:10.008433 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:59:10.008410 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-27e82b/distinct-w2-27e82b-kserve-75f475fcd9-mccsz" Oct 30 23:59:10.031141 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:59:10.031103 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ba8ced4-e93f-46ac-87c5-3d0b7fadc02d-dshm\") pod \"distinct-w2-27e82b-kserve-75f475fcd9-mccsz\" (UID: \"4ba8ced4-e93f-46ac-87c5-3d0b7fadc02d\") " pod="e2e-models-e2e-worker-w2-27e82b/distinct-w2-27e82b-kserve-75f475fcd9-mccsz" Oct 30 23:59:10.031293 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:59:10.031148 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ba8ced4-e93f-46ac-87c5-3d0b7fadc02d-model-cache\") pod \"distinct-w2-27e82b-kserve-75f475fcd9-mccsz\" (UID: \"4ba8ced4-e93f-46ac-87c5-3d0b7fadc02d\") " pod="e2e-models-e2e-worker-w2-27e82b/distinct-w2-27e82b-kserve-75f475fcd9-mccsz" Oct 30 23:59:10.031293 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:59:10.031188 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ba8ced4-e93f-46ac-87c5-3d0b7fadc02d-tmp-dir\") pod \"distinct-w2-27e82b-kserve-75f475fcd9-mccsz\" (UID: \"4ba8ced4-e93f-46ac-87c5-3d0b7fadc02d\") " pod="e2e-models-e2e-worker-w2-27e82b/distinct-w2-27e82b-kserve-75f475fcd9-mccsz" Oct 30 23:59:10.031293 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:59:10.031226 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hg497\" (UniqueName: \"kubernetes.io/projected/4ba8ced4-e93f-46ac-87c5-3d0b7fadc02d-kube-api-access-hg497\") pod \"distinct-w2-27e82b-kserve-75f475fcd9-mccsz\" (UID: \"4ba8ced4-e93f-46ac-87c5-3d0b7fadc02d\") " pod="e2e-models-e2e-worker-w2-27e82b/distinct-w2-27e82b-kserve-75f475fcd9-mccsz" Oct 30 23:59:10.031511 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:59:10.031383 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ba8ced4-e93f-46ac-87c5-3d0b7fadc02d-home\") pod \"distinct-w2-27e82b-kserve-75f475fcd9-mccsz\" (UID: \"4ba8ced4-e93f-46ac-87c5-3d0b7fadc02d\") " pod="e2e-models-e2e-worker-w2-27e82b/distinct-w2-27e82b-kserve-75f475fcd9-mccsz" Oct 30 23:59:10.031511 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:59:10.031493 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ba8ced4-e93f-46ac-87c5-3d0b7fadc02d-tls-certs\") pod \"distinct-w2-27e82b-kserve-75f475fcd9-mccsz\" (UID: \"4ba8ced4-e93f-46ac-87c5-3d0b7fadc02d\") " pod="e2e-models-e2e-worker-w2-27e82b/distinct-w2-27e82b-kserve-75f475fcd9-mccsz" Oct 30 23:59:10.065272 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:59:10.065201 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-27e82b/distinct-w2-27e82b-kserve-75f475fcd9-mccsz"] Oct 30 23:59:10.098880 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:59:10.098847 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:59:10.110299 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:59:10.110268 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-27e82b\"/\"kube-root-ca.crt\"" Oct 30 23:59:10.114084 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:59:10.114064 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-27e82b\"/\"openshift-service-ca.crt\"" Oct 30 23:59:10.114084 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:59:10.114079 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-27e82b\"/\"distinct-w2-27e82b-kserve-self-signed-certs\"" Oct 30 23:59:10.114235 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:59:10.114067 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-27e82b\"/\"default-dockercfg-bz7t5\"" Oct 30 23:59:10.132655 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:59:10.132633 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ba8ced4-e93f-46ac-87c5-3d0b7fadc02d-tls-certs\") pod \"distinct-w2-27e82b-kserve-75f475fcd9-mccsz\" (UID: \"4ba8ced4-e93f-46ac-87c5-3d0b7fadc02d\") " pod="e2e-models-e2e-worker-w2-27e82b/distinct-w2-27e82b-kserve-75f475fcd9-mccsz" Oct 30 23:59:10.132767 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:59:10.132719 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ba8ced4-e93f-46ac-87c5-3d0b7fadc02d-dshm\") pod \"distinct-w2-27e82b-kserve-75f475fcd9-mccsz\" (UID: \"4ba8ced4-e93f-46ac-87c5-3d0b7fadc02d\") " pod="e2e-models-e2e-worker-w2-27e82b/distinct-w2-27e82b-kserve-75f475fcd9-mccsz" Oct 30 23:59:10.132767 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:59:10.132750 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ba8ced4-e93f-46ac-87c5-3d0b7fadc02d-model-cache\") pod \"distinct-w2-27e82b-kserve-75f475fcd9-mccsz\" (UID: \"4ba8ced4-e93f-46ac-87c5-3d0b7fadc02d\") " pod="e2e-models-e2e-worker-w2-27e82b/distinct-w2-27e82b-kserve-75f475fcd9-mccsz" Oct 30 23:59:10.132885 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:59:10.132786 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ba8ced4-e93f-46ac-87c5-3d0b7fadc02d-tmp-dir\") pod \"distinct-w2-27e82b-kserve-75f475fcd9-mccsz\" (UID: \"4ba8ced4-e93f-46ac-87c5-3d0b7fadc02d\") " pod="e2e-models-e2e-worker-w2-27e82b/distinct-w2-27e82b-kserve-75f475fcd9-mccsz" Oct 30 23:59:10.132885 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:59:10.132820 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hg497\" (UniqueName: \"kubernetes.io/projected/4ba8ced4-e93f-46ac-87c5-3d0b7fadc02d-kube-api-access-hg497\") pod \"distinct-w2-27e82b-kserve-75f475fcd9-mccsz\" (UID: \"4ba8ced4-e93f-46ac-87c5-3d0b7fadc02d\") " pod="e2e-models-e2e-worker-w2-27e82b/distinct-w2-27e82b-kserve-75f475fcd9-mccsz" Oct 30 23:59:10.132992 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:59:10.132968 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ba8ced4-e93f-46ac-87c5-3d0b7fadc02d-home\") pod \"distinct-w2-27e82b-kserve-75f475fcd9-mccsz\" (UID: \"4ba8ced4-e93f-46ac-87c5-3d0b7fadc02d\") " pod="e2e-models-e2e-worker-w2-27e82b/distinct-w2-27e82b-kserve-75f475fcd9-mccsz" Oct 30 23:59:10.133386 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:59:10.133362 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ba8ced4-e93f-46ac-87c5-3d0b7fadc02d-tmp-dir\") pod \"distinct-w2-27e82b-kserve-75f475fcd9-mccsz\" (UID: \"4ba8ced4-e93f-46ac-87c5-3d0b7fadc02d\") " pod="e2e-models-e2e-worker-w2-27e82b/distinct-w2-27e82b-kserve-75f475fcd9-mccsz" Oct 30 23:59:10.133508 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:59:10.133490 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ba8ced4-e93f-46ac-87c5-3d0b7fadc02d-model-cache\") pod \"distinct-w2-27e82b-kserve-75f475fcd9-mccsz\" (UID: \"4ba8ced4-e93f-46ac-87c5-3d0b7fadc02d\") " pod="e2e-models-e2e-worker-w2-27e82b/distinct-w2-27e82b-kserve-75f475fcd9-mccsz" Oct 30 23:59:10.136172 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:59:10.136124 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ba8ced4-e93f-46ac-87c5-3d0b7fadc02d-tls-certs\") pod \"distinct-w2-27e82b-kserve-75f475fcd9-mccsz\" (UID: \"4ba8ced4-e93f-46ac-87c5-3d0b7fadc02d\") " pod="e2e-models-e2e-worker-w2-27e82b/distinct-w2-27e82b-kserve-75f475fcd9-mccsz" Oct 30 23:59:10.146624 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:59:10.146493 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ba8ced4-e93f-46ac-87c5-3d0b7fadc02d-home\") pod \"distinct-w2-27e82b-kserve-75f475fcd9-mccsz\" (UID: \"4ba8ced4-e93f-46ac-87c5-3d0b7fadc02d\") " pod="e2e-models-e2e-worker-w2-27e82b/distinct-w2-27e82b-kserve-75f475fcd9-mccsz" Oct 30 23:59:10.170928 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:59:10.170849 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ba8ced4-e93f-46ac-87c5-3d0b7fadc02d-dshm\") pod \"distinct-w2-27e82b-kserve-75f475fcd9-mccsz\" (UID: \"4ba8ced4-e93f-46ac-87c5-3d0b7fadc02d\") " pod="e2e-models-e2e-worker-w2-27e82b/distinct-w2-27e82b-kserve-75f475fcd9-mccsz" Oct 30 23:59:10.231087 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:59:10.231056 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hg497\" (UniqueName: \"kubernetes.io/projected/4ba8ced4-e93f-46ac-87c5-3d0b7fadc02d-kube-api-access-hg497\") pod \"distinct-w2-27e82b-kserve-75f475fcd9-mccsz\" (UID: \"4ba8ced4-e93f-46ac-87c5-3d0b7fadc02d\") " pod="e2e-models-e2e-worker-w2-27e82b/distinct-w2-27e82b-kserve-75f475fcd9-mccsz" Oct 30 23:59:10.322997 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:59:10.322911 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-27e82b/distinct-w2-27e82b-kserve-75f475fcd9-mccsz" Oct 30 23:59:10.781451 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:59:10.781424 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-27e82b/distinct-w2-27e82b-kserve-75f475fcd9-mccsz"] Oct 30 23:59:10.781574 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:59:10.781555 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4ba8ced4_e93f_46ac_87c5_3d0b7fadc02d.slice/crio-db62b5e71af88671d8e269905bd21aae59d4a08e06fd8d900c901628effa1bf7 WatchSource:0}: Error finding container db62b5e71af88671d8e269905bd21aae59d4a08e06fd8d900c901628effa1bf7: Status 404 returned error can't find the container with id db62b5e71af88671d8e269905bd21aae59d4a08e06fd8d900c901628effa1bf7 Oct 30 23:59:11.541005 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:59:11.540970 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-27e82b/distinct-w2-27e82b-kserve-75f475fcd9-mccsz" event={"ID":"4ba8ced4-e93f-46ac-87c5-3d0b7fadc02d","Type":"ContainerStarted","Data":"823eae6a03e711e6eef4109ae26cf244ea9989cfd22ead0df68af2c8e41e5076"} Oct 30 23:59:11.541005 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:59:11.541011 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-27e82b/distinct-w2-27e82b-kserve-75f475fcd9-mccsz" event={"ID":"4ba8ced4-e93f-46ac-87c5-3d0b7fadc02d","Type":"ContainerStarted","Data":"db62b5e71af88671d8e269905bd21aae59d4a08e06fd8d900c901628effa1bf7"} Oct 30 23:59:11.663076 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:59:11.663022 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-27e82b/distinct-w2-27e82b-kserve-75f475fcd9-mccsz" podStartSLOduration=2.663008896 podStartE2EDuration="2.663008896s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:59:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:59:11.66041225 +0000 UTC m=+1064.696743073" watchObservedRunningTime="2026-10-30 23:59:11.663008896 +0000 UTC m=+1064.699339730" Oct 30 23:59:12.323118 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:59:12.323086 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-27e82b/distinct-w2-27e82b-kserve-75f475fcd9-mccsz" Oct 30 23:59:14.390152 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:59:14.390119 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:59:18.985122 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:59:18.985087 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:59:19.678967 ip-10-0-141-180 kubenswrapper[4927]: E1030 23:59:19.678935 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sqv69" containerName="extractor" Oct 30 23:59:20.324462 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:59:20.323657 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-27e82b/distinct-w2-27e82b-kserve-75f475fcd9-mccsz" Oct 30 23:59:20.337154 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:59:20.337118 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-27e82b/distinct-w2-27e82b-kserve-75f475fcd9-mccsz" Oct 30 23:59:20.627165 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:59:20.627083 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-27e82b/distinct-w2-27e82b-kserve-75f475fcd9-mccsz" Oct 30 23:59:23.124144 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:59:23.124104 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:59:23.306519 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:59:23.306483 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:59:23.679627 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:59:23.679587 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:59:28.300480 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:59:28.300444 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:59:28.976464 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:59:28.976418 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-klh6t"] Oct 30 23:59:28.981050 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:59:28.981026 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-klh6t" Oct 30 23:59:29.099751 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:59:29.099714 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7eb6748-2997-4527-91b3-86900ea70eed-workload-certs\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-klh6t\" (UID: \"b7eb6748-2997-4527-91b3-86900ea70eed\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-klh6t" Oct 30 23:59:29.099937 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:59:29.099786 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7eb6748-2997-4527-91b3-86900ea70eed-istio-token\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-klh6t\" (UID: \"b7eb6748-2997-4527-91b3-86900ea70eed\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-klh6t" Oct 30 23:59:29.099937 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:59:29.099865 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7eb6748-2997-4527-91b3-86900ea70eed-istio-podinfo\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-klh6t\" (UID: \"b7eb6748-2997-4527-91b3-86900ea70eed\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-klh6t" Oct 30 23:59:29.099937 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:59:29.099888 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7eb6748-2997-4527-91b3-86900ea70eed-istio-data\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-klh6t\" (UID: \"b7eb6748-2997-4527-91b3-86900ea70eed\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-klh6t" Oct 30 23:59:29.099937 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:59:29.099915 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlb46\" (UniqueName: \"kubernetes.io/projected/b7eb6748-2997-4527-91b3-86900ea70eed-kube-api-access-dlb46\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-klh6t\" (UID: \"b7eb6748-2997-4527-91b3-86900ea70eed\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-klh6t" Oct 30 23:59:29.100478 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:59:29.099986 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7eb6748-2997-4527-91b3-86900ea70eed-istio-envoy\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-klh6t\" (UID: \"b7eb6748-2997-4527-91b3-86900ea70eed\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-klh6t" Oct 30 23:59:29.100478 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:59:29.100096 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7eb6748-2997-4527-91b3-86900ea70eed-istiod-ca-cert\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-klh6t\" (UID: \"b7eb6748-2997-4527-91b3-86900ea70eed\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-klh6t" Oct 30 23:59:29.100478 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:59:29.100215 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7eb6748-2997-4527-91b3-86900ea70eed-workload-socket\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-klh6t\" (UID: \"b7eb6748-2997-4527-91b3-86900ea70eed\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-klh6t" Oct 30 23:59:29.100478 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:59:29.100260 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7eb6748-2997-4527-91b3-86900ea70eed-credential-socket\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-klh6t\" (UID: \"b7eb6748-2997-4527-91b3-86900ea70eed\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-klh6t" Oct 30 23:59:29.192841 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:59:29.192799 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-klh6t"] Oct 30 23:59:29.202200 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:59:29.202161 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7eb6748-2997-4527-91b3-86900ea70eed-istiod-ca-cert\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-klh6t\" (UID: \"b7eb6748-2997-4527-91b3-86900ea70eed\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-klh6t" Oct 30 23:59:29.202384 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:59:29.202287 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7eb6748-2997-4527-91b3-86900ea70eed-workload-socket\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-klh6t\" (UID: \"b7eb6748-2997-4527-91b3-86900ea70eed\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-klh6t" Oct 30 23:59:29.202384 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:59:29.202346 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7eb6748-2997-4527-91b3-86900ea70eed-credential-socket\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-klh6t\" (UID: \"b7eb6748-2997-4527-91b3-86900ea70eed\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-klh6t" Oct 30 23:59:29.202760 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:59:29.202735 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7eb6748-2997-4527-91b3-86900ea70eed-credential-socket\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-klh6t\" (UID: \"b7eb6748-2997-4527-91b3-86900ea70eed\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-klh6t" Oct 30 23:59:29.202841 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:59:29.202810 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7eb6748-2997-4527-91b3-86900ea70eed-workload-socket\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-klh6t\" (UID: \"b7eb6748-2997-4527-91b3-86900ea70eed\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-klh6t" Oct 30 23:59:29.202960 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:59:29.202944 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7eb6748-2997-4527-91b3-86900ea70eed-workload-certs\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-klh6t\" (UID: \"b7eb6748-2997-4527-91b3-86900ea70eed\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-klh6t" Oct 30 23:59:29.203037 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:59:29.203024 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7eb6748-2997-4527-91b3-86900ea70eed-istio-token\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-klh6t\" (UID: \"b7eb6748-2997-4527-91b3-86900ea70eed\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-klh6t" Oct 30 23:59:29.203099 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:59:29.203037 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7eb6748-2997-4527-91b3-86900ea70eed-istiod-ca-cert\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-klh6t\" (UID: \"b7eb6748-2997-4527-91b3-86900ea70eed\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-klh6t" Oct 30 23:59:29.203150 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:59:29.203111 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7eb6748-2997-4527-91b3-86900ea70eed-istio-podinfo\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-klh6t\" (UID: \"b7eb6748-2997-4527-91b3-86900ea70eed\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-klh6t" Oct 30 23:59:29.203206 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:59:29.203158 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7eb6748-2997-4527-91b3-86900ea70eed-istio-data\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-klh6t\" (UID: \"b7eb6748-2997-4527-91b3-86900ea70eed\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-klh6t" Oct 30 23:59:29.203257 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:59:29.203222 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlb46\" (UniqueName: \"kubernetes.io/projected/b7eb6748-2997-4527-91b3-86900ea70eed-kube-api-access-dlb46\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-klh6t\" (UID: \"b7eb6748-2997-4527-91b3-86900ea70eed\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-klh6t" Oct 30 23:59:29.203310 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:59:29.203263 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7eb6748-2997-4527-91b3-86900ea70eed-istio-envoy\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-klh6t\" (UID: \"b7eb6748-2997-4527-91b3-86900ea70eed\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-klh6t" Oct 30 23:59:29.203401 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:59:29.203358 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7eb6748-2997-4527-91b3-86900ea70eed-workload-certs\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-klh6t\" (UID: \"b7eb6748-2997-4527-91b3-86900ea70eed\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-klh6t" Oct 30 23:59:29.203835 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:59:29.203811 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7eb6748-2997-4527-91b3-86900ea70eed-istio-data\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-klh6t\" (UID: \"b7eb6748-2997-4527-91b3-86900ea70eed\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-klh6t" Oct 30 23:59:29.205496 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:59:29.205475 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7eb6748-2997-4527-91b3-86900ea70eed-istio-envoy\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-klh6t\" (UID: \"b7eb6748-2997-4527-91b3-86900ea70eed\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-klh6t" Oct 30 23:59:29.205996 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:59:29.205977 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7eb6748-2997-4527-91b3-86900ea70eed-istio-podinfo\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-klh6t\" (UID: \"b7eb6748-2997-4527-91b3-86900ea70eed\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-klh6t" Oct 30 23:59:29.267138 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:59:29.267070 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7eb6748-2997-4527-91b3-86900ea70eed-istio-token\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-klh6t\" (UID: \"b7eb6748-2997-4527-91b3-86900ea70eed\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-klh6t" Oct 30 23:59:29.267260 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:59:29.267176 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlb46\" (UniqueName: \"kubernetes.io/projected/b7eb6748-2997-4527-91b3-86900ea70eed-kube-api-access-dlb46\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-klh6t\" (UID: \"b7eb6748-2997-4527-91b3-86900ea70eed\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-klh6t" Oct 30 23:59:29.295023 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:59:29.294996 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-klh6t" Oct 30 23:59:29.736881 ip-10-0-141-180 kubenswrapper[4927]: W1030 23:59:29.736845 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7eb6748_2997_4527_91b3_86900ea70eed.slice/crio-abcc70304f20a388439e4f3cfb51c9c1f49c64350f375da074508b2bbc46560f WatchSource:0}: Error finding container abcc70304f20a388439e4f3cfb51c9c1f49c64350f375da074508b2bbc46560f: Status 404 returned error can't find the container with id abcc70304f20a388439e4f3cfb51c9c1f49c64350f375da074508b2bbc46560f Oct 30 23:59:29.738147 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:59:29.738124 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-klh6t"] Oct 30 23:59:29.739815 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:59:29.739784 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 23:59:29.739891 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:59:29.739845 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 23:59:29.739891 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:59:29.739876 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 23:59:30.741341 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:59:30.741293 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-klh6t" event={"ID":"b7eb6748-2997-4527-91b3-86900ea70eed","Type":"ContainerStarted","Data":"c4717a5f3b503ab1937f5d3e01e2b12f6a6927be6491842a06b68ade45ca38da"} Oct 30 23:59:30.741768 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:59:30.741373 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-klh6t" event={"ID":"b7eb6748-2997-4527-91b3-86900ea70eed","Type":"ContainerStarted","Data":"abcc70304f20a388439e4f3cfb51c9c1f49c64350f375da074508b2bbc46560f"} Oct 30 23:59:30.937921 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:59:30.937846 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-klh6t" podStartSLOduration=2.937825861 podStartE2EDuration="2.937825861s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:59:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:59:30.927838663 +0000 UTC m=+1083.964169521" watchObservedRunningTime="2026-10-30 23:59:30.937825861 +0000 UTC m=+1083.974156684" Oct 30 23:59:31.295686 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:59:31.295643 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-klh6t" Oct 30 23:59:31.297032 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:59:31.297000 4927 patch_prober.go:29] interesting pod/e2e-worker-w6-35a187-openshift-default-5b54c65f75-klh6t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Oct 30 23:59:31.297179 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:59:31.297058 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-klh6t" podUID="b7eb6748-2997-4527-91b3-86900ea70eed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Oct 30 23:59:32.295718 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:59:32.295685 4927 patch_prober.go:29] interesting pod/e2e-worker-w6-35a187-openshift-default-5b54c65f75-klh6t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Oct 30 23:59:32.296124 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:59:32.295743 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-klh6t" podUID="b7eb6748-2997-4527-91b3-86900ea70eed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Oct 30 23:59:33.030357 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:59:33.029744 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:59:33.296208 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:59:33.296120 4927 patch_prober.go:29] interesting pod/e2e-worker-w6-35a187-openshift-default-5b54c65f75-klh6t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Oct 30 23:59:33.296208 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:59:33.296182 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-klh6t" podUID="b7eb6748-2997-4527-91b3-86900ea70eed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Oct 30 23:59:33.339200 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:59:33.339135 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:59:34.296133 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:59:34.296100 4927 patch_prober.go:29] interesting pod/e2e-worker-w6-35a187-openshift-default-5b54c65f75-klh6t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Oct 30 23:59:34.296321 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:59:34.296163 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-klh6t" podUID="b7eb6748-2997-4527-91b3-86900ea70eed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Oct 30 23:59:35.295618 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:59:35.295585 4927 patch_prober.go:29] interesting pod/e2e-worker-w6-35a187-openshift-default-5b54c65f75-klh6t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Oct 30 23:59:35.295792 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:59:35.295641 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-klh6t" podUID="b7eb6748-2997-4527-91b3-86900ea70eed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Oct 30 23:59:36.296079 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:59:36.296040 4927 patch_prober.go:29] interesting pod/e2e-worker-w6-35a187-openshift-default-5b54c65f75-klh6t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Oct 30 23:59:36.296566 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:59:36.296112 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-klh6t" podUID="b7eb6748-2997-4527-91b3-86900ea70eed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Oct 30 23:59:37.299696 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:59:37.299664 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-klh6t" Oct 30 23:59:37.300202 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:59:37.300184 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-klh6t" Oct 30 23:59:37.300887 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:59:37.300866 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-klh6t" Oct 30 23:59:37.722663 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:59:37.722604 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:59:42.281924 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:59:42.281887 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:59:46.933018 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:59:46.932969 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:59:49.679283 ip-10-0-141-180 kubenswrapper[4927]: E1030 23:59:49.679247 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sqv69" containerName="extractor" Oct 30 23:59:51.798973 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:59:51.798918 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:59:56.272542 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:59:56.272503 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 30 23:59:59.568878 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:59:59.568841 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-kzqr4"] Oct 30 23:59:59.573444 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:59:59.573422 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-kzqr4" Oct 30 23:59:59.620130 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:59:59.620096 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-jdcvx"] Oct 30 23:59:59.624580 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:59:59.624558 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-jdcvx" Oct 30 23:59:59.677174 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:59:59.677137 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-kzqr4"] Oct 30 23:59:59.678342 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:59:59.678305 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kz484\" (UniqueName: \"kubernetes.io/projected/a550117e-ad41-4e7c-998d-910a8d13d617-kube-api-access-kz484\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-kzqr4\" (UID: \"a550117e-ad41-4e7c-998d-910a8d13d617\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-kzqr4" Oct 30 23:59:59.678454 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:59:59.678434 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a550117e-ad41-4e7c-998d-910a8d13d617-istio-token\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-kzqr4\" (UID: \"a550117e-ad41-4e7c-998d-910a8d13d617\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-kzqr4" Oct 30 23:59:59.678611 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:59:59.678587 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a550117e-ad41-4e7c-998d-910a8d13d617-istio-envoy\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-kzqr4\" (UID: \"a550117e-ad41-4e7c-998d-910a8d13d617\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-kzqr4" Oct 30 23:59:59.678664 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:59:59.678628 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a550117e-ad41-4e7c-998d-910a8d13d617-istio-podinfo\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-kzqr4\" (UID: \"a550117e-ad41-4e7c-998d-910a8d13d617\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-kzqr4" Oct 30 23:59:59.678836 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:59:59.678811 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a550117e-ad41-4e7c-998d-910a8d13d617-istiod-ca-cert\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-kzqr4\" (UID: \"a550117e-ad41-4e7c-998d-910a8d13d617\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-kzqr4" Oct 30 23:59:59.678937 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:59:59.678915 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a550117e-ad41-4e7c-998d-910a8d13d617-istio-data\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-kzqr4\" (UID: \"a550117e-ad41-4e7c-998d-910a8d13d617\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-kzqr4" Oct 30 23:59:59.678987 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:59:59.678949 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a550117e-ad41-4e7c-998d-910a8d13d617-workload-socket\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-kzqr4\" (UID: \"a550117e-ad41-4e7c-998d-910a8d13d617\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-kzqr4" Oct 30 23:59:59.679022 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:59:59.678992 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a550117e-ad41-4e7c-998d-910a8d13d617-credential-socket\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-kzqr4\" (UID: \"a550117e-ad41-4e7c-998d-910a8d13d617\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-kzqr4" Oct 30 23:59:59.679129 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:59:59.679110 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a550117e-ad41-4e7c-998d-910a8d13d617-workload-certs\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-kzqr4\" (UID: \"a550117e-ad41-4e7c-998d-910a8d13d617\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-kzqr4" Oct 30 23:59:59.731209 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:59:59.731171 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-jdcvx"] Oct 30 23:59:59.780713 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:59:59.780666 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a550117e-ad41-4e7c-998d-910a8d13d617-istio-envoy\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-kzqr4\" (UID: \"a550117e-ad41-4e7c-998d-910a8d13d617\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-kzqr4" Oct 30 23:59:59.780916 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:59:59.780719 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a550117e-ad41-4e7c-998d-910a8d13d617-istio-podinfo\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-kzqr4\" (UID: \"a550117e-ad41-4e7c-998d-910a8d13d617\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-kzqr4" Oct 30 23:59:59.780916 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:59:59.780843 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a520a0b5-deab-4825-9b36-329c399754f6-workload-certs\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-jdcvx\" (UID: \"a520a0b5-deab-4825-9b36-329c399754f6\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-jdcvx" Oct 30 23:59:59.781061 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:59:59.780984 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a520a0b5-deab-4825-9b36-329c399754f6-credential-socket\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-jdcvx\" (UID: \"a520a0b5-deab-4825-9b36-329c399754f6\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-jdcvx" Oct 30 23:59:59.781061 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:59:59.781006 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a520a0b5-deab-4825-9b36-329c399754f6-istio-token\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-jdcvx\" (UID: \"a520a0b5-deab-4825-9b36-329c399754f6\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-jdcvx" Oct 30 23:59:59.781061 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:59:59.781028 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a550117e-ad41-4e7c-998d-910a8d13d617-istiod-ca-cert\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-kzqr4\" (UID: \"a550117e-ad41-4e7c-998d-910a8d13d617\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-kzqr4" Oct 30 23:59:59.781236 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:59:59.781172 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a550117e-ad41-4e7c-998d-910a8d13d617-istio-data\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-kzqr4\" (UID: \"a550117e-ad41-4e7c-998d-910a8d13d617\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-kzqr4" Oct 30 23:59:59.781236 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:59:59.781205 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a520a0b5-deab-4825-9b36-329c399754f6-istio-podinfo\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-jdcvx\" (UID: \"a520a0b5-deab-4825-9b36-329c399754f6\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-jdcvx" Oct 30 23:59:59.781368 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:59:59.781234 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a550117e-ad41-4e7c-998d-910a8d13d617-workload-socket\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-kzqr4\" (UID: \"a550117e-ad41-4e7c-998d-910a8d13d617\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-kzqr4" Oct 30 23:59:59.781368 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:59:59.781258 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a550117e-ad41-4e7c-998d-910a8d13d617-credential-socket\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-kzqr4\" (UID: \"a550117e-ad41-4e7c-998d-910a8d13d617\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-kzqr4" Oct 30 23:59:59.781368 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:59:59.781292 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a520a0b5-deab-4825-9b36-329c399754f6-istio-data\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-jdcvx\" (UID: \"a520a0b5-deab-4825-9b36-329c399754f6\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-jdcvx" Oct 30 23:59:59.781368 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:59:59.781363 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a550117e-ad41-4e7c-998d-910a8d13d617-workload-certs\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-kzqr4\" (UID: \"a550117e-ad41-4e7c-998d-910a8d13d617\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-kzqr4" Oct 30 23:59:59.781586 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:59:59.781419 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kz484\" (UniqueName: \"kubernetes.io/projected/a550117e-ad41-4e7c-998d-910a8d13d617-kube-api-access-kz484\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-kzqr4\" (UID: \"a550117e-ad41-4e7c-998d-910a8d13d617\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-kzqr4" Oct 30 23:59:59.781586 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:59:59.781445 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4cgf\" (UniqueName: \"kubernetes.io/projected/a520a0b5-deab-4825-9b36-329c399754f6-kube-api-access-g4cgf\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-jdcvx\" (UID: \"a520a0b5-deab-4825-9b36-329c399754f6\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-jdcvx" Oct 30 23:59:59.781586 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:59:59.781514 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a550117e-ad41-4e7c-998d-910a8d13d617-istio-token\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-kzqr4\" (UID: \"a550117e-ad41-4e7c-998d-910a8d13d617\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-kzqr4" Oct 30 23:59:59.781586 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:59:59.781556 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a520a0b5-deab-4825-9b36-329c399754f6-istiod-ca-cert\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-jdcvx\" (UID: \"a520a0b5-deab-4825-9b36-329c399754f6\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-jdcvx" Oct 30 23:59:59.781784 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:59:59.781603 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a520a0b5-deab-4825-9b36-329c399754f6-workload-socket\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-jdcvx\" (UID: \"a520a0b5-deab-4825-9b36-329c399754f6\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-jdcvx" Oct 30 23:59:59.781784 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:59:59.781634 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a520a0b5-deab-4825-9b36-329c399754f6-istio-envoy\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-jdcvx\" (UID: \"a520a0b5-deab-4825-9b36-329c399754f6\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-jdcvx" Oct 30 23:59:59.781893 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:59:59.781803 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a550117e-ad41-4e7c-998d-910a8d13d617-istiod-ca-cert\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-kzqr4\" (UID: \"a550117e-ad41-4e7c-998d-910a8d13d617\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-kzqr4" Oct 30 23:59:59.782048 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:59:59.782027 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a550117e-ad41-4e7c-998d-910a8d13d617-istio-data\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-kzqr4\" (UID: \"a550117e-ad41-4e7c-998d-910a8d13d617\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-kzqr4" Oct 30 23:59:59.782048 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:59:59.782043 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a550117e-ad41-4e7c-998d-910a8d13d617-workload-certs\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-kzqr4\" (UID: \"a550117e-ad41-4e7c-998d-910a8d13d617\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-kzqr4" Oct 30 23:59:59.782357 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:59:59.782318 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a550117e-ad41-4e7c-998d-910a8d13d617-credential-socket\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-kzqr4\" (UID: \"a550117e-ad41-4e7c-998d-910a8d13d617\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-kzqr4" Oct 30 23:59:59.782710 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:59:59.782682 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a550117e-ad41-4e7c-998d-910a8d13d617-workload-socket\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-kzqr4\" (UID: \"a550117e-ad41-4e7c-998d-910a8d13d617\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-kzqr4" Oct 30 23:59:59.783598 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:59:59.783573 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a550117e-ad41-4e7c-998d-910a8d13d617-istio-envoy\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-kzqr4\" (UID: \"a550117e-ad41-4e7c-998d-910a8d13d617\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-kzqr4" Oct 30 23:59:59.783598 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:59:59.783592 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a550117e-ad41-4e7c-998d-910a8d13d617-istio-podinfo\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-kzqr4\" (UID: \"a550117e-ad41-4e7c-998d-910a8d13d617\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-kzqr4" Oct 30 23:59:59.844683 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:59:59.844610 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a550117e-ad41-4e7c-998d-910a8d13d617-istio-token\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-kzqr4\" (UID: \"a550117e-ad41-4e7c-998d-910a8d13d617\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-kzqr4" Oct 30 23:59:59.844842 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:59:59.844818 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kz484\" (UniqueName: \"kubernetes.io/projected/a550117e-ad41-4e7c-998d-910a8d13d617-kube-api-access-kz484\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-kzqr4\" (UID: \"a550117e-ad41-4e7c-998d-910a8d13d617\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-kzqr4" Oct 30 23:59:59.883116 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:59:59.883083 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a520a0b5-deab-4825-9b36-329c399754f6-workload-socket\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-jdcvx\" (UID: \"a520a0b5-deab-4825-9b36-329c399754f6\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-jdcvx" Oct 30 23:59:59.883116 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:59:59.883117 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a520a0b5-deab-4825-9b36-329c399754f6-istio-envoy\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-jdcvx\" (UID: \"a520a0b5-deab-4825-9b36-329c399754f6\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-jdcvx" Oct 30 23:59:59.883324 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:59:59.883169 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a520a0b5-deab-4825-9b36-329c399754f6-workload-certs\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-jdcvx\" (UID: \"a520a0b5-deab-4825-9b36-329c399754f6\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-jdcvx" Oct 30 23:59:59.883324 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:59:59.883233 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a520a0b5-deab-4825-9b36-329c399754f6-credential-socket\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-jdcvx\" (UID: \"a520a0b5-deab-4825-9b36-329c399754f6\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-jdcvx" Oct 30 23:59:59.883324 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:59:59.883249 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a520a0b5-deab-4825-9b36-329c399754f6-istio-token\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-jdcvx\" (UID: \"a520a0b5-deab-4825-9b36-329c399754f6\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-jdcvx" Oct 30 23:59:59.883324 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:59:59.883315 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a520a0b5-deab-4825-9b36-329c399754f6-istio-podinfo\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-jdcvx\" (UID: \"a520a0b5-deab-4825-9b36-329c399754f6\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-jdcvx" Oct 30 23:59:59.883537 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:59:59.883360 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a520a0b5-deab-4825-9b36-329c399754f6-istio-data\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-jdcvx\" (UID: \"a520a0b5-deab-4825-9b36-329c399754f6\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-jdcvx" Oct 30 23:59:59.883537 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:59:59.883423 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4cgf\" (UniqueName: \"kubernetes.io/projected/a520a0b5-deab-4825-9b36-329c399754f6-kube-api-access-g4cgf\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-jdcvx\" (UID: \"a520a0b5-deab-4825-9b36-329c399754f6\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-jdcvx" Oct 30 23:59:59.883537 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:59:59.883497 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a520a0b5-deab-4825-9b36-329c399754f6-istiod-ca-cert\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-jdcvx\" (UID: \"a520a0b5-deab-4825-9b36-329c399754f6\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-jdcvx" Oct 30 23:59:59.883686 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:59:59.883536 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a520a0b5-deab-4825-9b36-329c399754f6-workload-socket\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-jdcvx\" (UID: \"a520a0b5-deab-4825-9b36-329c399754f6\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-jdcvx" Oct 30 23:59:59.883853 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:59:59.883828 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a520a0b5-deab-4825-9b36-329c399754f6-workload-certs\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-jdcvx\" (UID: \"a520a0b5-deab-4825-9b36-329c399754f6\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-jdcvx" Oct 30 23:59:59.884016 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:59:59.883985 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a520a0b5-deab-4825-9b36-329c399754f6-istio-data\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-jdcvx\" (UID: \"a520a0b5-deab-4825-9b36-329c399754f6\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-jdcvx" Oct 30 23:59:59.884136 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:59:59.884097 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a520a0b5-deab-4825-9b36-329c399754f6-credential-socket\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-jdcvx\" (UID: \"a520a0b5-deab-4825-9b36-329c399754f6\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-jdcvx" Oct 30 23:59:59.884281 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:59:59.884260 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a520a0b5-deab-4825-9b36-329c399754f6-istiod-ca-cert\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-jdcvx\" (UID: \"a520a0b5-deab-4825-9b36-329c399754f6\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-jdcvx" Oct 30 23:59:59.885893 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:59:59.885875 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a520a0b5-deab-4825-9b36-329c399754f6-istio-envoy\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-jdcvx\" (UID: \"a520a0b5-deab-4825-9b36-329c399754f6\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-jdcvx" Oct 30 23:59:59.887413 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:59:59.887388 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a520a0b5-deab-4825-9b36-329c399754f6-istio-podinfo\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-jdcvx\" (UID: \"a520a0b5-deab-4825-9b36-329c399754f6\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-jdcvx" Oct 30 23:59:59.889212 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:59:59.889194 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-kzqr4" Oct 30 23:59:59.949650 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:59:59.949516 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a520a0b5-deab-4825-9b36-329c399754f6-istio-token\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-jdcvx\" (UID: \"a520a0b5-deab-4825-9b36-329c399754f6\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-jdcvx" Oct 30 23:59:59.951514 ip-10-0-141-180 kubenswrapper[4927]: I1030 23:59:59.951465 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4cgf\" (UniqueName: \"kubernetes.io/projected/a520a0b5-deab-4825-9b36-329c399754f6-kube-api-access-g4cgf\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-jdcvx\" (UID: \"a520a0b5-deab-4825-9b36-329c399754f6\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-jdcvx" Oct 31 00:00:00.229824 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:00.229787 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-fe01fa-29890080-j8xhq"] Oct 31 00:00:00.235746 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:00.235725 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-fe01fa-29890080-j8xhq" Oct 31 00:00:00.237246 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:00.237222 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-986d37-29890080-hp6pf"] Oct 31 00:00:00.240305 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:00.240281 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-jdcvx" Oct 31 00:00:00.242416 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:00.242397 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-986d37-29890080-hp6pf" Oct 31 00:00:00.243285 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:00.243263 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890080-cd2fx"] Oct 31 00:00:00.248899 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:00.248860 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890080-cd2fx" Oct 31 00:00:00.263025 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:00.263002 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-34f1ce-29890080-rjm5v"] Oct 31 00:00:00.269442 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:00.269419 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-34f1ce-29890080-rjm5v" Oct 31 00:00:00.270209 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:00.270188 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-46b586-29890080-w6n4c"] Oct 31 00:00:00.278631 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:00.278606 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-46b586-29890080-w6n4c" Oct 31 00:00:00.292772 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:00.292371 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4fdzs\"" Oct 31 00:00:00.297973 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:00.297950 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-pruner-29890080-nxlpd"] Oct 31 00:00:00.306995 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:00.306874 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-fe01fa-29890080-j8xhq"] Oct 31 00:00:00.307591 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:00.306962 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-pruner-29890080-nxlpd" Oct 31 00:00:00.320049 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:00.320018 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890080-cd2fx"] Oct 31 00:00:00.336075 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:00.336044 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-986d37-29890080-hp6pf"] Oct 31 00:00:00.343678 ip-10-0-141-180 kubenswrapper[4927]: W1031 00:00:00.343650 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda550117e_ad41_4e7c_998d_910a8d13d617.slice/crio-bee5c50d0ae3c67110da57bfdf45b6cd79ee2b556401181d0c987e2eae13d0b7 WatchSource:0}: Error finding container bee5c50d0ae3c67110da57bfdf45b6cd79ee2b556401181d0c987e2eae13d0b7: Status 404 returned error can't find the container with id bee5c50d0ae3c67110da57bfdf45b6cd79ee2b556401181d0c987e2eae13d0b7 Oct 31 00:00:00.350160 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:00.350127 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:00:00.351491 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:00.351462 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:00:00.351655 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:00.351635 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:00:00.361104 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:00.360699 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-kzqr4"] Oct 31 00:00:00.363540 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:00.363464 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"serviceca\"" Oct 31 00:00:00.368057 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:00.367987 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-pruner-29890080-nxlpd"] Oct 31 00:00:00.415419 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:00.401687 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"pruner-dockercfg-2wnqr\"" Oct 31 00:00:00.415419 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:00.405852 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-25nmv\" (UniqueName: \"kubernetes.io/projected/ffc39770-0916-4a31-8a21-2f7b91de7031-kube-api-access-25nmv\") pod \"maas-api-key-cleanup-29890080-cd2fx\" (UID: \"ffc39770-0916-4a31-8a21-2f7b91de7031\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890080-cd2fx" Oct 31 00:00:00.415419 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:00.406028 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zpmxb\" (UniqueName: \"kubernetes.io/projected/b22bb02d-b984-4b9b-9198-ec9584512919-kube-api-access-zpmxb\") pod \"maas-api-key-cleanup-e2e-worker-w4-34f1ce-29890080-rjm5v\" (UID: \"b22bb02d-b984-4b9b-9198-ec9584512919\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-34f1ce-29890080-rjm5v" Oct 31 00:00:00.415419 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:00.406190 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dmfmh\" (UniqueName: \"kubernetes.io/projected/7326c89a-a135-479e-8abb-3e28ede9179d-kube-api-access-dmfmh\") pod \"image-pruner-29890080-nxlpd\" (UID: \"7326c89a-a135-479e-8abb-3e28ede9179d\") " pod="openshift-image-registry/image-pruner-29890080-nxlpd" Oct 31 00:00:00.415419 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:00.406403 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b22bb02d-b984-4b9b-9198-ec9584512919-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-34f1ce-29890080-rjm5v\" (UID: \"b22bb02d-b984-4b9b-9198-ec9584512919\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-34f1ce-29890080-rjm5v" Oct 31 00:00:00.415419 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:00.406584 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7326c89a-a135-479e-8abb-3e28ede9179d-serviceca\") pod \"image-pruner-29890080-nxlpd\" (UID: \"7326c89a-a135-479e-8abb-3e28ede9179d\") " pod="openshift-image-registry/image-pruner-29890080-nxlpd" Oct 31 00:00:00.415419 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:00.406649 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gl5th\" (UniqueName: \"kubernetes.io/projected/5770165a-df6d-43b8-9997-b6fe1c76dc09-kube-api-access-gl5th\") pod \"maas-api-key-cleanup-e2e-worker-w5-fe01fa-29890080-j8xhq\" (UID: \"5770165a-df6d-43b8-9997-b6fe1c76dc09\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-fe01fa-29890080-j8xhq" Oct 31 00:00:00.415419 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:00.406848 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bgdvc\" (UniqueName: \"kubernetes.io/projected/512b87d9-7168-4bd4-a9a0-9e4d9c054d72-kube-api-access-bgdvc\") pod \"maas-api-key-cleanup-e2e-worker-w3-986d37-29890080-hp6pf\" (UID: \"512b87d9-7168-4bd4-a9a0-9e4d9c054d72\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-986d37-29890080-hp6pf" Oct 31 00:00:00.415419 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:00.407140 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ffc39770-0916-4a31-8a21-2f7b91de7031-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890080-cd2fx\" (UID: \"ffc39770-0916-4a31-8a21-2f7b91de7031\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890080-cd2fx" Oct 31 00:00:00.415419 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:00.407347 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fqfx7\" (UniqueName: \"kubernetes.io/projected/7a3a4a88-0a01-46c2-93be-4983b8fd5146-kube-api-access-fqfx7\") pod \"maas-api-key-cleanup-e2e-worker-w0-46b586-29890080-w6n4c\" (UID: \"7a3a4a88-0a01-46c2-93be-4983b8fd5146\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-46b586-29890080-w6n4c" Oct 31 00:00:00.415419 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:00.407627 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/512b87d9-7168-4bd4-a9a0-9e4d9c054d72-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-986d37-29890080-hp6pf\" (UID: \"512b87d9-7168-4bd4-a9a0-9e4d9c054d72\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-986d37-29890080-hp6pf" Oct 31 00:00:00.415419 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:00.407828 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5770165a-df6d-43b8-9997-b6fe1c76dc09-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-fe01fa-29890080-j8xhq\" (UID: \"5770165a-df6d-43b8-9997-b6fe1c76dc09\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-fe01fa-29890080-j8xhq" Oct 31 00:00:00.415419 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:00.408008 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a3a4a88-0a01-46c2-93be-4983b8fd5146-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-46b586-29890080-w6n4c\" (UID: \"7a3a4a88-0a01-46c2-93be-4983b8fd5146\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-46b586-29890080-w6n4c" Oct 31 00:00:00.415419 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:00.412039 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-34f1ce-29890080-rjm5v"] Oct 31 00:00:00.436546 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:00.436513 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-46b586-29890080-w6n4c"] Oct 31 00:00:00.511407 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:00.511353 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ffc39770-0916-4a31-8a21-2f7b91de7031-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890080-cd2fx\" (UID: \"ffc39770-0916-4a31-8a21-2f7b91de7031\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890080-cd2fx" Oct 31 00:00:00.511549 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:00.511421 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqfx7\" (UniqueName: \"kubernetes.io/projected/7a3a4a88-0a01-46c2-93be-4983b8fd5146-kube-api-access-fqfx7\") pod \"maas-api-key-cleanup-e2e-worker-w0-46b586-29890080-w6n4c\" (UID: \"7a3a4a88-0a01-46c2-93be-4983b8fd5146\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-46b586-29890080-w6n4c" Oct 31 00:00:00.511549 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:00.511516 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/512b87d9-7168-4bd4-a9a0-9e4d9c054d72-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-986d37-29890080-hp6pf\" (UID: \"512b87d9-7168-4bd4-a9a0-9e4d9c054d72\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-986d37-29890080-hp6pf" Oct 31 00:00:00.511839 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:00.511804 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5770165a-df6d-43b8-9997-b6fe1c76dc09-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-fe01fa-29890080-j8xhq\" (UID: \"5770165a-df6d-43b8-9997-b6fe1c76dc09\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-fe01fa-29890080-j8xhq" Oct 31 00:00:00.511974 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:00.511882 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a3a4a88-0a01-46c2-93be-4983b8fd5146-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-46b586-29890080-w6n4c\" (UID: \"7a3a4a88-0a01-46c2-93be-4983b8fd5146\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-46b586-29890080-w6n4c" Oct 31 00:00:00.512272 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:00.512229 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ffc39770-0916-4a31-8a21-2f7b91de7031-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890080-cd2fx\" (UID: \"ffc39770-0916-4a31-8a21-2f7b91de7031\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890080-cd2fx" Oct 31 00:00:00.512402 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:00.512314 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/512b87d9-7168-4bd4-a9a0-9e4d9c054d72-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-986d37-29890080-hp6pf\" (UID: \"512b87d9-7168-4bd4-a9a0-9e4d9c054d72\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-986d37-29890080-hp6pf" Oct 31 00:00:00.512629 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:00.512608 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5770165a-df6d-43b8-9997-b6fe1c76dc09-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-fe01fa-29890080-j8xhq\" (UID: \"5770165a-df6d-43b8-9997-b6fe1c76dc09\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-fe01fa-29890080-j8xhq" Oct 31 00:00:00.512799 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:00.512777 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25nmv\" (UniqueName: \"kubernetes.io/projected/ffc39770-0916-4a31-8a21-2f7b91de7031-kube-api-access-25nmv\") pod \"maas-api-key-cleanup-29890080-cd2fx\" (UID: \"ffc39770-0916-4a31-8a21-2f7b91de7031\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890080-cd2fx" Oct 31 00:00:00.512862 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:00.512816 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zpmxb\" (UniqueName: \"kubernetes.io/projected/b22bb02d-b984-4b9b-9198-ec9584512919-kube-api-access-zpmxb\") pod \"maas-api-key-cleanup-e2e-worker-w4-34f1ce-29890080-rjm5v\" (UID: \"b22bb02d-b984-4b9b-9198-ec9584512919\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-34f1ce-29890080-rjm5v" Oct 31 00:00:00.512920 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:00.512869 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmfmh\" (UniqueName: \"kubernetes.io/projected/7326c89a-a135-479e-8abb-3e28ede9179d-kube-api-access-dmfmh\") pod \"image-pruner-29890080-nxlpd\" (UID: \"7326c89a-a135-479e-8abb-3e28ede9179d\") " pod="openshift-image-registry/image-pruner-29890080-nxlpd" Oct 31 00:00:00.512920 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:00.512900 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b22bb02d-b984-4b9b-9198-ec9584512919-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-34f1ce-29890080-rjm5v\" (UID: \"b22bb02d-b984-4b9b-9198-ec9584512919\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-34f1ce-29890080-rjm5v" Oct 31 00:00:00.513081 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:00.513059 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a3a4a88-0a01-46c2-93be-4983b8fd5146-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-46b586-29890080-w6n4c\" (UID: \"7a3a4a88-0a01-46c2-93be-4983b8fd5146\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-46b586-29890080-w6n4c" Oct 31 00:00:00.515003 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:00.512931 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7326c89a-a135-479e-8abb-3e28ede9179d-serviceca\") pod \"image-pruner-29890080-nxlpd\" (UID: \"7326c89a-a135-479e-8abb-3e28ede9179d\") " pod="openshift-image-registry/image-pruner-29890080-nxlpd" Oct 31 00:00:00.515003 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:00.513865 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gl5th\" (UniqueName: \"kubernetes.io/projected/5770165a-df6d-43b8-9997-b6fe1c76dc09-kube-api-access-gl5th\") pod \"maas-api-key-cleanup-e2e-worker-w5-fe01fa-29890080-j8xhq\" (UID: \"5770165a-df6d-43b8-9997-b6fe1c76dc09\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-fe01fa-29890080-j8xhq" Oct 31 00:00:00.515003 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:00.513927 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgdvc\" (UniqueName: \"kubernetes.io/projected/512b87d9-7168-4bd4-a9a0-9e4d9c054d72-kube-api-access-bgdvc\") pod \"maas-api-key-cleanup-e2e-worker-w3-986d37-29890080-hp6pf\" (UID: \"512b87d9-7168-4bd4-a9a0-9e4d9c054d72\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-986d37-29890080-hp6pf" Oct 31 00:00:00.515003 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:00.514761 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7326c89a-a135-479e-8abb-3e28ede9179d-serviceca\") pod \"image-pruner-29890080-nxlpd\" (UID: \"7326c89a-a135-479e-8abb-3e28ede9179d\") " pod="openshift-image-registry/image-pruner-29890080-nxlpd" Oct 31 00:00:00.515003 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:00.514976 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b22bb02d-b984-4b9b-9198-ec9584512919-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-34f1ce-29890080-rjm5v\" (UID: \"b22bb02d-b984-4b9b-9198-ec9584512919\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-34f1ce-29890080-rjm5v" Oct 31 00:00:00.575971 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:00.575861 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqfx7\" (UniqueName: \"kubernetes.io/projected/7a3a4a88-0a01-46c2-93be-4983b8fd5146-kube-api-access-fqfx7\") pod \"maas-api-key-cleanup-e2e-worker-w0-46b586-29890080-w6n4c\" (UID: \"7a3a4a88-0a01-46c2-93be-4983b8fd5146\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-46b586-29890080-w6n4c" Oct 31 00:00:00.578178 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:00.578149 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gl5th\" (UniqueName: \"kubernetes.io/projected/5770165a-df6d-43b8-9997-b6fe1c76dc09-kube-api-access-gl5th\") pod \"maas-api-key-cleanup-e2e-worker-w5-fe01fa-29890080-j8xhq\" (UID: \"5770165a-df6d-43b8-9997-b6fe1c76dc09\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-fe01fa-29890080-j8xhq" Oct 31 00:00:00.578793 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:00.578739 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-25nmv\" (UniqueName: \"kubernetes.io/projected/ffc39770-0916-4a31-8a21-2f7b91de7031-kube-api-access-25nmv\") pod \"maas-api-key-cleanup-29890080-cd2fx\" (UID: \"ffc39770-0916-4a31-8a21-2f7b91de7031\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890080-cd2fx" Oct 31 00:00:00.579602 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:00.579154 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bgdvc\" (UniqueName: \"kubernetes.io/projected/512b87d9-7168-4bd4-a9a0-9e4d9c054d72-kube-api-access-bgdvc\") pod \"maas-api-key-cleanup-e2e-worker-w3-986d37-29890080-hp6pf\" (UID: \"512b87d9-7168-4bd4-a9a0-9e4d9c054d72\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-986d37-29890080-hp6pf" Oct 31 00:00:00.580157 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:00.580134 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dmfmh\" (UniqueName: \"kubernetes.io/projected/7326c89a-a135-479e-8abb-3e28ede9179d-kube-api-access-dmfmh\") pod \"image-pruner-29890080-nxlpd\" (UID: \"7326c89a-a135-479e-8abb-3e28ede9179d\") " pod="openshift-image-registry/image-pruner-29890080-nxlpd" Oct 31 00:00:00.580727 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:00.580704 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zpmxb\" (UniqueName: \"kubernetes.io/projected/b22bb02d-b984-4b9b-9198-ec9584512919-kube-api-access-zpmxb\") pod \"maas-api-key-cleanup-e2e-worker-w4-34f1ce-29890080-rjm5v\" (UID: \"b22bb02d-b984-4b9b-9198-ec9584512919\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-34f1ce-29890080-rjm5v" Oct 31 00:00:00.583432 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:00.583409 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-fe01fa-29890080-j8xhq" Oct 31 00:00:00.600276 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:00.600085 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-986d37-29890080-hp6pf" Oct 31 00:00:00.618033 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:00.618002 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890080-cd2fx" Oct 31 00:00:00.648901 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:00.648662 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-34f1ce-29890080-rjm5v" Oct 31 00:00:00.696370 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:00.695995 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-46b586-29890080-w6n4c" Oct 31 00:00:00.696370 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:00.696273 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-jdcvx"] Oct 31 00:00:00.739798 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:00.737810 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-pruner-29890080-nxlpd" Oct 31 00:00:00.741483 ip-10-0-141-180 kubenswrapper[4927]: W1031 00:00:00.740954 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda520a0b5_deab_4825_9b36_329c399754f6.slice/crio-fd9f345acb4b480c88d6746c033f65a00a195c33416f0dbcc07a463f8c7de337 WatchSource:0}: Error finding container fd9f345acb4b480c88d6746c033f65a00a195c33416f0dbcc07a463f8c7de337: Status 404 returned error can't find the container with id fd9f345acb4b480c88d6746c033f65a00a195c33416f0dbcc07a463f8c7de337 Oct 31 00:00:00.764199 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:00.762323 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:00:00.764199 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:00.763450 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:00:00.764199 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:00.763545 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:00:00.916793 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:00.916625 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:00:01.130211 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:01.118245 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-fe01fa-29890080-j8xhq"] Oct 31 00:00:01.130211 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:01.120940 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-jdcvx" event={"ID":"a520a0b5-deab-4825-9b36-329c399754f6","Type":"ContainerStarted","Data":"fd9f345acb4b480c88d6746c033f65a00a195c33416f0dbcc07a463f8c7de337"} Oct 31 00:00:01.140684 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:01.131317 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-kzqr4" event={"ID":"a550117e-ad41-4e7c-998d-910a8d13d617","Type":"ContainerStarted","Data":"8351cbcebc39fc6a3f1bd858ac5bebc698fad85fb20361588f689aae67eaefa9"} Oct 31 00:00:01.140684 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:01.140627 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-kzqr4" event={"ID":"a550117e-ad41-4e7c-998d-910a8d13d617","Type":"ContainerStarted","Data":"bee5c50d0ae3c67110da57bfdf45b6cd79ee2b556401181d0c987e2eae13d0b7"} Oct 31 00:00:01.258710 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:01.250870 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-986d37-29890080-hp6pf"] Oct 31 00:00:01.274565 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:01.274478 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-kzqr4" podStartSLOduration=2.2744557309999998 podStartE2EDuration="2.274455731s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:59:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:00:01.270024336 +0000 UTC m=+1114.306355165" watchObservedRunningTime="2026-10-31 00:00:01.274455731 +0000 UTC m=+1114.310786548" Oct 31 00:00:01.291419 ip-10-0-141-180 kubenswrapper[4927]: W1031 00:00:01.290462 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod512b87d9_7168_4bd4_a9a0_9e4d9c054d72.slice/crio-f492a23b399ee0d6c2f3206b940d757835650ba0da77bf0f76aa221057803593 WatchSource:0}: Error finding container f492a23b399ee0d6c2f3206b940d757835650ba0da77bf0f76aa221057803593: Status 404 returned error can't find the container with id f492a23b399ee0d6c2f3206b940d757835650ba0da77bf0f76aa221057803593 Oct 31 00:00:01.379205 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:01.379162 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-34f1ce-29890080-rjm5v"] Oct 31 00:00:01.379422 ip-10-0-141-180 kubenswrapper[4927]: W1031 00:00:01.379372 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb22bb02d_b984_4b9b_9198_ec9584512919.slice/crio-eab5c7e02d2e54de1f0ee6ed6c2284aa2f03ade56672ded68384276a8a467058 WatchSource:0}: Error finding container eab5c7e02d2e54de1f0ee6ed6c2284aa2f03ade56672ded68384276a8a467058: Status 404 returned error can't find the container with id eab5c7e02d2e54de1f0ee6ed6c2284aa2f03ade56672ded68384276a8a467058 Oct 31 00:00:01.472848 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:01.472816 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890080-cd2fx"] Oct 31 00:00:01.513134 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:01.513085 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-46b586-29890080-w6n4c"] Oct 31 00:00:01.552881 ip-10-0-141-180 kubenswrapper[4927]: W1031 00:00:01.552792 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a3a4a88_0a01_46c2_93be_4983b8fd5146.slice/crio-466f2dcba77a024730b6e6f9db5bb9ed9167a75448d149e0c94ddebffb35db90 WatchSource:0}: Error finding container 466f2dcba77a024730b6e6f9db5bb9ed9167a75448d149e0c94ddebffb35db90: Status 404 returned error can't find the container with id 466f2dcba77a024730b6e6f9db5bb9ed9167a75448d149e0c94ddebffb35db90 Oct 31 00:00:01.588728 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:01.588662 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-pruner-29890080-nxlpd"] Oct 31 00:00:01.593116 ip-10-0-141-180 kubenswrapper[4927]: W1031 00:00:01.593083 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7326c89a_a135_479e_8abb_3e28ede9179d.slice/crio-2163401518f77c3f695fea9af9bc6aaaa0fea5e3994975ee2df416bb2c030b95 WatchSource:0}: Error finding container 2163401518f77c3f695fea9af9bc6aaaa0fea5e3994975ee2df416bb2c030b95: Status 404 returned error can't find the container with id 2163401518f77c3f695fea9af9bc6aaaa0fea5e3994975ee2df416bb2c030b95 Oct 31 00:00:01.891375 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:01.890127 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-kzqr4" Oct 31 00:00:01.892514 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:01.892482 4927 patch_prober.go:29] interesting pod/e2e-worker-w0-46b586-openshift-default-86c598c96c-kzqr4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Oct 31 00:00:01.892614 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:01.892540 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-kzqr4" podUID="a550117e-ad41-4e7c-998d-910a8d13d617" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Oct 31 00:00:02.186678 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:02.186638 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-jdcvx" event={"ID":"a520a0b5-deab-4825-9b36-329c399754f6","Type":"ContainerStarted","Data":"7176a65eae8ecdf19042d0452aac23fa54b6cb2c488a78b8fd12c4e09bc31744"} Oct 31 00:00:02.196140 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:02.195877 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-46b586-29890080-w6n4c" event={"ID":"7a3a4a88-0a01-46c2-93be-4983b8fd5146","Type":"ContainerStarted","Data":"466f2dcba77a024730b6e6f9db5bb9ed9167a75448d149e0c94ddebffb35db90"} Oct 31 00:00:02.199850 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:02.198958 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-pruner-29890080-nxlpd" event={"ID":"7326c89a-a135-479e-8abb-3e28ede9179d","Type":"ContainerStarted","Data":"e762332b206d017930113501c1f1cb8fa923fd6e6028cc3a24a58354954fa4bd"} Oct 31 00:00:02.200007 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:02.199884 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-pruner-29890080-nxlpd" event={"ID":"7326c89a-a135-479e-8abb-3e28ede9179d","Type":"ContainerStarted","Data":"2163401518f77c3f695fea9af9bc6aaaa0fea5e3994975ee2df416bb2c030b95"} Oct 31 00:00:02.202071 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:02.202043 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-34f1ce-29890080-rjm5v" event={"ID":"b22bb02d-b984-4b9b-9198-ec9584512919","Type":"ContainerStarted","Data":"eab5c7e02d2e54de1f0ee6ed6c2284aa2f03ade56672ded68384276a8a467058"} Oct 31 00:00:02.204233 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:02.204202 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-fe01fa-29890080-j8xhq" event={"ID":"5770165a-df6d-43b8-9997-b6fe1c76dc09","Type":"ContainerStarted","Data":"55d8ed9cfb30a5164f4cebc7a6b9cf389b0c916407dedc9fd8b713a03f7c4db5"} Oct 31 00:00:02.204367 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:02.204245 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-fe01fa-29890080-j8xhq" event={"ID":"5770165a-df6d-43b8-9997-b6fe1c76dc09","Type":"ContainerStarted","Data":"ce544920adfa12c5d299e943d3e6d56ca8f2fee67370ee8ecc5cd11ccd2beb74"} Oct 31 00:00:02.206619 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:02.206376 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-986d37-29890080-hp6pf" event={"ID":"512b87d9-7168-4bd4-a9a0-9e4d9c054d72","Type":"ContainerStarted","Data":"54409a5d31e35773e479c7afc78ec3b0954d7133c9c421976b9daf80c4c02838"} Oct 31 00:00:02.206619 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:02.206436 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-986d37-29890080-hp6pf" event={"ID":"512b87d9-7168-4bd4-a9a0-9e4d9c054d72","Type":"ContainerStarted","Data":"f492a23b399ee0d6c2f3206b940d757835650ba0da77bf0f76aa221057803593"} Oct 31 00:00:02.233378 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:02.230195 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890080-cd2fx" event={"ID":"ffc39770-0916-4a31-8a21-2f7b91de7031","Type":"ContainerStarted","Data":"8bfac1177dcff191ab802d779b5ec4024eccf4bdd77ebbe0a86af092539b4a2c"} Oct 31 00:00:02.267246 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:02.265752 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-jdcvx" Oct 31 00:00:02.269413 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:02.269257 4927 patch_prober.go:29] interesting pod/e2e-worker-w0-46b586-openshift-default-86c598c96c-jdcvx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body= Oct 31 00:00:02.269413 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:02.269312 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-jdcvx" podUID="a520a0b5-deab-4825-9b36-329c399754f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" Oct 31 00:00:02.379787 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:02.379727 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-jdcvx" podStartSLOduration=3.379709952 podStartE2EDuration="3.379709952s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:59:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:00:02.33888972 +0000 UTC m=+1115.375220543" watchObservedRunningTime="2026-10-31 00:00:02.379709952 +0000 UTC m=+1115.416040775" Oct 31 00:00:02.484692 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:02.484558 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/image-pruner-29890080-nxlpd" podStartSLOduration=2.484542792 podStartE2EDuration="2.484542792s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:00:02.467264686 +0000 UTC m=+1115.503595508" watchObservedRunningTime="2026-10-31 00:00:02.484542792 +0000 UTC m=+1115.520873646" Oct 31 00:00:02.596491 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:02.596390 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890080-cd2fx" podStartSLOduration=2.5963678569999997 podStartE2EDuration="2.596367857s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:00:02.590584154 +0000 UTC m=+1115.626914978" watchObservedRunningTime="2026-10-31 00:00:02.596367857 +0000 UTC m=+1115.632698680" Oct 31 00:00:02.771118 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:02.771008 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-986d37-29890080-hp6pf" podStartSLOduration=2.770986287 podStartE2EDuration="2.770986287s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:00:02.716894173 +0000 UTC m=+1115.753225020" watchObservedRunningTime="2026-10-31 00:00:02.770986287 +0000 UTC m=+1115.807317131" Oct 31 00:00:02.878382 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:02.878314 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-fe01fa-29890080-j8xhq" podStartSLOduration=2.878300475 podStartE2EDuration="2.878300475s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:00:02.873369089 +0000 UTC m=+1115.909699912" watchObservedRunningTime="2026-10-31 00:00:02.878300475 +0000 UTC m=+1115.914631290" Oct 31 00:00:02.890902 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:02.890480 4927 patch_prober.go:29] interesting pod/e2e-worker-w0-46b586-openshift-default-86c598c96c-kzqr4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Oct 31 00:00:02.890902 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:02.890531 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-kzqr4" podUID="a550117e-ad41-4e7c-998d-910a8d13d617" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Oct 31 00:00:02.998127 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:02.998058 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-34f1ce-29890080-rjm5v" podStartSLOduration=2.998039469 podStartE2EDuration="2.998039469s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:00:02.994498178 +0000 UTC m=+1116.030829001" watchObservedRunningTime="2026-10-31 00:00:02.998039469 +0000 UTC m=+1116.034370293" Oct 31 00:00:03.250951 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:03.250467 4927 patch_prober.go:29] interesting pod/e2e-worker-w0-46b586-openshift-default-86c598c96c-jdcvx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body= Oct 31 00:00:03.250951 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:03.250547 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-jdcvx" podUID="a520a0b5-deab-4825-9b36-329c399754f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" Oct 31 00:00:03.275354 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:03.273884 4927 generic.go:356] "Generic (PLEG): container finished" podID="b22bb02d-b984-4b9b-9198-ec9584512919" containerID="b39fbb2e0813291aac7711069b5ce7bdc0c5e36236f32ed7dd20719de53ec398" exitCode=60 Oct 31 00:00:03.275354 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:03.274080 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-34f1ce-29890080-rjm5v" event={"ID":"b22bb02d-b984-4b9b-9198-ec9584512919","Type":"ContainerDied","Data":"b39fbb2e0813291aac7711069b5ce7bdc0c5e36236f32ed7dd20719de53ec398"} Oct 31 00:00:03.275354 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:03.274571 4927 scope.go:122] "RemoveContainer" containerID="b39fbb2e0813291aac7711069b5ce7bdc0c5e36236f32ed7dd20719de53ec398" Oct 31 00:00:03.280806 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:03.279528 4927 generic.go:356] "Generic (PLEG): container finished" podID="5770165a-df6d-43b8-9997-b6fe1c76dc09" containerID="55d8ed9cfb30a5164f4cebc7a6b9cf389b0c916407dedc9fd8b713a03f7c4db5" exitCode=60 Oct 31 00:00:03.280806 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:03.279605 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-fe01fa-29890080-j8xhq" event={"ID":"5770165a-df6d-43b8-9997-b6fe1c76dc09","Type":"ContainerDied","Data":"55d8ed9cfb30a5164f4cebc7a6b9cf389b0c916407dedc9fd8b713a03f7c4db5"} Oct 31 00:00:03.280806 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:03.279840 4927 scope.go:122] "RemoveContainer" containerID="55d8ed9cfb30a5164f4cebc7a6b9cf389b0c916407dedc9fd8b713a03f7c4db5" Oct 31 00:00:03.281816 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:03.281759 4927 generic.go:356] "Generic (PLEG): container finished" podID="512b87d9-7168-4bd4-a9a0-9e4d9c054d72" containerID="54409a5d31e35773e479c7afc78ec3b0954d7133c9c421976b9daf80c4c02838" exitCode=60 Oct 31 00:00:03.281915 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:03.281904 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-986d37-29890080-hp6pf" event={"ID":"512b87d9-7168-4bd4-a9a0-9e4d9c054d72","Type":"ContainerDied","Data":"54409a5d31e35773e479c7afc78ec3b0954d7133c9c421976b9daf80c4c02838"} Oct 31 00:00:03.282262 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:03.282249 4927 scope.go:122] "RemoveContainer" containerID="54409a5d31e35773e479c7afc78ec3b0954d7133c9c421976b9daf80c4c02838" Oct 31 00:00:03.284571 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:03.284548 4927 generic.go:356] "Generic (PLEG): container finished" podID="ffc39770-0916-4a31-8a21-2f7b91de7031" containerID="a200445da081318f88393b8ac64b444a3984999638fb3239606bea68ff7fc7af" exitCode=60 Oct 31 00:00:03.284671 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:03.284638 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890080-cd2fx" event={"ID":"ffc39770-0916-4a31-8a21-2f7b91de7031","Type":"ContainerDied","Data":"a200445da081318f88393b8ac64b444a3984999638fb3239606bea68ff7fc7af"} Oct 31 00:00:03.284940 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:03.284922 4927 scope.go:122] "RemoveContainer" containerID="a200445da081318f88393b8ac64b444a3984999638fb3239606bea68ff7fc7af" Oct 31 00:00:03.287110 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:03.287091 4927 generic.go:356] "Generic (PLEG): container finished" podID="7a3a4a88-0a01-46c2-93be-4983b8fd5146" containerID="c10a0b86f44b19093ab903c2b6bac42c808d57f65383af3497b78988103708cd" exitCode=60 Oct 31 00:00:03.287211 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:03.287172 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-46b586-29890080-w6n4c" event={"ID":"7a3a4a88-0a01-46c2-93be-4983b8fd5146","Type":"ContainerDied","Data":"c10a0b86f44b19093ab903c2b6bac42c808d57f65383af3497b78988103708cd"} Oct 31 00:00:03.287617 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:03.287600 4927 scope.go:122] "RemoveContainer" containerID="c10a0b86f44b19093ab903c2b6bac42c808d57f65383af3497b78988103708cd" Oct 31 00:00:03.647357 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:03.640600 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:00:03.896787 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:03.896748 4927 patch_prober.go:29] interesting pod/e2e-worker-w0-46b586-openshift-default-86c598c96c-kzqr4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Oct 31 00:00:03.898912 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:03.898870 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-kzqr4" podUID="a550117e-ad41-4e7c-998d-910a8d13d617" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Oct 31 00:00:04.244428 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:04.243147 4927 patch_prober.go:29] interesting pod/e2e-worker-w0-46b586-openshift-default-86c598c96c-jdcvx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body= Oct 31 00:00:04.244428 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:04.243303 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-jdcvx" podUID="a520a0b5-deab-4825-9b36-329c399754f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" Oct 31 00:00:04.297938 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:04.297904 4927 generic.go:356] "Generic (PLEG): container finished" podID="7326c89a-a135-479e-8abb-3e28ede9179d" containerID="e762332b206d017930113501c1f1cb8fa923fd6e6028cc3a24a58354954fa4bd" exitCode=0 Oct 31 00:00:04.298085 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:04.297976 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-pruner-29890080-nxlpd" event={"ID":"7326c89a-a135-479e-8abb-3e28ede9179d","Type":"ContainerDied","Data":"e762332b206d017930113501c1f1cb8fa923fd6e6028cc3a24a58354954fa4bd"} Oct 31 00:00:04.299780 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:04.299729 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-34f1ce-29890080-rjm5v" event={"ID":"b22bb02d-b984-4b9b-9198-ec9584512919","Type":"ContainerStarted","Data":"f9031fefe28a5a69984b888ddf9fec1563728d25f7360a410a779d78982e2e84"} Oct 31 00:00:04.301601 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:04.301579 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-fe01fa-29890080-j8xhq" event={"ID":"5770165a-df6d-43b8-9997-b6fe1c76dc09","Type":"ContainerStarted","Data":"7e91290ddc2de5b8b9c3d339b37db548f6a15fa4c455c24f4edef82c6d27dc3e"} Oct 31 00:00:04.303949 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:04.303926 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-986d37-29890080-hp6pf" event={"ID":"512b87d9-7168-4bd4-a9a0-9e4d9c054d72","Type":"ContainerStarted","Data":"dbba66efe8a5e62ffbafcc6bd71292520b6330c4c4387cdade6ea109f7b98a1d"} Oct 31 00:00:04.305734 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:04.305711 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890080-cd2fx" event={"ID":"ffc39770-0916-4a31-8a21-2f7b91de7031","Type":"ContainerStarted","Data":"33e0ef4f041082c7ca9b918800c59a6abdee9c424543f3babf19c496b0596d4b"} Oct 31 00:00:04.317365 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:04.313052 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-46b586-29890080-w6n4c" event={"ID":"7a3a4a88-0a01-46c2-93be-4983b8fd5146","Type":"ContainerStarted","Data":"106fb15f9c3d5eed43102e4abfda6045220ff30cbdcfa356f70b53cccc6aca46"} Oct 31 00:00:04.890014 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:04.889934 4927 patch_prober.go:29] interesting pod/e2e-worker-w0-46b586-openshift-default-86c598c96c-kzqr4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Oct 31 00:00:04.890480 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:04.890446 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-kzqr4" podUID="a550117e-ad41-4e7c-998d-910a8d13d617" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Oct 31 00:00:04.933951 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:04.933881 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-46b586-29890080-w6n4c" podStartSLOduration=4.9338626009999995 podStartE2EDuration="4.933862601s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:00:04.93174131 +0000 UTC m=+1117.968072134" watchObservedRunningTime="2026-10-31 00:00:04.933862601 +0000 UTC m=+1117.970193426" Oct 31 00:00:05.245357 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:05.244754 4927 patch_prober.go:29] interesting pod/e2e-worker-w0-46b586-openshift-default-86c598c96c-jdcvx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body= Oct 31 00:00:05.245357 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:05.244818 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-jdcvx" podUID="a520a0b5-deab-4825-9b36-329c399754f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" Oct 31 00:00:05.367388 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:05.353467 4927 generic.go:356] "Generic (PLEG): container finished" podID="b22bb02d-b984-4b9b-9198-ec9584512919" containerID="f9031fefe28a5a69984b888ddf9fec1563728d25f7360a410a779d78982e2e84" exitCode=60 Oct 31 00:00:05.367930 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:05.355538 4927 scope.go:122] "RemoveContainer" containerID="f9031fefe28a5a69984b888ddf9fec1563728d25f7360a410a779d78982e2e84" Oct 31 00:00:05.368494 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:00:05.368465 4927 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-34f1ce-29890080-rjm5v_odh-ai-gateway-infra(b22bb02d-b984-4b9b-9198-ec9584512919)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-34f1ce-29890080-rjm5v" podUID="b22bb02d-b984-4b9b-9198-ec9584512919" Oct 31 00:00:05.368660 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:05.356509 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-34f1ce-29890080-rjm5v" event={"ID":"b22bb02d-b984-4b9b-9198-ec9584512919","Type":"ContainerDied","Data":"f9031fefe28a5a69984b888ddf9fec1563728d25f7360a410a779d78982e2e84"} Oct 31 00:00:05.368794 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:05.368783 4927 scope.go:122] "RemoveContainer" containerID="b39fbb2e0813291aac7711069b5ce7bdc0c5e36236f32ed7dd20719de53ec398" Oct 31 00:00:05.413796 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:05.412545 4927 generic.go:356] "Generic (PLEG): container finished" podID="5770165a-df6d-43b8-9997-b6fe1c76dc09" containerID="7e91290ddc2de5b8b9c3d339b37db548f6a15fa4c455c24f4edef82c6d27dc3e" exitCode=60 Oct 31 00:00:05.413796 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:05.412679 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-fe01fa-29890080-j8xhq" event={"ID":"5770165a-df6d-43b8-9997-b6fe1c76dc09","Type":"ContainerDied","Data":"7e91290ddc2de5b8b9c3d339b37db548f6a15fa4c455c24f4edef82c6d27dc3e"} Oct 31 00:00:05.413796 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:05.413070 4927 scope.go:122] "RemoveContainer" containerID="7e91290ddc2de5b8b9c3d339b37db548f6a15fa4c455c24f4edef82c6d27dc3e" Oct 31 00:00:05.413796 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:00:05.413397 4927 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-fe01fa-29890080-j8xhq_odh-ai-gateway-infra(5770165a-df6d-43b8-9997-b6fe1c76dc09)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-fe01fa-29890080-j8xhq" podUID="5770165a-df6d-43b8-9997-b6fe1c76dc09" Oct 31 00:00:05.455367 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:05.449374 4927 generic.go:356] "Generic (PLEG): container finished" podID="512b87d9-7168-4bd4-a9a0-9e4d9c054d72" containerID="dbba66efe8a5e62ffbafcc6bd71292520b6330c4c4387cdade6ea109f7b98a1d" exitCode=60 Oct 31 00:00:05.455367 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:05.449572 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-986d37-29890080-hp6pf" event={"ID":"512b87d9-7168-4bd4-a9a0-9e4d9c054d72","Type":"ContainerDied","Data":"dbba66efe8a5e62ffbafcc6bd71292520b6330c4c4387cdade6ea109f7b98a1d"} Oct 31 00:00:05.455983 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:05.455954 4927 scope.go:122] "RemoveContainer" containerID="dbba66efe8a5e62ffbafcc6bd71292520b6330c4c4387cdade6ea109f7b98a1d" Oct 31 00:00:05.456655 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:00:05.456586 4927 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-986d37-29890080-hp6pf_odh-ai-gateway-infra(512b87d9-7168-4bd4-a9a0-9e4d9c054d72)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-986d37-29890080-hp6pf" podUID="512b87d9-7168-4bd4-a9a0-9e4d9c054d72" Oct 31 00:00:05.489406 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:05.488936 4927 scope.go:122] "RemoveContainer" containerID="55d8ed9cfb30a5164f4cebc7a6b9cf389b0c916407dedc9fd8b713a03f7c4db5" Oct 31 00:00:05.527473 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:05.525730 4927 generic.go:356] "Generic (PLEG): container finished" podID="ffc39770-0916-4a31-8a21-2f7b91de7031" containerID="33e0ef4f041082c7ca9b918800c59a6abdee9c424543f3babf19c496b0596d4b" exitCode=60 Oct 31 00:00:05.527473 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:05.525877 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890080-cd2fx" event={"ID":"ffc39770-0916-4a31-8a21-2f7b91de7031","Type":"ContainerDied","Data":"33e0ef4f041082c7ca9b918800c59a6abdee9c424543f3babf19c496b0596d4b"} Oct 31 00:00:05.527473 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:05.526514 4927 scope.go:122] "RemoveContainer" containerID="33e0ef4f041082c7ca9b918800c59a6abdee9c424543f3babf19c496b0596d4b" Oct 31 00:00:05.527473 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:00:05.526896 4927 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29890080-cd2fx_odh-ai-gateway-infra(ffc39770-0916-4a31-8a21-2f7b91de7031)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890080-cd2fx" podUID="ffc39770-0916-4a31-8a21-2f7b91de7031" Oct 31 00:00:05.576940 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:05.576888 4927 generic.go:356] "Generic (PLEG): container finished" podID="7a3a4a88-0a01-46c2-93be-4983b8fd5146" containerID="106fb15f9c3d5eed43102e4abfda6045220ff30cbdcfa356f70b53cccc6aca46" exitCode=60 Oct 31 00:00:05.577556 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:05.577527 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-46b586-29890080-w6n4c" event={"ID":"7a3a4a88-0a01-46c2-93be-4983b8fd5146","Type":"ContainerDied","Data":"106fb15f9c3d5eed43102e4abfda6045220ff30cbdcfa356f70b53cccc6aca46"} Oct 31 00:00:05.578426 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:05.578398 4927 scope.go:122] "RemoveContainer" containerID="106fb15f9c3d5eed43102e4abfda6045220ff30cbdcfa356f70b53cccc6aca46" Oct 31 00:00:05.579057 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:00:05.579034 4927 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-46b586-29890080-w6n4c_odh-ai-gateway-infra(7a3a4a88-0a01-46c2-93be-4983b8fd5146)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-46b586-29890080-w6n4c" podUID="7a3a4a88-0a01-46c2-93be-4983b8fd5146" Oct 31 00:00:05.591564 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:05.591537 4927 scope.go:122] "RemoveContainer" containerID="54409a5d31e35773e479c7afc78ec3b0954d7133c9c421976b9daf80c4c02838" Oct 31 00:00:05.651542 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:05.641413 4927 scope.go:122] "RemoveContainer" containerID="a200445da081318f88393b8ac64b444a3984999638fb3239606bea68ff7fc7af" Oct 31 00:00:05.695917 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:05.695889 4927 scope.go:122] "RemoveContainer" containerID="c10a0b86f44b19093ab903c2b6bac42c808d57f65383af3497b78988103708cd" Oct 31 00:00:05.735364 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:05.732857 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:00:05.925388 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:05.919732 4927 patch_prober.go:29] interesting pod/e2e-worker-w0-46b586-openshift-default-86c598c96c-kzqr4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Oct 31 00:00:05.925388 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:05.919798 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-kzqr4" podUID="a550117e-ad41-4e7c-998d-910a8d13d617" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Oct 31 00:00:06.193301 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:06.193267 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-pruner-29890080-nxlpd" Oct 31 00:00:06.245157 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:06.244703 4927 patch_prober.go:29] interesting pod/e2e-worker-w0-46b586-openshift-default-86c598c96c-jdcvx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body= Oct 31 00:00:06.245157 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:06.244767 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-jdcvx" podUID="a520a0b5-deab-4825-9b36-329c399754f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" Oct 31 00:00:06.383417 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:06.382819 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7326c89a-a135-479e-8abb-3e28ede9179d-kube-api-access-dmfmh\" (UniqueName: \"kubernetes.io/projected/7326c89a-a135-479e-8abb-3e28ede9179d-kube-api-access-dmfmh\") pod \"7326c89a-a135-479e-8abb-3e28ede9179d\" (UID: \"7326c89a-a135-479e-8abb-3e28ede9179d\") " Oct 31 00:00:06.383417 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:06.383027 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7326c89a-a135-479e-8abb-3e28ede9179d-serviceca\" (UniqueName: \"kubernetes.io/configmap/7326c89a-a135-479e-8abb-3e28ede9179d-serviceca\") pod \"7326c89a-a135-479e-8abb-3e28ede9179d\" (UID: \"7326c89a-a135-479e-8abb-3e28ede9179d\") " Oct 31 00:00:06.384196 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:06.384166 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7326c89a-a135-479e-8abb-3e28ede9179d-serviceca" pod "7326c89a-a135-479e-8abb-3e28ede9179d" (UID: "7326c89a-a135-479e-8abb-3e28ede9179d"). InnerVolumeSpecName "serviceca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:00:06.387564 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:06.387529 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7326c89a-a135-479e-8abb-3e28ede9179d-kube-api-access-dmfmh" pod "7326c89a-a135-479e-8abb-3e28ede9179d" (UID: "7326c89a-a135-479e-8abb-3e28ede9179d"). InnerVolumeSpecName "kube-api-access-dmfmh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:00:06.485489 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:06.485454 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dmfmh\" (UniqueName: \"kubernetes.io/projected/7326c89a-a135-479e-8abb-3e28ede9179d-kube-api-access-dmfmh\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:00:06.485748 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:06.485729 4927 reconciler_common.go:299] "Volume detached for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7326c89a-a135-479e-8abb-3e28ede9179d-serviceca\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:00:06.587824 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:06.587796 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-pruner-29890080-nxlpd" Oct 31 00:00:06.588400 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:06.588197 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-pruner-29890080-nxlpd" event={"ID":"7326c89a-a135-479e-8abb-3e28ede9179d","Type":"ContainerDied","Data":"2163401518f77c3f695fea9af9bc6aaaa0fea5e3994975ee2df416bb2c030b95"} Oct 31 00:00:06.588400 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:06.588247 4927 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2163401518f77c3f695fea9af9bc6aaaa0fea5e3994975ee2df416bb2c030b95" Oct 31 00:00:06.590628 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:06.590144 4927 scope.go:122] "RemoveContainer" containerID="f9031fefe28a5a69984b888ddf9fec1563728d25f7360a410a779d78982e2e84" Oct 31 00:00:06.590628 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:00:06.590371 4927 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-34f1ce-29890080-rjm5v_odh-ai-gateway-infra(b22bb02d-b984-4b9b-9198-ec9584512919)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-34f1ce-29890080-rjm5v" podUID="b22bb02d-b984-4b9b-9198-ec9584512919" Oct 31 00:00:06.591896 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:06.591877 4927 scope.go:122] "RemoveContainer" containerID="7e91290ddc2de5b8b9c3d339b37db548f6a15fa4c455c24f4edef82c6d27dc3e" Oct 31 00:00:06.592256 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:00:06.592120 4927 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-fe01fa-29890080-j8xhq_odh-ai-gateway-infra(5770165a-df6d-43b8-9997-b6fe1c76dc09)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-fe01fa-29890080-j8xhq" podUID="5770165a-df6d-43b8-9997-b6fe1c76dc09" Oct 31 00:00:06.593451 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:06.593392 4927 scope.go:122] "RemoveContainer" containerID="dbba66efe8a5e62ffbafcc6bd71292520b6330c4c4387cdade6ea109f7b98a1d" Oct 31 00:00:06.593821 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:00:06.593801 4927 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-986d37-29890080-hp6pf_odh-ai-gateway-infra(512b87d9-7168-4bd4-a9a0-9e4d9c054d72)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-986d37-29890080-hp6pf" podUID="512b87d9-7168-4bd4-a9a0-9e4d9c054d72" Oct 31 00:00:06.594880 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:06.594864 4927 scope.go:122] "RemoveContainer" containerID="33e0ef4f041082c7ca9b918800c59a6abdee9c424543f3babf19c496b0596d4b" Oct 31 00:00:06.595069 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:00:06.595032 4927 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29890080-cd2fx_odh-ai-gateway-infra(ffc39770-0916-4a31-8a21-2f7b91de7031)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890080-cd2fx" podUID="ffc39770-0916-4a31-8a21-2f7b91de7031" Oct 31 00:00:06.596618 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:06.596602 4927 scope.go:122] "RemoveContainer" containerID="106fb15f9c3d5eed43102e4abfda6045220ff30cbdcfa356f70b53cccc6aca46" Oct 31 00:00:06.596903 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:00:06.596881 4927 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-46b586-29890080-w6n4c_odh-ai-gateway-infra(7a3a4a88-0a01-46c2-93be-4983b8fd5146)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-46b586-29890080-w6n4c" podUID="7a3a4a88-0a01-46c2-93be-4983b8fd5146" Oct 31 00:00:06.890137 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:06.890056 4927 patch_prober.go:29] interesting pod/e2e-worker-w0-46b586-openshift-default-86c598c96c-kzqr4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Oct 31 00:00:06.890378 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:06.890324 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-kzqr4" podUID="a550117e-ad41-4e7c-998d-910a8d13d617" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Oct 31 00:00:07.242355 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:07.241715 4927 patch_prober.go:29] interesting pod/e2e-worker-w0-46b586-openshift-default-86c598c96c-jdcvx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body= Oct 31 00:00:07.242355 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:07.241779 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-jdcvx" podUID="a520a0b5-deab-4825-9b36-329c399754f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" Oct 31 00:00:07.890600 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:07.890558 4927 patch_prober.go:29] interesting pod/e2e-worker-w0-46b586-openshift-default-86c598c96c-kzqr4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Oct 31 00:00:07.890787 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:07.890614 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-kzqr4" podUID="a550117e-ad41-4e7c-998d-910a8d13d617" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Oct 31 00:00:08.241794 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:08.241743 4927 patch_prober.go:29] interesting pod/e2e-worker-w0-46b586-openshift-default-86c598c96c-jdcvx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body= Oct 31 00:00:08.242123 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:08.242077 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-jdcvx" podUID="a520a0b5-deab-4825-9b36-329c399754f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" Oct 31 00:00:08.895944 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:08.895911 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-kzqr4" Oct 31 00:00:08.896466 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:08.896441 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-kzqr4" Oct 31 00:00:08.897368 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:08.897317 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-kzqr4" Oct 31 00:00:09.252359 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:09.249798 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-jdcvx" Oct 31 00:00:09.252359 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:09.251693 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-jdcvx" Oct 31 00:00:09.252621 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:09.252565 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-jdcvx" Oct 31 00:00:10.328972 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:10.328937 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:00:14.964612 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:14.964566 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:00:18.078616 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:18.078581 4927 scope.go:122] "RemoveContainer" containerID="f9031fefe28a5a69984b888ddf9fec1563728d25f7360a410a779d78982e2e84" Oct 31 00:00:18.079091 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:18.078672 4927 scope.go:122] "RemoveContainer" containerID="7e91290ddc2de5b8b9c3d339b37db548f6a15fa4c455c24f4edef82c6d27dc3e" Oct 31 00:00:18.079091 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:18.078790 4927 scope.go:122] "RemoveContainer" containerID="dbba66efe8a5e62ffbafcc6bd71292520b6330c4c4387cdade6ea109f7b98a1d" Oct 31 00:00:18.079091 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:18.079009 4927 scope.go:122] "RemoveContainer" containerID="106fb15f9c3d5eed43102e4abfda6045220ff30cbdcfa356f70b53cccc6aca46" Oct 31 00:00:18.079250 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:18.079158 4927 scope.go:122] "RemoveContainer" containerID="33e0ef4f041082c7ca9b918800c59a6abdee9c424543f3babf19c496b0596d4b" Oct 31 00:00:18.721088 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:18.721008 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-34f1ce-29890080-rjm5v" event={"ID":"b22bb02d-b984-4b9b-9198-ec9584512919","Type":"ContainerStarted","Data":"9b41937e36a7f830f80922eff01b34671ebde48bc52631d2d86ad656a4792ce4"} Oct 31 00:00:18.723022 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:18.722995 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-fe01fa-29890080-j8xhq" event={"ID":"5770165a-df6d-43b8-9997-b6fe1c76dc09","Type":"ContainerStarted","Data":"6ef294ad349ac32e7b088d9adfb73419d4f6ce56782ff1673bc93ba14bf2c0f0"} Oct 31 00:00:18.724861 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:18.724835 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-986d37-29890080-hp6pf" event={"ID":"512b87d9-7168-4bd4-a9a0-9e4d9c054d72","Type":"ContainerStarted","Data":"1019a916bdc21a3c9d0bfc8e1794298aa31cb226681ec241aec8b62d582324f3"} Oct 31 00:00:18.726785 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:18.726767 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890080-cd2fx" event={"ID":"ffc39770-0916-4a31-8a21-2f7b91de7031","Type":"ContainerStarted","Data":"52dfb071cdbc59ef7a2cf125e594be3db30818e713b524c6de7a92045e3435b3"} Oct 31 00:00:18.728570 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:18.728552 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-46b586-29890080-w6n4c" event={"ID":"7a3a4a88-0a01-46c2-93be-4983b8fd5146","Type":"ContainerStarted","Data":"1f9a911b9ffbffee62191f8a7c4b79abb8b9177f159487d02d4163981b0ecf46"} Oct 31 00:00:19.626694 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:19.626602 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:00:19.750405 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:00:19.749171 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sqv69" containerName="extractor" Oct 31 00:00:19.752161 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:19.752136 4927 generic.go:356] "Generic (PLEG): container finished" podID="5770165a-df6d-43b8-9997-b6fe1c76dc09" containerID="6ef294ad349ac32e7b088d9adfb73419d4f6ce56782ff1673bc93ba14bf2c0f0" exitCode=60 Oct 31 00:00:19.752303 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:19.752209 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-fe01fa-29890080-j8xhq" event={"ID":"5770165a-df6d-43b8-9997-b6fe1c76dc09","Type":"ContainerDied","Data":"6ef294ad349ac32e7b088d9adfb73419d4f6ce56782ff1673bc93ba14bf2c0f0"} Oct 31 00:00:19.752303 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:19.752269 4927 scope.go:122] "RemoveContainer" containerID="7e91290ddc2de5b8b9c3d339b37db548f6a15fa4c455c24f4edef82c6d27dc3e" Oct 31 00:00:19.752759 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:19.752730 4927 scope.go:122] "RemoveContainer" containerID="6ef294ad349ac32e7b088d9adfb73419d4f6ce56782ff1673bc93ba14bf2c0f0" Oct 31 00:00:19.753149 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:00:19.753113 4927 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-fe01fa-29890080-j8xhq_odh-ai-gateway-infra(5770165a-df6d-43b8-9997-b6fe1c76dc09)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-fe01fa-29890080-j8xhq" podUID="5770165a-df6d-43b8-9997-b6fe1c76dc09" Oct 31 00:00:19.755261 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:19.755241 4927 generic.go:356] "Generic (PLEG): container finished" podID="512b87d9-7168-4bd4-a9a0-9e4d9c054d72" containerID="1019a916bdc21a3c9d0bfc8e1794298aa31cb226681ec241aec8b62d582324f3" exitCode=60 Oct 31 00:00:19.755378 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:19.755311 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-986d37-29890080-hp6pf" event={"ID":"512b87d9-7168-4bd4-a9a0-9e4d9c054d72","Type":"ContainerDied","Data":"1019a916bdc21a3c9d0bfc8e1794298aa31cb226681ec241aec8b62d582324f3"} Oct 31 00:00:19.755760 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:19.755742 4927 scope.go:122] "RemoveContainer" containerID="1019a916bdc21a3c9d0bfc8e1794298aa31cb226681ec241aec8b62d582324f3" Oct 31 00:00:19.756099 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:00:19.756062 4927 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-986d37-29890080-hp6pf_odh-ai-gateway-infra(512b87d9-7168-4bd4-a9a0-9e4d9c054d72)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-986d37-29890080-hp6pf" podUID="512b87d9-7168-4bd4-a9a0-9e4d9c054d72" Oct 31 00:00:19.757848 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:19.757827 4927 generic.go:356] "Generic (PLEG): container finished" podID="ffc39770-0916-4a31-8a21-2f7b91de7031" containerID="52dfb071cdbc59ef7a2cf125e594be3db30818e713b524c6de7a92045e3435b3" exitCode=60 Oct 31 00:00:19.757961 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:19.757859 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890080-cd2fx" event={"ID":"ffc39770-0916-4a31-8a21-2f7b91de7031","Type":"ContainerDied","Data":"52dfb071cdbc59ef7a2cf125e594be3db30818e713b524c6de7a92045e3435b3"} Oct 31 00:00:19.758264 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:19.758241 4927 scope.go:122] "RemoveContainer" containerID="52dfb071cdbc59ef7a2cf125e594be3db30818e713b524c6de7a92045e3435b3" Oct 31 00:00:19.759010 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:00:19.758616 4927 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29890080-cd2fx_odh-ai-gateway-infra(ffc39770-0916-4a31-8a21-2f7b91de7031)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890080-cd2fx" podUID="ffc39770-0916-4a31-8a21-2f7b91de7031" Oct 31 00:00:19.759901 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:19.759878 4927 generic.go:356] "Generic (PLEG): container finished" podID="7a3a4a88-0a01-46c2-93be-4983b8fd5146" containerID="1f9a911b9ffbffee62191f8a7c4b79abb8b9177f159487d02d4163981b0ecf46" exitCode=60 Oct 31 00:00:19.760043 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:19.760028 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-46b586-29890080-w6n4c" event={"ID":"7a3a4a88-0a01-46c2-93be-4983b8fd5146","Type":"ContainerDied","Data":"1f9a911b9ffbffee62191f8a7c4b79abb8b9177f159487d02d4163981b0ecf46"} Oct 31 00:00:19.760300 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:19.760286 4927 scope.go:122] "RemoveContainer" containerID="1f9a911b9ffbffee62191f8a7c4b79abb8b9177f159487d02d4163981b0ecf46" Oct 31 00:00:19.760960 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:00:19.760520 4927 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-46b586-29890080-w6n4c_odh-ai-gateway-infra(7a3a4a88-0a01-46c2-93be-4983b8fd5146)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-46b586-29890080-w6n4c" podUID="7a3a4a88-0a01-46c2-93be-4983b8fd5146" Oct 31 00:00:19.763185 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:19.763167 4927 generic.go:356] "Generic (PLEG): container finished" podID="b22bb02d-b984-4b9b-9198-ec9584512919" containerID="9b41937e36a7f830f80922eff01b34671ebde48bc52631d2d86ad656a4792ce4" exitCode=60 Oct 31 00:00:19.763272 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:19.763243 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-34f1ce-29890080-rjm5v" event={"ID":"b22bb02d-b984-4b9b-9198-ec9584512919","Type":"ContainerDied","Data":"9b41937e36a7f830f80922eff01b34671ebde48bc52631d2d86ad656a4792ce4"} Oct 31 00:00:19.763633 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:19.763617 4927 scope.go:122] "RemoveContainer" containerID="9b41937e36a7f830f80922eff01b34671ebde48bc52631d2d86ad656a4792ce4" Oct 31 00:00:19.763926 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:00:19.763906 4927 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-34f1ce-29890080-rjm5v_odh-ai-gateway-infra(b22bb02d-b984-4b9b-9198-ec9584512919)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-34f1ce-29890080-rjm5v" podUID="b22bb02d-b984-4b9b-9198-ec9584512919" Oct 31 00:00:19.772568 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:19.772550 4927 scope.go:122] "RemoveContainer" containerID="dbba66efe8a5e62ffbafcc6bd71292520b6330c4c4387cdade6ea109f7b98a1d" Oct 31 00:00:19.787948 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:19.787930 4927 scope.go:122] "RemoveContainer" containerID="33e0ef4f041082c7ca9b918800c59a6abdee9c424543f3babf19c496b0596d4b" Oct 31 00:00:19.802986 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:19.802956 4927 scope.go:122] "RemoveContainer" containerID="106fb15f9c3d5eed43102e4abfda6045220ff30cbdcfa356f70b53cccc6aca46" Oct 31 00:00:19.816807 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:19.816787 4927 scope.go:122] "RemoveContainer" containerID="f9031fefe28a5a69984b888ddf9fec1563728d25f7360a410a779d78982e2e84" Oct 31 00:00:19.864782 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:19.864752 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-34f1ce-29890080-rjm5v"] Oct 31 00:00:19.987214 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:19.987181 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-46b586-29890080-w6n4c"] Oct 31 00:00:20.100015 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:20.099980 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-fe01fa-29890080-j8xhq"] Oct 31 00:00:20.233023 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:20.232989 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890080-cd2fx"] Oct 31 00:00:20.352934 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:20.352856 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-986d37-29890080-hp6pf"] Oct 31 00:00:21.003388 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:21.003359 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-46b586-29890080-w6n4c" Oct 31 00:00:21.092791 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:21.092662 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-fe01fa-29890080-j8xhq" Oct 31 00:00:21.103267 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:21.103240 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890080-cd2fx" Oct 31 00:00:21.108373 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:21.107861 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-34f1ce-29890080-rjm5v" Oct 31 00:00:21.108373 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:21.108122 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7a3a4a88-0a01-46c2-93be-4983b8fd5146-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a3a4a88-0a01-46c2-93be-4983b8fd5146-maas-api-service-ca\") pod \"7a3a4a88-0a01-46c2-93be-4983b8fd5146\" (UID: \"7a3a4a88-0a01-46c2-93be-4983b8fd5146\") " Oct 31 00:00:21.108373 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:21.108235 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a3a4a88-0a01-46c2-93be-4983b8fd5146-kube-api-access-fqfx7\" (UniqueName: \"kubernetes.io/projected/7a3a4a88-0a01-46c2-93be-4983b8fd5146-kube-api-access-fqfx7\") pod \"7a3a4a88-0a01-46c2-93be-4983b8fd5146\" (UID: \"7a3a4a88-0a01-46c2-93be-4983b8fd5146\") " Oct 31 00:00:21.108718 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:21.108693 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a3a4a88-0a01-46c2-93be-4983b8fd5146-maas-api-service-ca" pod "7a3a4a88-0a01-46c2-93be-4983b8fd5146" (UID: "7a3a4a88-0a01-46c2-93be-4983b8fd5146"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:00:21.110958 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:21.110931 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a3a4a88-0a01-46c2-93be-4983b8fd5146-kube-api-access-fqfx7" pod "7a3a4a88-0a01-46c2-93be-4983b8fd5146" (UID: "7a3a4a88-0a01-46c2-93be-4983b8fd5146"). InnerVolumeSpecName "kube-api-access-fqfx7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:00:21.112426 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:21.112409 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-986d37-29890080-hp6pf" Oct 31 00:00:21.209293 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:21.209255 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/512b87d9-7168-4bd4-a9a0-9e4d9c054d72-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/512b87d9-7168-4bd4-a9a0-9e4d9c054d72-maas-api-service-ca\") pod \"512b87d9-7168-4bd4-a9a0-9e4d9c054d72\" (UID: \"512b87d9-7168-4bd4-a9a0-9e4d9c054d72\") " Oct 31 00:00:21.209471 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:21.209322 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5770165a-df6d-43b8-9997-b6fe1c76dc09-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5770165a-df6d-43b8-9997-b6fe1c76dc09-maas-api-service-ca\") pod \"5770165a-df6d-43b8-9997-b6fe1c76dc09\" (UID: \"5770165a-df6d-43b8-9997-b6fe1c76dc09\") " Oct 31 00:00:21.209471 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:21.209370 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b22bb02d-b984-4b9b-9198-ec9584512919-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b22bb02d-b984-4b9b-9198-ec9584512919-maas-api-service-ca\") pod \"b22bb02d-b984-4b9b-9198-ec9584512919\" (UID: \"b22bb02d-b984-4b9b-9198-ec9584512919\") " Oct 31 00:00:21.209605 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:21.209474 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ffc39770-0916-4a31-8a21-2f7b91de7031-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ffc39770-0916-4a31-8a21-2f7b91de7031-maas-api-service-ca\") pod \"ffc39770-0916-4a31-8a21-2f7b91de7031\" (UID: \"ffc39770-0916-4a31-8a21-2f7b91de7031\") " Oct 31 00:00:21.209605 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:21.209529 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b22bb02d-b984-4b9b-9198-ec9584512919-kube-api-access-zpmxb\" (UniqueName: \"kubernetes.io/projected/b22bb02d-b984-4b9b-9198-ec9584512919-kube-api-access-zpmxb\") pod \"b22bb02d-b984-4b9b-9198-ec9584512919\" (UID: \"b22bb02d-b984-4b9b-9198-ec9584512919\") " Oct 31 00:00:21.209605 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:21.209588 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/512b87d9-7168-4bd4-a9a0-9e4d9c054d72-kube-api-access-bgdvc\" (UniqueName: \"kubernetes.io/projected/512b87d9-7168-4bd4-a9a0-9e4d9c054d72-kube-api-access-bgdvc\") pod \"512b87d9-7168-4bd4-a9a0-9e4d9c054d72\" (UID: \"512b87d9-7168-4bd4-a9a0-9e4d9c054d72\") " Oct 31 00:00:21.209740 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:21.209622 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5770165a-df6d-43b8-9997-b6fe1c76dc09-kube-api-access-gl5th\" (UniqueName: \"kubernetes.io/projected/5770165a-df6d-43b8-9997-b6fe1c76dc09-kube-api-access-gl5th\") pod \"5770165a-df6d-43b8-9997-b6fe1c76dc09\" (UID: \"5770165a-df6d-43b8-9997-b6fe1c76dc09\") " Oct 31 00:00:21.209740 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:21.209682 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ffc39770-0916-4a31-8a21-2f7b91de7031-kube-api-access-25nmv\" (UniqueName: \"kubernetes.io/projected/ffc39770-0916-4a31-8a21-2f7b91de7031-kube-api-access-25nmv\") pod \"ffc39770-0916-4a31-8a21-2f7b91de7031\" (UID: \"ffc39770-0916-4a31-8a21-2f7b91de7031\") " Oct 31 00:00:21.209838 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:21.209735 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/512b87d9-7168-4bd4-a9a0-9e4d9c054d72-maas-api-service-ca" pod "512b87d9-7168-4bd4-a9a0-9e4d9c054d72" (UID: "512b87d9-7168-4bd4-a9a0-9e4d9c054d72"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:00:21.209887 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:21.209848 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b22bb02d-b984-4b9b-9198-ec9584512919-maas-api-service-ca" pod "b22bb02d-b984-4b9b-9198-ec9584512919" (UID: "b22bb02d-b984-4b9b-9198-ec9584512919"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:00:21.210141 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:21.210118 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ffc39770-0916-4a31-8a21-2f7b91de7031-maas-api-service-ca" pod "ffc39770-0916-4a31-8a21-2f7b91de7031" (UID: "ffc39770-0916-4a31-8a21-2f7b91de7031"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:00:21.210235 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:21.210165 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5770165a-df6d-43b8-9997-b6fe1c76dc09-maas-api-service-ca" pod "5770165a-df6d-43b8-9997-b6fe1c76dc09" (UID: "5770165a-df6d-43b8-9997-b6fe1c76dc09"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:00:21.210858 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:21.210831 4927 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/512b87d9-7168-4bd4-a9a0-9e4d9c054d72-maas-api-service-ca\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:00:21.210964 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:21.210875 4927 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5770165a-df6d-43b8-9997-b6fe1c76dc09-maas-api-service-ca\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:00:21.210964 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:21.210890 4927 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b22bb02d-b984-4b9b-9198-ec9584512919-maas-api-service-ca\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:00:21.210964 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:21.210903 4927 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a3a4a88-0a01-46c2-93be-4983b8fd5146-maas-api-service-ca\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:00:21.210964 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:21.210918 4927 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ffc39770-0916-4a31-8a21-2f7b91de7031-maas-api-service-ca\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:00:21.210964 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:21.210946 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fqfx7\" (UniqueName: \"kubernetes.io/projected/7a3a4a88-0a01-46c2-93be-4983b8fd5146-kube-api-access-fqfx7\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:00:21.212421 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:21.212391 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffc39770-0916-4a31-8a21-2f7b91de7031-kube-api-access-25nmv" pod "ffc39770-0916-4a31-8a21-2f7b91de7031" (UID: "ffc39770-0916-4a31-8a21-2f7b91de7031"). InnerVolumeSpecName "kube-api-access-25nmv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:00:21.212509 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:21.212480 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5770165a-df6d-43b8-9997-b6fe1c76dc09-kube-api-access-gl5th" pod "5770165a-df6d-43b8-9997-b6fe1c76dc09" (UID: "5770165a-df6d-43b8-9997-b6fe1c76dc09"). InnerVolumeSpecName "kube-api-access-gl5th". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:00:21.212613 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:21.212594 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b22bb02d-b984-4b9b-9198-ec9584512919-kube-api-access-zpmxb" pod "b22bb02d-b984-4b9b-9198-ec9584512919" (UID: "b22bb02d-b984-4b9b-9198-ec9584512919"). InnerVolumeSpecName "kube-api-access-zpmxb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:00:21.212700 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:21.212647 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/512b87d9-7168-4bd4-a9a0-9e4d9c054d72-kube-api-access-bgdvc" pod "512b87d9-7168-4bd4-a9a0-9e4d9c054d72" (UID: "512b87d9-7168-4bd4-a9a0-9e4d9c054d72"). InnerVolumeSpecName "kube-api-access-bgdvc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:00:21.312922 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:21.312834 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-25nmv\" (UniqueName: \"kubernetes.io/projected/ffc39770-0916-4a31-8a21-2f7b91de7031-kube-api-access-25nmv\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:00:21.312922 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:21.312870 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zpmxb\" (UniqueName: \"kubernetes.io/projected/b22bb02d-b984-4b9b-9198-ec9584512919-kube-api-access-zpmxb\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:00:21.312922 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:21.312886 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bgdvc\" (UniqueName: \"kubernetes.io/projected/512b87d9-7168-4bd4-a9a0-9e4d9c054d72-kube-api-access-bgdvc\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:00:21.312922 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:21.312900 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gl5th\" (UniqueName: \"kubernetes.io/projected/5770165a-df6d-43b8-9997-b6fe1c76dc09-kube-api-access-gl5th\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:00:21.792964 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:21.792931 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-fe01fa-29890080-j8xhq" Oct 31 00:00:21.793174 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:21.792955 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-fe01fa-29890080-j8xhq" event={"ID":"5770165a-df6d-43b8-9997-b6fe1c76dc09","Type":"ContainerDied","Data":"ce544920adfa12c5d299e943d3e6d56ca8f2fee67370ee8ecc5cd11ccd2beb74"} Oct 31 00:00:21.793174 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:21.793032 4927 scope.go:122] "RemoveContainer" containerID="6ef294ad349ac32e7b088d9adfb73419d4f6ce56782ff1673bc93ba14bf2c0f0" Oct 31 00:00:21.796605 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:21.796574 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-986d37-29890080-hp6pf" event={"ID":"512b87d9-7168-4bd4-a9a0-9e4d9c054d72","Type":"ContainerDied","Data":"f492a23b399ee0d6c2f3206b940d757835650ba0da77bf0f76aa221057803593"} Oct 31 00:00:21.796721 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:21.796679 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-986d37-29890080-hp6pf" Oct 31 00:00:21.806392 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:21.806348 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890080-cd2fx" event={"ID":"ffc39770-0916-4a31-8a21-2f7b91de7031","Type":"ContainerDied","Data":"8bfac1177dcff191ab802d779b5ec4024eccf4bdd77ebbe0a86af092539b4a2c"} Oct 31 00:00:21.806483 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:21.806434 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890080-cd2fx" Oct 31 00:00:21.811407 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:21.811390 4927 scope.go:122] "RemoveContainer" containerID="1019a916bdc21a3c9d0bfc8e1794298aa31cb226681ec241aec8b62d582324f3" Oct 31 00:00:21.812432 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:21.812411 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-46b586-29890080-w6n4c" Oct 31 00:00:21.812622 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:21.812597 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-46b586-29890080-w6n4c" event={"ID":"7a3a4a88-0a01-46c2-93be-4983b8fd5146","Type":"ContainerDied","Data":"466f2dcba77a024730b6e6f9db5bb9ed9167a75448d149e0c94ddebffb35db90"} Oct 31 00:00:21.818145 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:21.814928 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-34f1ce-29890080-rjm5v" Oct 31 00:00:21.818145 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:21.817542 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-34f1ce-29890080-rjm5v" event={"ID":"b22bb02d-b984-4b9b-9198-ec9584512919","Type":"ContainerDied","Data":"eab5c7e02d2e54de1f0ee6ed6c2284aa2f03ade56672ded68384276a8a467058"} Oct 31 00:00:21.831927 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:21.831903 4927 scope.go:122] "RemoveContainer" containerID="52dfb071cdbc59ef7a2cf125e594be3db30818e713b524c6de7a92045e3435b3" Oct 31 00:00:21.842903 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:21.842882 4927 scope.go:122] "RemoveContainer" containerID="1f9a911b9ffbffee62191f8a7c4b79abb8b9177f159487d02d4163981b0ecf46" Oct 31 00:00:21.852417 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:21.852393 4927 scope.go:122] "RemoveContainer" containerID="9b41937e36a7f830f80922eff01b34671ebde48bc52631d2d86ad656a4792ce4" Oct 31 00:00:21.870766 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:00:21.870737 4927 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5770165a_df6d_43b8_9997_b6fe1c76dc09.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a3a4a88_0a01_46c2_93be_4983b8fd5146.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod512b87d9_7168_4bd4_a9a0_9e4d9c054d72.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb22bb02d_b984_4b9b_9198_ec9584512919.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffc39770_0916_4a31_8a21_2f7b91de7031.slice/crio-8bfac1177dcff191ab802d779b5ec4024eccf4bdd77ebbe0a86af092539b4a2c\": RecentStats: unable to find data in memory cache]" Oct 31 00:00:22.108874 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:22.108795 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-34f1ce-29890080-rjm5v"] Oct 31 00:00:22.113795 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:22.113767 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-34f1ce-29890080-rjm5v"] Oct 31 00:00:22.230029 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:22.229989 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890080-cd2fx"] Oct 31 00:00:22.237464 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:22.237434 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890080-cd2fx"] Oct 31 00:00:22.347764 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:22.347734 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-986d37-29890080-hp6pf"] Oct 31 00:00:22.354943 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:22.354920 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-986d37-29890080-hp6pf"] Oct 31 00:00:22.468165 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:22.468132 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-46b586-29890080-w6n4c"] Oct 31 00:00:22.472611 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:22.472580 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-46b586-29890080-w6n4c"] Oct 31 00:00:22.584474 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:22.584446 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-fe01fa-29890080-j8xhq"] Oct 31 00:00:22.589400 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:22.589373 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-fe01fa-29890080-j8xhq"] Oct 31 00:00:23.086702 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:23.086673 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="512b87d9-7168-4bd4-a9a0-9e4d9c054d72" path="/var/lib/kubelet/pods/512b87d9-7168-4bd4-a9a0-9e4d9c054d72/volumes" Oct 31 00:00:23.087110 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:23.087096 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5770165a-df6d-43b8-9997-b6fe1c76dc09" path="/var/lib/kubelet/pods/5770165a-df6d-43b8-9997-b6fe1c76dc09/volumes" Oct 31 00:00:23.087501 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:23.087488 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a3a4a88-0a01-46c2-93be-4983b8fd5146" path="/var/lib/kubelet/pods/7a3a4a88-0a01-46c2-93be-4983b8fd5146/volumes" Oct 31 00:00:23.088149 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:23.088134 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b22bb02d-b984-4b9b-9198-ec9584512919" path="/var/lib/kubelet/pods/b22bb02d-b984-4b9b-9198-ec9584512919/volumes" Oct 31 00:00:23.088549 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:23.088537 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ffc39770-0916-4a31-8a21-2f7b91de7031" path="/var/lib/kubelet/pods/ffc39770-0916-4a31-8a21-2f7b91de7031/volumes" Oct 31 00:00:23.480347 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:23.480301 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:00:24.401010 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:24.400977 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:00:29.129137 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:29.129101 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:00:33.860360 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:33.853464 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:00:38.655357 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:38.651090 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:00:43.215829 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:43.215794 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:00:46.637531 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:46.637211 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:00:47.971352 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:47.970875 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:00:49.678096 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:00:49.678053 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sqv69" containerName="extractor" Oct 31 00:00:52.601167 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:52.601127 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:00:53.167910 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:53.167877 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-fe01fa/distinct-2-w5-fe01fa-kserve-79b99966d8-dkhzd"] Oct 31 00:00:53.168501 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:53.168486 4927 cpu_manager.go:414] "containerMap: removing container" podUID="5770165a-df6d-43b8-9997-b6fe1c76dc09" containerName="cleanup" Oct 31 00:00:53.168566 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:53.168502 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5770165a-df6d-43b8-9997-b6fe1c76dc09" containerName="cleanup" Oct 31 00:00:53.168566 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:53.168523 4927 cpu_manager.go:414] "containerMap: removing container" podUID="7a3a4a88-0a01-46c2-93be-4983b8fd5146" containerName="cleanup" Oct 31 00:00:53.168566 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:53.168529 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a3a4a88-0a01-46c2-93be-4983b8fd5146" containerName="cleanup" Oct 31 00:00:53.168566 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:53.168536 4927 cpu_manager.go:414] "containerMap: removing container" podUID="512b87d9-7168-4bd4-a9a0-9e4d9c054d72" containerName="cleanup" Oct 31 00:00:53.168566 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:53.168542 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="512b87d9-7168-4bd4-a9a0-9e4d9c054d72" containerName="cleanup" Oct 31 00:00:53.168566 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:53.168550 4927 cpu_manager.go:414] "containerMap: removing container" podUID="b22bb02d-b984-4b9b-9198-ec9584512919" containerName="cleanup" Oct 31 00:00:53.168566 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:53.168554 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b22bb02d-b984-4b9b-9198-ec9584512919" containerName="cleanup" Oct 31 00:00:53.168566 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:53.168562 4927 cpu_manager.go:414] "containerMap: removing container" podUID="7326c89a-a135-479e-8abb-3e28ede9179d" containerName="image-pruner" Oct 31 00:00:53.168566 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:53.168567 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7326c89a-a135-479e-8abb-3e28ede9179d" containerName="image-pruner" Oct 31 00:00:53.168833 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:53.168576 4927 cpu_manager.go:414] "containerMap: removing container" podUID="7a3a4a88-0a01-46c2-93be-4983b8fd5146" containerName="cleanup" Oct 31 00:00:53.168833 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:53.168581 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a3a4a88-0a01-46c2-93be-4983b8fd5146" containerName="cleanup" Oct 31 00:00:53.168833 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:53.168589 4927 cpu_manager.go:414] "containerMap: removing container" podUID="512b87d9-7168-4bd4-a9a0-9e4d9c054d72" containerName="cleanup" Oct 31 00:00:53.168833 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:53.168593 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="512b87d9-7168-4bd4-a9a0-9e4d9c054d72" containerName="cleanup" Oct 31 00:00:53.168833 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:53.168606 4927 cpu_manager.go:414] "containerMap: removing container" podUID="b22bb02d-b984-4b9b-9198-ec9584512919" containerName="cleanup" Oct 31 00:00:53.168833 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:53.168611 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b22bb02d-b984-4b9b-9198-ec9584512919" containerName="cleanup" Oct 31 00:00:53.168833 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:53.168617 4927 cpu_manager.go:414] "containerMap: removing container" podUID="5770165a-df6d-43b8-9997-b6fe1c76dc09" containerName="cleanup" Oct 31 00:00:53.168833 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:53.168621 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5770165a-df6d-43b8-9997-b6fe1c76dc09" containerName="cleanup" Oct 31 00:00:53.168833 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:53.168630 4927 cpu_manager.go:414] "containerMap: removing container" podUID="ffc39770-0916-4a31-8a21-2f7b91de7031" containerName="cleanup" Oct 31 00:00:53.168833 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:53.168634 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ffc39770-0916-4a31-8a21-2f7b91de7031" containerName="cleanup" Oct 31 00:00:53.168833 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:53.168646 4927 cpu_manager.go:414] "containerMap: removing container" podUID="ffc39770-0916-4a31-8a21-2f7b91de7031" containerName="cleanup" Oct 31 00:00:53.168833 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:53.168650 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ffc39770-0916-4a31-8a21-2f7b91de7031" containerName="cleanup" Oct 31 00:00:53.168833 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:53.168742 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="b22bb02d-b984-4b9b-9198-ec9584512919" containerName="cleanup" Oct 31 00:00:53.168833 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:53.168748 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="ffc39770-0916-4a31-8a21-2f7b91de7031" containerName="cleanup" Oct 31 00:00:53.168833 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:53.168757 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="ffc39770-0916-4a31-8a21-2f7b91de7031" containerName="cleanup" Oct 31 00:00:53.168833 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:53.168763 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="5770165a-df6d-43b8-9997-b6fe1c76dc09" containerName="cleanup" Oct 31 00:00:53.168833 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:53.168770 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a3a4a88-0a01-46c2-93be-4983b8fd5146" containerName="cleanup" Oct 31 00:00:53.168833 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:53.168776 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="512b87d9-7168-4bd4-a9a0-9e4d9c054d72" containerName="cleanup" Oct 31 00:00:53.168833 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:53.168780 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="b22bb02d-b984-4b9b-9198-ec9584512919" containerName="cleanup" Oct 31 00:00:53.168833 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:53.168785 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="7326c89a-a135-479e-8abb-3e28ede9179d" containerName="image-pruner" Oct 31 00:00:53.168833 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:53.168790 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a3a4a88-0a01-46c2-93be-4983b8fd5146" containerName="cleanup" Oct 31 00:00:53.168833 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:53.168795 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="512b87d9-7168-4bd4-a9a0-9e4d9c054d72" containerName="cleanup" Oct 31 00:00:53.168833 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:53.168802 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="5770165a-df6d-43b8-9997-b6fe1c76dc09" containerName="cleanup" Oct 31 00:00:53.168833 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:53.168808 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="b22bb02d-b984-4b9b-9198-ec9584512919" containerName="cleanup" Oct 31 00:00:53.168833 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:53.168814 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="5770165a-df6d-43b8-9997-b6fe1c76dc09" containerName="cleanup" Oct 31 00:00:53.168833 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:53.168826 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="512b87d9-7168-4bd4-a9a0-9e4d9c054d72" containerName="cleanup" Oct 31 00:00:53.172544 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:53.172528 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-fe01fa/distinct-2-w5-fe01fa-kserve-79b99966d8-dkhzd" Oct 31 00:00:53.251344 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:53.251308 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-fe01fa\"/\"openshift-service-ca.crt\"" Oct 31 00:00:53.262096 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:53.262063 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ksfpk\" (UniqueName: \"kubernetes.io/projected/d5c1c9e8-ce47-4868-860d-b8413ff74ab5-kube-api-access-ksfpk\") pod \"distinct-2-w5-fe01fa-kserve-79b99966d8-dkhzd\" (UID: \"d5c1c9e8-ce47-4868-860d-b8413ff74ab5\") " pod="e2e-models-e2e-worker-w5-fe01fa/distinct-2-w5-fe01fa-kserve-79b99966d8-dkhzd" Oct 31 00:00:53.262280 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:53.262119 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d5c1c9e8-ce47-4868-860d-b8413ff74ab5-home\") pod \"distinct-2-w5-fe01fa-kserve-79b99966d8-dkhzd\" (UID: \"d5c1c9e8-ce47-4868-860d-b8413ff74ab5\") " pod="e2e-models-e2e-worker-w5-fe01fa/distinct-2-w5-fe01fa-kserve-79b99966d8-dkhzd" Oct 31 00:00:53.262461 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:53.262268 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d5c1c9e8-ce47-4868-860d-b8413ff74ab5-model-cache\") pod \"distinct-2-w5-fe01fa-kserve-79b99966d8-dkhzd\" (UID: \"d5c1c9e8-ce47-4868-860d-b8413ff74ab5\") " pod="e2e-models-e2e-worker-w5-fe01fa/distinct-2-w5-fe01fa-kserve-79b99966d8-dkhzd" Oct 31 00:00:53.262461 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:53.262389 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d5c1c9e8-ce47-4868-860d-b8413ff74ab5-tls-certs\") pod \"distinct-2-w5-fe01fa-kserve-79b99966d8-dkhzd\" (UID: \"d5c1c9e8-ce47-4868-860d-b8413ff74ab5\") " pod="e2e-models-e2e-worker-w5-fe01fa/distinct-2-w5-fe01fa-kserve-79b99966d8-dkhzd" Oct 31 00:00:53.262461 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:53.262435 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d5c1c9e8-ce47-4868-860d-b8413ff74ab5-dshm\") pod \"distinct-2-w5-fe01fa-kserve-79b99966d8-dkhzd\" (UID: \"d5c1c9e8-ce47-4868-860d-b8413ff74ab5\") " pod="e2e-models-e2e-worker-w5-fe01fa/distinct-2-w5-fe01fa-kserve-79b99966d8-dkhzd" Oct 31 00:00:53.262646 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:53.262534 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d5c1c9e8-ce47-4868-860d-b8413ff74ab5-tmp-dir\") pod \"distinct-2-w5-fe01fa-kserve-79b99966d8-dkhzd\" (UID: \"d5c1c9e8-ce47-4868-860d-b8413ff74ab5\") " pod="e2e-models-e2e-worker-w5-fe01fa/distinct-2-w5-fe01fa-kserve-79b99966d8-dkhzd" Oct 31 00:00:53.280480 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:53.280449 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-fe01fa\"/\"kube-root-ca.crt\"" Oct 31 00:00:53.280638 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:53.280556 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-fe01fa\"/\"distinct-2-w5-fe01fa-kserve-self-signed-certs\"" Oct 31 00:00:53.280710 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:53.280659 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-fe01fa\"/\"default-dockercfg-kv2s5\"" Oct 31 00:00:53.285997 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:53.285968 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:00:53.286409 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:53.286390 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-fe01fa/distinct-2-w5-fe01fa-kserve-79b99966d8-dkhzd"] Oct 31 00:00:53.366357 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:53.366187 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ksfpk\" (UniqueName: \"kubernetes.io/projected/d5c1c9e8-ce47-4868-860d-b8413ff74ab5-kube-api-access-ksfpk\") pod \"distinct-2-w5-fe01fa-kserve-79b99966d8-dkhzd\" (UID: \"d5c1c9e8-ce47-4868-860d-b8413ff74ab5\") " pod="e2e-models-e2e-worker-w5-fe01fa/distinct-2-w5-fe01fa-kserve-79b99966d8-dkhzd" Oct 31 00:00:53.366357 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:53.366255 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d5c1c9e8-ce47-4868-860d-b8413ff74ab5-home\") pod \"distinct-2-w5-fe01fa-kserve-79b99966d8-dkhzd\" (UID: \"d5c1c9e8-ce47-4868-860d-b8413ff74ab5\") " pod="e2e-models-e2e-worker-w5-fe01fa/distinct-2-w5-fe01fa-kserve-79b99966d8-dkhzd" Oct 31 00:00:53.366357 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:53.366357 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d5c1c9e8-ce47-4868-860d-b8413ff74ab5-model-cache\") pod \"distinct-2-w5-fe01fa-kserve-79b99966d8-dkhzd\" (UID: \"d5c1c9e8-ce47-4868-860d-b8413ff74ab5\") " pod="e2e-models-e2e-worker-w5-fe01fa/distinct-2-w5-fe01fa-kserve-79b99966d8-dkhzd" Oct 31 00:00:53.366661 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:53.366454 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d5c1c9e8-ce47-4868-860d-b8413ff74ab5-tls-certs\") pod \"distinct-2-w5-fe01fa-kserve-79b99966d8-dkhzd\" (UID: \"d5c1c9e8-ce47-4868-860d-b8413ff74ab5\") " pod="e2e-models-e2e-worker-w5-fe01fa/distinct-2-w5-fe01fa-kserve-79b99966d8-dkhzd" Oct 31 00:00:53.366661 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:53.366502 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d5c1c9e8-ce47-4868-860d-b8413ff74ab5-dshm\") pod \"distinct-2-w5-fe01fa-kserve-79b99966d8-dkhzd\" (UID: \"d5c1c9e8-ce47-4868-860d-b8413ff74ab5\") " pod="e2e-models-e2e-worker-w5-fe01fa/distinct-2-w5-fe01fa-kserve-79b99966d8-dkhzd" Oct 31 00:00:53.366661 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:53.366596 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d5c1c9e8-ce47-4868-860d-b8413ff74ab5-tmp-dir\") pod \"distinct-2-w5-fe01fa-kserve-79b99966d8-dkhzd\" (UID: \"d5c1c9e8-ce47-4868-860d-b8413ff74ab5\") " pod="e2e-models-e2e-worker-w5-fe01fa/distinct-2-w5-fe01fa-kserve-79b99966d8-dkhzd" Oct 31 00:00:53.370204 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:53.367134 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d5c1c9e8-ce47-4868-860d-b8413ff74ab5-tmp-dir\") pod \"distinct-2-w5-fe01fa-kserve-79b99966d8-dkhzd\" (UID: \"d5c1c9e8-ce47-4868-860d-b8413ff74ab5\") " pod="e2e-models-e2e-worker-w5-fe01fa/distinct-2-w5-fe01fa-kserve-79b99966d8-dkhzd" Oct 31 00:00:53.370204 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:53.367735 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d5c1c9e8-ce47-4868-860d-b8413ff74ab5-home\") pod \"distinct-2-w5-fe01fa-kserve-79b99966d8-dkhzd\" (UID: \"d5c1c9e8-ce47-4868-860d-b8413ff74ab5\") " pod="e2e-models-e2e-worker-w5-fe01fa/distinct-2-w5-fe01fa-kserve-79b99966d8-dkhzd" Oct 31 00:00:53.370204 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:53.367968 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d5c1c9e8-ce47-4868-860d-b8413ff74ab5-model-cache\") pod \"distinct-2-w5-fe01fa-kserve-79b99966d8-dkhzd\" (UID: \"d5c1c9e8-ce47-4868-860d-b8413ff74ab5\") " pod="e2e-models-e2e-worker-w5-fe01fa/distinct-2-w5-fe01fa-kserve-79b99966d8-dkhzd" Oct 31 00:00:53.373763 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:53.373721 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d5c1c9e8-ce47-4868-860d-b8413ff74ab5-tls-certs\") pod \"distinct-2-w5-fe01fa-kserve-79b99966d8-dkhzd\" (UID: \"d5c1c9e8-ce47-4868-860d-b8413ff74ab5\") " pod="e2e-models-e2e-worker-w5-fe01fa/distinct-2-w5-fe01fa-kserve-79b99966d8-dkhzd" Oct 31 00:00:53.391356 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:53.383290 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d5c1c9e8-ce47-4868-860d-b8413ff74ab5-dshm\") pod \"distinct-2-w5-fe01fa-kserve-79b99966d8-dkhzd\" (UID: \"d5c1c9e8-ce47-4868-860d-b8413ff74ab5\") " pod="e2e-models-e2e-worker-w5-fe01fa/distinct-2-w5-fe01fa-kserve-79b99966d8-dkhzd" Oct 31 00:00:53.448406 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:53.445977 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ksfpk\" (UniqueName: \"kubernetes.io/projected/d5c1c9e8-ce47-4868-860d-b8413ff74ab5-kube-api-access-ksfpk\") pod \"distinct-2-w5-fe01fa-kserve-79b99966d8-dkhzd\" (UID: \"d5c1c9e8-ce47-4868-860d-b8413ff74ab5\") " pod="e2e-models-e2e-worker-w5-fe01fa/distinct-2-w5-fe01fa-kserve-79b99966d8-dkhzd" Oct 31 00:00:53.485050 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:53.485012 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-fe01fa/distinct-2-w5-fe01fa-kserve-79b99966d8-dkhzd" Oct 31 00:00:53.901476 ip-10-0-141-180 kubenswrapper[4927]: W1031 00:00:53.901449 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd5c1c9e8_ce47_4868_860d_b8413ff74ab5.slice/crio-3eefe03eb843117afeb5f40d3a01897fec7324076d59182319b448a52dc8d5e0 WatchSource:0}: Error finding container 3eefe03eb843117afeb5f40d3a01897fec7324076d59182319b448a52dc8d5e0: Status 404 returned error can't find the container with id 3eefe03eb843117afeb5f40d3a01897fec7324076d59182319b448a52dc8d5e0 Oct 31 00:00:53.902673 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:53.902649 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-fe01fa/distinct-2-w5-fe01fa-kserve-79b99966d8-dkhzd"] Oct 31 00:00:54.166103 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:54.166064 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-fe01fa/distinct-2-w5-fe01fa-kserve-79b99966d8-dkhzd" event={"ID":"d5c1c9e8-ce47-4868-860d-b8413ff74ab5","Type":"ContainerStarted","Data":"a3f6dc6ac8204c37462a61c318ca30feb277f09f4ed9fbe6e854ef2d90b48a4f"} Oct 31 00:00:54.166281 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:54.166115 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-fe01fa/distinct-2-w5-fe01fa-kserve-79b99966d8-dkhzd" event={"ID":"d5c1c9e8-ce47-4868-860d-b8413ff74ab5","Type":"ContainerStarted","Data":"3eefe03eb843117afeb5f40d3a01897fec7324076d59182319b448a52dc8d5e0"} Oct 31 00:00:54.288148 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:54.288099 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-fe01fa/distinct-2-w5-fe01fa-kserve-79b99966d8-dkhzd" podStartSLOduration=1.28808494 podStartE2EDuration="1.28808494s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:00:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:00:54.286561193 +0000 UTC m=+1167.322892015" watchObservedRunningTime="2026-10-31 00:00:54.28808494 +0000 UTC m=+1167.324415761" Oct 31 00:00:54.485732 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:54.485643 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-fe01fa/distinct-2-w5-fe01fa-kserve-79b99966d8-dkhzd" Oct 31 00:00:56.538623 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:56.538583 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-46e53d8c-h6qr7"] Oct 31 00:00:56.539467 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:56.539438 4927 cpu_manager.go:414] "containerMap: removing container" podUID="b22bb02d-b984-4b9b-9198-ec9584512919" containerName="cleanup" Oct 31 00:00:56.539467 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:56.539465 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b22bb02d-b984-4b9b-9198-ec9584512919" containerName="cleanup" Oct 31 00:00:56.539630 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:56.539483 4927 cpu_manager.go:414] "containerMap: removing container" podUID="7a3a4a88-0a01-46c2-93be-4983b8fd5146" containerName="cleanup" Oct 31 00:00:56.539630 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:56.539489 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a3a4a88-0a01-46c2-93be-4983b8fd5146" containerName="cleanup" Oct 31 00:00:56.539630 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:56.539498 4927 cpu_manager.go:414] "containerMap: removing container" podUID="512b87d9-7168-4bd4-a9a0-9e4d9c054d72" containerName="cleanup" Oct 31 00:00:56.539630 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:56.539503 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="512b87d9-7168-4bd4-a9a0-9e4d9c054d72" containerName="cleanup" Oct 31 00:00:56.539630 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:56.539518 4927 cpu_manager.go:414] "containerMap: removing container" podUID="5770165a-df6d-43b8-9997-b6fe1c76dc09" containerName="cleanup" Oct 31 00:00:56.539630 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:56.539522 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5770165a-df6d-43b8-9997-b6fe1c76dc09" containerName="cleanup" Oct 31 00:00:56.539630 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:56.539532 4927 cpu_manager.go:414] "containerMap: removing container" podUID="ffc39770-0916-4a31-8a21-2f7b91de7031" containerName="cleanup" Oct 31 00:00:56.539630 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:56.539537 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ffc39770-0916-4a31-8a21-2f7b91de7031" containerName="cleanup" Oct 31 00:00:56.539870 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:56.539640 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="ffc39770-0916-4a31-8a21-2f7b91de7031" containerName="cleanup" Oct 31 00:00:56.539870 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:56.539651 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a3a4a88-0a01-46c2-93be-4983b8fd5146" containerName="cleanup" Oct 31 00:00:56.543424 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:56.543403 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46e53d8c-h6qr7" Oct 31 00:00:56.603759 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:56.603730 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-46e53d8c-h6qr7"] Oct 31 00:00:56.613993 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:56.613964 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91a1209c-7d60-4ea7-af17-ad33b985792e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-46e53d8c-h6qr7\" (UID: \"91a1209c-7d60-4ea7-af17-ad33b985792e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46e53d8c-h6qr7" Oct 31 00:00:56.614274 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:56.614254 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91a1209c-7d60-4ea7-af17-ad33b985792e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-46e53d8c-h6qr7\" (UID: \"91a1209c-7d60-4ea7-af17-ad33b985792e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46e53d8c-h6qr7" Oct 31 00:00:56.614410 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:56.614286 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rnqrh\" (UniqueName: \"kubernetes.io/projected/91a1209c-7d60-4ea7-af17-ad33b985792e-kube-api-access-rnqrh\") pod \"maas-api-revoke-sub-46e53d8c-h6qr7\" (UID: \"91a1209c-7d60-4ea7-af17-ad33b985792e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46e53d8c-h6qr7" Oct 31 00:00:56.649901 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:56.648624 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:00:56.649901 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:56.648659 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4fdzs\"" Oct 31 00:00:56.715376 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:56.715340 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91a1209c-7d60-4ea7-af17-ad33b985792e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-46e53d8c-h6qr7\" (UID: \"91a1209c-7d60-4ea7-af17-ad33b985792e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46e53d8c-h6qr7" Oct 31 00:00:56.715376 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:56.715378 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rnqrh\" (UniqueName: \"kubernetes.io/projected/91a1209c-7d60-4ea7-af17-ad33b985792e-kube-api-access-rnqrh\") pod \"maas-api-revoke-sub-46e53d8c-h6qr7\" (UID: \"91a1209c-7d60-4ea7-af17-ad33b985792e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46e53d8c-h6qr7" Oct 31 00:00:56.715644 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:56.715421 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91a1209c-7d60-4ea7-af17-ad33b985792e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-46e53d8c-h6qr7\" (UID: \"91a1209c-7d60-4ea7-af17-ad33b985792e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46e53d8c-h6qr7" Oct 31 00:00:56.716101 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:56.716075 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91a1209c-7d60-4ea7-af17-ad33b985792e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-46e53d8c-h6qr7\" (UID: \"91a1209c-7d60-4ea7-af17-ad33b985792e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46e53d8c-h6qr7" Oct 31 00:00:56.717728 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:56.717704 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91a1209c-7d60-4ea7-af17-ad33b985792e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-46e53d8c-h6qr7\" (UID: \"91a1209c-7d60-4ea7-af17-ad33b985792e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46e53d8c-h6qr7" Oct 31 00:00:56.778801 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:56.778771 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rnqrh\" (UniqueName: \"kubernetes.io/projected/91a1209c-7d60-4ea7-af17-ad33b985792e-kube-api-access-rnqrh\") pod \"maas-api-revoke-sub-46e53d8c-h6qr7\" (UID: \"91a1209c-7d60-4ea7-af17-ad33b985792e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46e53d8c-h6qr7" Oct 31 00:00:56.861379 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:56.861282 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46e53d8c-h6qr7" Oct 31 00:00:57.232181 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:57.232143 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:00:57.263501 ip-10-0-141-180 kubenswrapper[4927]: W1031 00:00:57.263471 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91a1209c_7d60_4ea7_af17_ad33b985792e.slice/crio-7e66a4e33d2f7b9fc0d455252288024ff3b44971bfc4943f2fa5e5d8dbcb62f0 WatchSource:0}: Error finding container 7e66a4e33d2f7b9fc0d455252288024ff3b44971bfc4943f2fa5e5d8dbcb62f0: Status 404 returned error can't find the container with id 7e66a4e33d2f7b9fc0d455252288024ff3b44971bfc4943f2fa5e5d8dbcb62f0 Oct 31 00:00:57.263635 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:57.263601 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-46e53d8c-h6qr7"] Oct 31 00:00:58.198515 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:58.198477 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46e53d8c-h6qr7" event={"ID":"91a1209c-7d60-4ea7-af17-ad33b985792e","Type":"ContainerStarted","Data":"0a21023ac4719723f21af0c0e10d57cd8ac19b11e93eb5444f5a0b08de348335"} Oct 31 00:00:58.198916 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:58.198575 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46e53d8c-h6qr7" event={"ID":"91a1209c-7d60-4ea7-af17-ad33b985792e","Type":"ContainerStarted","Data":"7e66a4e33d2f7b9fc0d455252288024ff3b44971bfc4943f2fa5e5d8dbcb62f0"} Oct 31 00:00:58.319697 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:58.319642 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46e53d8c-h6qr7" podStartSLOduration=2.319629657 podStartE2EDuration="2.319629657s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:00:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:00:58.318808871 +0000 UTC m=+1171.355139696" watchObservedRunningTime="2026-10-31 00:00:58.319629657 +0000 UTC m=+1171.355960529" Oct 31 00:00:59.210404 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:59.210357 4927 generic.go:356] "Generic (PLEG): container finished" podID="91a1209c-7d60-4ea7-af17-ad33b985792e" containerID="0a21023ac4719723f21af0c0e10d57cd8ac19b11e93eb5444f5a0b08de348335" exitCode=0 Oct 31 00:00:59.210960 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:00:59.210406 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46e53d8c-h6qr7" event={"ID":"91a1209c-7d60-4ea7-af17-ad33b985792e","Type":"ContainerDied","Data":"0a21023ac4719723f21af0c0e10d57cd8ac19b11e93eb5444f5a0b08de348335"} Oct 31 00:01:00.428931 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:00.428902 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46e53d8c-h6qr7" Oct 31 00:01:00.499089 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:00.499021 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/91a1209c-7d60-4ea7-af17-ad33b985792e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91a1209c-7d60-4ea7-af17-ad33b985792e-maas-api-xks-ca\") pod \"91a1209c-7d60-4ea7-af17-ad33b985792e\" (UID: \"91a1209c-7d60-4ea7-af17-ad33b985792e\") " Oct 31 00:01:00.499236 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:00.499205 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/91a1209c-7d60-4ea7-af17-ad33b985792e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91a1209c-7d60-4ea7-af17-ad33b985792e-maas-api-openshift-ca\") pod \"91a1209c-7d60-4ea7-af17-ad33b985792e\" (UID: \"91a1209c-7d60-4ea7-af17-ad33b985792e\") " Oct 31 00:01:00.499299 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:00.499279 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91a1209c-7d60-4ea7-af17-ad33b985792e-kube-api-access-rnqrh\" (UniqueName: \"kubernetes.io/projected/91a1209c-7d60-4ea7-af17-ad33b985792e-kube-api-access-rnqrh\") pod \"91a1209c-7d60-4ea7-af17-ad33b985792e\" (UID: \"91a1209c-7d60-4ea7-af17-ad33b985792e\") " Oct 31 00:01:00.499589 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:00.499549 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91a1209c-7d60-4ea7-af17-ad33b985792e-maas-api-openshift-ca" pod "91a1209c-7d60-4ea7-af17-ad33b985792e" (UID: "91a1209c-7d60-4ea7-af17-ad33b985792e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:01:00.499794 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:00.499779 4927 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91a1209c-7d60-4ea7-af17-ad33b985792e-maas-api-openshift-ca\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:01:00.501208 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:00.501184 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/91a1209c-7d60-4ea7-af17-ad33b985792e-maas-api-xks-ca" pod "91a1209c-7d60-4ea7-af17-ad33b985792e" (UID: "91a1209c-7d60-4ea7-af17-ad33b985792e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:01:00.501298 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:00.501255 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91a1209c-7d60-4ea7-af17-ad33b985792e-kube-api-access-rnqrh" pod "91a1209c-7d60-4ea7-af17-ad33b985792e" (UID: "91a1209c-7d60-4ea7-af17-ad33b985792e"). InnerVolumeSpecName "kube-api-access-rnqrh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:01:00.600567 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:00.600534 4927 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91a1209c-7d60-4ea7-af17-ad33b985792e-maas-api-xks-ca\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:01:00.600567 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:00.600565 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rnqrh\" (UniqueName: \"kubernetes.io/projected/91a1209c-7d60-4ea7-af17-ad33b985792e-kube-api-access-rnqrh\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:01:01.234658 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:01.234615 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46e53d8c-h6qr7" event={"ID":"91a1209c-7d60-4ea7-af17-ad33b985792e","Type":"ContainerDied","Data":"7e66a4e33d2f7b9fc0d455252288024ff3b44971bfc4943f2fa5e5d8dbcb62f0"} Oct 31 00:01:01.234841 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:01.234678 4927 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7e66a4e33d2f7b9fc0d455252288024ff3b44971bfc4943f2fa5e5d8dbcb62f0" Oct 31 00:01:01.234841 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:01.234639 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46e53d8c-h6qr7" Oct 31 00:01:01.453268 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:01.453229 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:01:01.797417 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:01.797103 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:01:03.065984 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:03.065948 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a414f1b5-nwttq"] Oct 31 00:01:03.117031 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:03.116992 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a414f1b5-nwttq"] Oct 31 00:01:03.485580 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:03.485537 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-fe01fa/distinct-2-w5-fe01fa-kserve-79b99966d8-dkhzd" Oct 31 00:01:03.498633 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:03.498605 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-fe01fa/distinct-2-w5-fe01fa-kserve-79b99966d8-dkhzd" Oct 31 00:01:04.275386 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:04.275357 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-fe01fa/distinct-2-w5-fe01fa-kserve-79b99966d8-dkhzd" Oct 31 00:01:05.087137 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:05.087105 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa953e7a-5616-4ec2-aea5-c554ddcc648c" path="/var/lib/kubelet/pods/aa953e7a-5616-4ec2-aea5-c554ddcc648c/volumes" Oct 31 00:01:06.378327 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:06.378293 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:01:10.987591 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:10.987557 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:01:11.116438 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:11.116396 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b0a3b296-6qrvt"] Oct 31 00:01:11.125716 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:11.124512 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b0a3b296-6qrvt"] Oct 31 00:01:13.087542 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:13.087502 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a2ebfe5-08c3-4d7e-a32c-e5ee5ef4673e" path="/var/lib/kubelet/pods/0a2ebfe5-08c3-4d7e-a32c-e5ee5ef4673e/volumes" Oct 31 00:01:13.087915 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:13.087853 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-35a187/distinct-2-w6-35a187-kserve-7ff4b6dc55-944g6"] Oct 31 00:01:13.088462 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:13.088437 4927 cpu_manager.go:414] "containerMap: removing container" podUID="91a1209c-7d60-4ea7-af17-ad33b985792e" containerName="revoke-keys" Oct 31 00:01:13.088462 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:13.088463 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91a1209c-7d60-4ea7-af17-ad33b985792e" containerName="revoke-keys" Oct 31 00:01:13.088596 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:13.088586 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="91a1209c-7d60-4ea7-af17-ad33b985792e" containerName="revoke-keys" Oct 31 00:01:13.093194 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:13.093176 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-35a187/distinct-2-w6-35a187-kserve-7ff4b6dc55-944g6" Oct 31 00:01:13.150130 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:13.150092 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-35a187/distinct-2-w6-35a187-kserve-7ff4b6dc55-944g6"] Oct 31 00:01:13.190069 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:13.190038 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:01:13.193571 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:13.193547 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rkn2x\" (UniqueName: \"kubernetes.io/projected/429ee092-42d8-475e-b0c6-b5a9930766fc-kube-api-access-rkn2x\") pod \"distinct-2-w6-35a187-kserve-7ff4b6dc55-944g6\" (UID: \"429ee092-42d8-475e-b0c6-b5a9930766fc\") " pod="e2e-models-e2e-worker-w6-35a187/distinct-2-w6-35a187-kserve-7ff4b6dc55-944g6" Oct 31 00:01:13.193662 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:13.193588 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/429ee092-42d8-475e-b0c6-b5a9930766fc-tmp-dir\") pod \"distinct-2-w6-35a187-kserve-7ff4b6dc55-944g6\" (UID: \"429ee092-42d8-475e-b0c6-b5a9930766fc\") " pod="e2e-models-e2e-worker-w6-35a187/distinct-2-w6-35a187-kserve-7ff4b6dc55-944g6" Oct 31 00:01:13.193705 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:13.193667 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/429ee092-42d8-475e-b0c6-b5a9930766fc-model-cache\") pod \"distinct-2-w6-35a187-kserve-7ff4b6dc55-944g6\" (UID: \"429ee092-42d8-475e-b0c6-b5a9930766fc\") " pod="e2e-models-e2e-worker-w6-35a187/distinct-2-w6-35a187-kserve-7ff4b6dc55-944g6" Oct 31 00:01:13.193798 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:13.193764 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/429ee092-42d8-475e-b0c6-b5a9930766fc-tls-certs\") pod \"distinct-2-w6-35a187-kserve-7ff4b6dc55-944g6\" (UID: \"429ee092-42d8-475e-b0c6-b5a9930766fc\") " pod="e2e-models-e2e-worker-w6-35a187/distinct-2-w6-35a187-kserve-7ff4b6dc55-944g6" Oct 31 00:01:13.193852 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:13.193839 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/429ee092-42d8-475e-b0c6-b5a9930766fc-home\") pod \"distinct-2-w6-35a187-kserve-7ff4b6dc55-944g6\" (UID: \"429ee092-42d8-475e-b0c6-b5a9930766fc\") " pod="e2e-models-e2e-worker-w6-35a187/distinct-2-w6-35a187-kserve-7ff4b6dc55-944g6" Oct 31 00:01:13.193887 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:13.193864 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/429ee092-42d8-475e-b0c6-b5a9930766fc-dshm\") pod \"distinct-2-w6-35a187-kserve-7ff4b6dc55-944g6\" (UID: \"429ee092-42d8-475e-b0c6-b5a9930766fc\") " pod="e2e-models-e2e-worker-w6-35a187/distinct-2-w6-35a187-kserve-7ff4b6dc55-944g6" Oct 31 00:01:13.196220 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:13.196200 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-35a187\"/\"openshift-service-ca.crt\"" Oct 31 00:01:13.196220 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:13.196217 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-35a187\"/\"kube-root-ca.crt\"" Oct 31 00:01:13.196410 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:13.196289 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-35a187\"/\"default-dockercfg-bz6sc\"" Oct 31 00:01:13.198007 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:13.197982 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-35a187\"/\"distinct-2-w6-35a187-kserve-self-signed-certs\"" Oct 31 00:01:13.307363 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:13.297742 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rkn2x\" (UniqueName: \"kubernetes.io/projected/429ee092-42d8-475e-b0c6-b5a9930766fc-kube-api-access-rkn2x\") pod \"distinct-2-w6-35a187-kserve-7ff4b6dc55-944g6\" (UID: \"429ee092-42d8-475e-b0c6-b5a9930766fc\") " pod="e2e-models-e2e-worker-w6-35a187/distinct-2-w6-35a187-kserve-7ff4b6dc55-944g6" Oct 31 00:01:13.307645 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:13.307616 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/429ee092-42d8-475e-b0c6-b5a9930766fc-tmp-dir\") pod \"distinct-2-w6-35a187-kserve-7ff4b6dc55-944g6\" (UID: \"429ee092-42d8-475e-b0c6-b5a9930766fc\") " pod="e2e-models-e2e-worker-w6-35a187/distinct-2-w6-35a187-kserve-7ff4b6dc55-944g6" Oct 31 00:01:13.307802 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:13.307788 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/429ee092-42d8-475e-b0c6-b5a9930766fc-model-cache\") pod \"distinct-2-w6-35a187-kserve-7ff4b6dc55-944g6\" (UID: \"429ee092-42d8-475e-b0c6-b5a9930766fc\") " pod="e2e-models-e2e-worker-w6-35a187/distinct-2-w6-35a187-kserve-7ff4b6dc55-944g6" Oct 31 00:01:13.307960 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:13.307948 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/429ee092-42d8-475e-b0c6-b5a9930766fc-tls-certs\") pod \"distinct-2-w6-35a187-kserve-7ff4b6dc55-944g6\" (UID: \"429ee092-42d8-475e-b0c6-b5a9930766fc\") " pod="e2e-models-e2e-worker-w6-35a187/distinct-2-w6-35a187-kserve-7ff4b6dc55-944g6" Oct 31 00:01:13.312567 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:13.312521 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/429ee092-42d8-475e-b0c6-b5a9930766fc-tmp-dir\") pod \"distinct-2-w6-35a187-kserve-7ff4b6dc55-944g6\" (UID: \"429ee092-42d8-475e-b0c6-b5a9930766fc\") " pod="e2e-models-e2e-worker-w6-35a187/distinct-2-w6-35a187-kserve-7ff4b6dc55-944g6" Oct 31 00:01:13.313693 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:13.313658 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/429ee092-42d8-475e-b0c6-b5a9930766fc-home\") pod \"distinct-2-w6-35a187-kserve-7ff4b6dc55-944g6\" (UID: \"429ee092-42d8-475e-b0c6-b5a9930766fc\") " pod="e2e-models-e2e-worker-w6-35a187/distinct-2-w6-35a187-kserve-7ff4b6dc55-944g6" Oct 31 00:01:13.313823 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:13.313704 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/429ee092-42d8-475e-b0c6-b5a9930766fc-dshm\") pod \"distinct-2-w6-35a187-kserve-7ff4b6dc55-944g6\" (UID: \"429ee092-42d8-475e-b0c6-b5a9930766fc\") " pod="e2e-models-e2e-worker-w6-35a187/distinct-2-w6-35a187-kserve-7ff4b6dc55-944g6" Oct 31 00:01:13.316038 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:13.316013 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/429ee092-42d8-475e-b0c6-b5a9930766fc-model-cache\") pod \"distinct-2-w6-35a187-kserve-7ff4b6dc55-944g6\" (UID: \"429ee092-42d8-475e-b0c6-b5a9930766fc\") " pod="e2e-models-e2e-worker-w6-35a187/distinct-2-w6-35a187-kserve-7ff4b6dc55-944g6" Oct 31 00:01:13.316255 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:13.316238 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/429ee092-42d8-475e-b0c6-b5a9930766fc-home\") pod \"distinct-2-w6-35a187-kserve-7ff4b6dc55-944g6\" (UID: \"429ee092-42d8-475e-b0c6-b5a9930766fc\") " pod="e2e-models-e2e-worker-w6-35a187/distinct-2-w6-35a187-kserve-7ff4b6dc55-944g6" Oct 31 00:01:13.328345 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:13.324218 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/429ee092-42d8-475e-b0c6-b5a9930766fc-dshm\") pod \"distinct-2-w6-35a187-kserve-7ff4b6dc55-944g6\" (UID: \"429ee092-42d8-475e-b0c6-b5a9930766fc\") " pod="e2e-models-e2e-worker-w6-35a187/distinct-2-w6-35a187-kserve-7ff4b6dc55-944g6" Oct 31 00:01:13.328345 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:13.326202 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/429ee092-42d8-475e-b0c6-b5a9930766fc-tls-certs\") pod \"distinct-2-w6-35a187-kserve-7ff4b6dc55-944g6\" (UID: \"429ee092-42d8-475e-b0c6-b5a9930766fc\") " pod="e2e-models-e2e-worker-w6-35a187/distinct-2-w6-35a187-kserve-7ff4b6dc55-944g6" Oct 31 00:01:13.376915 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:13.376836 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rkn2x\" (UniqueName: \"kubernetes.io/projected/429ee092-42d8-475e-b0c6-b5a9930766fc-kube-api-access-rkn2x\") pod \"distinct-2-w6-35a187-kserve-7ff4b6dc55-944g6\" (UID: \"429ee092-42d8-475e-b0c6-b5a9930766fc\") " pod="e2e-models-e2e-worker-w6-35a187/distinct-2-w6-35a187-kserve-7ff4b6dc55-944g6" Oct 31 00:01:13.405839 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:13.405804 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-35a187/distinct-2-w6-35a187-kserve-7ff4b6dc55-944g6" Oct 31 00:01:13.794939 ip-10-0-141-180 kubenswrapper[4927]: W1031 00:01:13.794906 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod429ee092_42d8_475e_b0c6_b5a9930766fc.slice/crio-c940816fcfa592cfa2a3b0fe41de09398a92f0c33cb06e601a0535d40dcd1dd1 WatchSource:0}: Error finding container c940816fcfa592cfa2a3b0fe41de09398a92f0c33cb06e601a0535d40dcd1dd1: Status 404 returned error can't find the container with id c940816fcfa592cfa2a3b0fe41de09398a92f0c33cb06e601a0535d40dcd1dd1 Oct 31 00:01:13.796122 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:13.796098 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-35a187/distinct-2-w6-35a187-kserve-7ff4b6dc55-944g6"] Oct 31 00:01:14.052971 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:14.052887 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d7c68c1c-gjckg"] Oct 31 00:01:14.057147 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:14.057125 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d7c68c1c-gjckg"] Oct 31 00:01:14.385860 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:14.385765 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-35a187/distinct-2-w6-35a187-kserve-7ff4b6dc55-944g6" event={"ID":"429ee092-42d8-475e-b0c6-b5a9930766fc","Type":"ContainerStarted","Data":"395cd9912d86ee5a63a52e819da9b8d1baf7ae77f4add38912ff14a53f428ec1"} Oct 31 00:01:14.385860 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:14.385813 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-35a187/distinct-2-w6-35a187-kserve-7ff4b6dc55-944g6" event={"ID":"429ee092-42d8-475e-b0c6-b5a9930766fc","Type":"ContainerStarted","Data":"c940816fcfa592cfa2a3b0fe41de09398a92f0c33cb06e601a0535d40dcd1dd1"} Oct 31 00:01:14.406146 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:14.406111 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-35a187/distinct-2-w6-35a187-kserve-7ff4b6dc55-944g6" Oct 31 00:01:14.509071 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:14.509014 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-35a187/distinct-2-w6-35a187-kserve-7ff4b6dc55-944g6" podStartSLOduration=1.508997841 podStartE2EDuration="1.508997841s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:01:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:01:14.505493053 +0000 UTC m=+1187.541823891" watchObservedRunningTime="2026-10-31 00:01:14.508997841 +0000 UTC m=+1187.545328701" Oct 31 00:01:15.088214 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:15.088173 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="298bd1f1-9653-4535-9e79-682eb391f6d6" path="/var/lib/kubelet/pods/298bd1f1-9653-4535-9e79-682eb391f6d6/volumes" Oct 31 00:01:15.870364 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:15.870321 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:01:18.545860 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:18.545823 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:01:19.678708 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:01:19.678662 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sqv69" containerName="extractor" Oct 31 00:01:20.589429 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:20.589393 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:01:23.406449 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:23.406419 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-35a187/distinct-2-w6-35a187-kserve-7ff4b6dc55-944g6" Oct 31 00:01:23.423531 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:23.423504 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-35a187/distinct-2-w6-35a187-kserve-7ff4b6dc55-944g6" Oct 31 00:01:23.534655 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:23.534621 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-35a187/distinct-2-w6-35a187-kserve-7ff4b6dc55-944g6" Oct 31 00:01:23.814787 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:23.814754 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:01:25.193279 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:25.193244 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:01:25.260315 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:25.260273 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b4d752ef-g2sz5"] Oct 31 00:01:25.295088 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:25.295057 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4d752ef-g2sz5" Oct 31 00:01:25.379828 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:25.379085 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:01:25.388748 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:25.388720 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b4d752ef-g2sz5"] Oct 31 00:01:25.430262 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:25.430230 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4fdzs\"" Oct 31 00:01:25.441126 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:25.441089 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhjb2\" (UniqueName: \"kubernetes.io/projected/bcbadb2f-3539-4d46-9100-2bae9ba06d14-kube-api-access-xhjb2\") pod \"maas-api-revoke-sub-b4d752ef-g2sz5\" (UID: \"bcbadb2f-3539-4d46-9100-2bae9ba06d14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4d752ef-g2sz5" Oct 31 00:01:25.441288 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:25.441208 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bcbadb2f-3539-4d46-9100-2bae9ba06d14-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b4d752ef-g2sz5\" (UID: \"bcbadb2f-3539-4d46-9100-2bae9ba06d14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4d752ef-g2sz5" Oct 31 00:01:25.441288 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:25.441259 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bcbadb2f-3539-4d46-9100-2bae9ba06d14-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b4d752ef-g2sz5\" (UID: \"bcbadb2f-3539-4d46-9100-2bae9ba06d14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4d752ef-g2sz5" Oct 31 00:01:25.553020 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:25.552923 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bcbadb2f-3539-4d46-9100-2bae9ba06d14-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b4d752ef-g2sz5\" (UID: \"bcbadb2f-3539-4d46-9100-2bae9ba06d14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4d752ef-g2sz5" Oct 31 00:01:25.553647 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:25.553617 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhjb2\" (UniqueName: \"kubernetes.io/projected/bcbadb2f-3539-4d46-9100-2bae9ba06d14-kube-api-access-xhjb2\") pod \"maas-api-revoke-sub-b4d752ef-g2sz5\" (UID: \"bcbadb2f-3539-4d46-9100-2bae9ba06d14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4d752ef-g2sz5" Oct 31 00:01:25.554077 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:25.554046 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bcbadb2f-3539-4d46-9100-2bae9ba06d14-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b4d752ef-g2sz5\" (UID: \"bcbadb2f-3539-4d46-9100-2bae9ba06d14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4d752ef-g2sz5" Oct 31 00:01:25.555635 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:25.555611 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bcbadb2f-3539-4d46-9100-2bae9ba06d14-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b4d752ef-g2sz5\" (UID: \"bcbadb2f-3539-4d46-9100-2bae9ba06d14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4d752ef-g2sz5" Oct 31 00:01:25.608844 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:25.608810 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bcbadb2f-3539-4d46-9100-2bae9ba06d14-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b4d752ef-g2sz5\" (UID: \"bcbadb2f-3539-4d46-9100-2bae9ba06d14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4d752ef-g2sz5" Oct 31 00:01:25.696367 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:25.695218 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhjb2\" (UniqueName: \"kubernetes.io/projected/bcbadb2f-3539-4d46-9100-2bae9ba06d14-kube-api-access-xhjb2\") pod \"maas-api-revoke-sub-b4d752ef-g2sz5\" (UID: \"bcbadb2f-3539-4d46-9100-2bae9ba06d14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4d752ef-g2sz5" Oct 31 00:01:25.696367 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:25.695485 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4d752ef-g2sz5" Oct 31 00:01:26.261375 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:26.245776 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2821202e-ql97w"] Oct 31 00:01:26.314364 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:26.313818 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2821202e-ql97w"] Oct 31 00:01:26.771055 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:26.754038 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b4d752ef-g2sz5"] Oct 31 00:01:27.600793 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:27.597503 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-986d37/premium-w3-986d37-kserve-5c7c7757fb-r99j7" podUID="f4e38df7-355f-4b47-a0b5-7e0b907f3326" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:01:27.735359 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:27.697694 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df44c490-7c92-4af4-908f-d4a01e2ab80c" path="/var/lib/kubelet/pods/df44c490-7c92-4af4-908f-d4a01e2ab80c/volumes" Oct 31 00:01:27.842412 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:27.829805 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4d752ef-g2sz5" event={"ID":"bcbadb2f-3539-4d46-9100-2bae9ba06d14","Type":"ContainerStarted","Data":"24151378351b9d041b33b1050925d646d547b81e1fa479e376e84f7ca6df394f"} Oct 31 00:01:27.842412 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:27.829875 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4d752ef-g2sz5" event={"ID":"bcbadb2f-3539-4d46-9100-2bae9ba06d14","Type":"ContainerStarted","Data":"971d2bf74f0fd40ac81fdba2d7fbd0e4ecac73e0e37eaec1c5f28b0b1931d055"} Oct 31 00:01:27.976367 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:27.975394 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4d752ef-g2sz5" podStartSLOduration=2.975371095 podStartE2EDuration="2.975371095s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:01:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:01:27.956987599 +0000 UTC m=+1200.993318454" watchObservedRunningTime="2026-10-31 00:01:27.975371095 +0000 UTC m=+1201.011701914" Oct 31 00:01:28.009389 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:28.000478 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-46b586/simulator-w0-46b586-kserve-67588b8dbd-z8xmn" podUID="9a25e252-223d-41f2-abf7-632067c77e0b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:01:28.077464 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:28.077414 4927 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-46b586/simulator-w0-46b586-kserve-67588b8dbd-z8xmn" podUID="9a25e252-223d-41f2-abf7-632067c77e0b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/health\": context deadline exceeded" Oct 31 00:01:28.155801 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:28.155713 4927 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-pqt2c_6382dfb5-a363-47a2-bc79-0b96f7bec0c3/console-operator/3.log" Oct 31 00:01:28.326354 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:28.286495 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5c3374/premium-w1-5c3374-kserve-6c56d4dcd-llwcl" podUID="dc813f02-e077-4251-8c1e-fea0986b0f36" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded" Oct 31 00:01:28.333073 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:28.333034 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fe01fa/distinct-2-w5-fe01fa-kserve-79b99966d8-dkhzd" podUID="d5c1c9e8-ce47-4868-860d-b8413ff74ab5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.111:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:01:28.523356 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:28.521261 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-35a187/distinct-2-w6-35a187-kserve-7ff4b6dc55-944g6" podUID="429ee092-42d8-475e-b0c6-b5a9930766fc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.113:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:01:28.600359 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:28.599577 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-986d37/premium-w3-986d37-kserve-5c7c7757fb-r99j7" podUID="f4e38df7-355f-4b47-a0b5-7e0b907f3326" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:01:28.600359 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:28.600087 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-986d37/premium-w3-986d37-kserve-5c7c7757fb-r99j7" Oct 31 00:01:28.640357 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:28.623741 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-27e82b/distinct-w2-27e82b-kserve-75f475fcd9-mccsz" podUID="4ba8ced4-e93f-46ac-87c5-3d0b7fadc02d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:01:28.736359 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:28.732041 4927 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-pqt2c_6382dfb5-a363-47a2-bc79-0b96f7bec0c3/console-operator/3.log" Oct 31 00:01:28.958360 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:28.952017 4927 generic.go:356] "Generic (PLEG): container finished" podID="bcbadb2f-3539-4d46-9100-2bae9ba06d14" containerID="24151378351b9d041b33b1050925d646d547b81e1fa479e376e84f7ca6df394f" exitCode=0 Oct 31 00:01:28.958360 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:28.952310 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4d752ef-g2sz5" event={"ID":"bcbadb2f-3539-4d46-9100-2bae9ba06d14","Type":"ContainerDied","Data":"24151378351b9d041b33b1050925d646d547b81e1fa479e376e84f7ca6df394f"} Oct 31 00:01:29.012365 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:29.005096 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-46b586/simulator-w0-46b586-kserve-67588b8dbd-z8xmn" podUID="9a25e252-223d-41f2-abf7-632067c77e0b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded" Oct 31 00:01:29.012365 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:29.005456 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-46b586/simulator-w0-46b586-kserve-67588b8dbd-z8xmn" Oct 31 00:01:29.031368 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:29.026600 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-46b586/simulator-w0-46b586-kserve-67588b8dbd-z8xmn" Oct 31 00:01:29.146528 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:29.145043 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-q7dtb"] Oct 31 00:01:29.151066 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:29.149953 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-986d37/premium-w3-986d37-kserve-5c7c7757fb-r99j7" Oct 31 00:01:29.151066 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:29.150767 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-q7dtb" Oct 31 00:01:29.234041 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:29.233957 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-q7dtb"] Oct 31 00:01:29.274133 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:29.274088 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f3dbfce-11a6-4176-9a52-c9772a349dbe-istio-envoy\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-q7dtb\" (UID: \"1f3dbfce-11a6-4176-9a52-c9772a349dbe\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-q7dtb" Oct 31 00:01:29.274319 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:29.274151 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-96n2w\" (UniqueName: \"kubernetes.io/projected/1f3dbfce-11a6-4176-9a52-c9772a349dbe-kube-api-access-96n2w\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-q7dtb\" (UID: \"1f3dbfce-11a6-4176-9a52-c9772a349dbe\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-q7dtb" Oct 31 00:01:29.274319 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:29.274261 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f3dbfce-11a6-4176-9a52-c9772a349dbe-credential-socket\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-q7dtb\" (UID: \"1f3dbfce-11a6-4176-9a52-c9772a349dbe\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-q7dtb" Oct 31 00:01:29.274319 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:29.274303 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f3dbfce-11a6-4176-9a52-c9772a349dbe-istiod-ca-cert\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-q7dtb\" (UID: \"1f3dbfce-11a6-4176-9a52-c9772a349dbe\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-q7dtb" Oct 31 00:01:29.274536 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:29.274511 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f3dbfce-11a6-4176-9a52-c9772a349dbe-workload-certs\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-q7dtb\" (UID: \"1f3dbfce-11a6-4176-9a52-c9772a349dbe\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-q7dtb" Oct 31 00:01:29.274592 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:29.274551 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f3dbfce-11a6-4176-9a52-c9772a349dbe-istio-podinfo\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-q7dtb\" (UID: \"1f3dbfce-11a6-4176-9a52-c9772a349dbe\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-q7dtb" Oct 31 00:01:29.274682 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:29.274666 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f3dbfce-11a6-4176-9a52-c9772a349dbe-istio-data\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-q7dtb\" (UID: \"1f3dbfce-11a6-4176-9a52-c9772a349dbe\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-q7dtb" Oct 31 00:01:29.274735 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:29.274703 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f3dbfce-11a6-4176-9a52-c9772a349dbe-workload-socket\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-q7dtb\" (UID: \"1f3dbfce-11a6-4176-9a52-c9772a349dbe\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-q7dtb" Oct 31 00:01:29.274871 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:29.274855 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f3dbfce-11a6-4176-9a52-c9772a349dbe-istio-token\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-q7dtb\" (UID: \"1f3dbfce-11a6-4176-9a52-c9772a349dbe\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-q7dtb" Oct 31 00:01:29.376600 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:29.376562 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f3dbfce-11a6-4176-9a52-c9772a349dbe-istio-token\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-q7dtb\" (UID: \"1f3dbfce-11a6-4176-9a52-c9772a349dbe\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-q7dtb" Oct 31 00:01:29.376787 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:29.376614 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f3dbfce-11a6-4176-9a52-c9772a349dbe-istio-envoy\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-q7dtb\" (UID: \"1f3dbfce-11a6-4176-9a52-c9772a349dbe\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-q7dtb" Oct 31 00:01:29.376787 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:29.376634 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96n2w\" (UniqueName: \"kubernetes.io/projected/1f3dbfce-11a6-4176-9a52-c9772a349dbe-kube-api-access-96n2w\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-q7dtb\" (UID: \"1f3dbfce-11a6-4176-9a52-c9772a349dbe\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-q7dtb" Oct 31 00:01:29.376787 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:29.376666 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f3dbfce-11a6-4176-9a52-c9772a349dbe-credential-socket\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-q7dtb\" (UID: \"1f3dbfce-11a6-4176-9a52-c9772a349dbe\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-q7dtb" Oct 31 00:01:29.376787 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:29.376686 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f3dbfce-11a6-4176-9a52-c9772a349dbe-istiod-ca-cert\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-q7dtb\" (UID: \"1f3dbfce-11a6-4176-9a52-c9772a349dbe\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-q7dtb" Oct 31 00:01:29.377080 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:29.376799 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f3dbfce-11a6-4176-9a52-c9772a349dbe-workload-certs\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-q7dtb\" (UID: \"1f3dbfce-11a6-4176-9a52-c9772a349dbe\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-q7dtb" Oct 31 00:01:29.377080 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:29.376823 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f3dbfce-11a6-4176-9a52-c9772a349dbe-istio-podinfo\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-q7dtb\" (UID: \"1f3dbfce-11a6-4176-9a52-c9772a349dbe\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-q7dtb" Oct 31 00:01:29.377080 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:29.376904 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f3dbfce-11a6-4176-9a52-c9772a349dbe-istio-data\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-q7dtb\" (UID: \"1f3dbfce-11a6-4176-9a52-c9772a349dbe\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-q7dtb" Oct 31 00:01:29.377080 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:29.376938 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f3dbfce-11a6-4176-9a52-c9772a349dbe-workload-socket\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-q7dtb\" (UID: \"1f3dbfce-11a6-4176-9a52-c9772a349dbe\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-q7dtb" Oct 31 00:01:29.377301 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:29.377157 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f3dbfce-11a6-4176-9a52-c9772a349dbe-credential-socket\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-q7dtb\" (UID: \"1f3dbfce-11a6-4176-9a52-c9772a349dbe\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-q7dtb" Oct 31 00:01:29.377401 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:29.377379 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f3dbfce-11a6-4176-9a52-c9772a349dbe-istio-data\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-q7dtb\" (UID: \"1f3dbfce-11a6-4176-9a52-c9772a349dbe\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-q7dtb" Oct 31 00:01:29.377469 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:29.377409 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f3dbfce-11a6-4176-9a52-c9772a349dbe-workload-socket\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-q7dtb\" (UID: \"1f3dbfce-11a6-4176-9a52-c9772a349dbe\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-q7dtb" Oct 31 00:01:29.377529 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:29.377489 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f3dbfce-11a6-4176-9a52-c9772a349dbe-workload-certs\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-q7dtb\" (UID: \"1f3dbfce-11a6-4176-9a52-c9772a349dbe\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-q7dtb" Oct 31 00:01:29.377667 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:29.377650 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f3dbfce-11a6-4176-9a52-c9772a349dbe-istiod-ca-cert\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-q7dtb\" (UID: \"1f3dbfce-11a6-4176-9a52-c9772a349dbe\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-q7dtb" Oct 31 00:01:29.379793 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:29.379771 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f3dbfce-11a6-4176-9a52-c9772a349dbe-istio-envoy\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-q7dtb\" (UID: \"1f3dbfce-11a6-4176-9a52-c9772a349dbe\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-q7dtb" Oct 31 00:01:29.380027 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:29.380007 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f3dbfce-11a6-4176-9a52-c9772a349dbe-istio-podinfo\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-q7dtb\" (UID: \"1f3dbfce-11a6-4176-9a52-c9772a349dbe\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-q7dtb" Oct 31 00:01:29.439423 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:29.439380 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f3dbfce-11a6-4176-9a52-c9772a349dbe-istio-token\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-q7dtb\" (UID: \"1f3dbfce-11a6-4176-9a52-c9772a349dbe\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-q7dtb" Oct 31 00:01:29.439659 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:29.439633 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-96n2w\" (UniqueName: \"kubernetes.io/projected/1f3dbfce-11a6-4176-9a52-c9772a349dbe-kube-api-access-96n2w\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-q7dtb\" (UID: \"1f3dbfce-11a6-4176-9a52-c9772a349dbe\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-q7dtb" Oct 31 00:01:29.481066 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:29.481033 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-q7dtb" Oct 31 00:01:29.906684 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:29.906600 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:01:29.992919 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:29.992143 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:01:29.992919 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:29.992233 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:01:29.992919 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:29.992277 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:01:30.006352 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:30.001270 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-q7dtb"] Oct 31 00:01:30.245367 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:30.245108 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4d752ef-g2sz5" Oct 31 00:01:30.398431 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:30.398394 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bcbadb2f-3539-4d46-9100-2bae9ba06d14-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bcbadb2f-3539-4d46-9100-2bae9ba06d14-maas-api-openshift-ca\") pod \"bcbadb2f-3539-4d46-9100-2bae9ba06d14\" (UID: \"bcbadb2f-3539-4d46-9100-2bae9ba06d14\") " Oct 31 00:01:30.402076 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:30.402047 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bcbadb2f-3539-4d46-9100-2bae9ba06d14-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bcbadb2f-3539-4d46-9100-2bae9ba06d14-maas-api-xks-ca\") pod \"bcbadb2f-3539-4d46-9100-2bae9ba06d14\" (UID: \"bcbadb2f-3539-4d46-9100-2bae9ba06d14\") " Oct 31 00:01:30.402189 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:30.402104 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bcbadb2f-3539-4d46-9100-2bae9ba06d14-kube-api-access-xhjb2\" (UniqueName: \"kubernetes.io/projected/bcbadb2f-3539-4d46-9100-2bae9ba06d14-kube-api-access-xhjb2\") pod \"bcbadb2f-3539-4d46-9100-2bae9ba06d14\" (UID: \"bcbadb2f-3539-4d46-9100-2bae9ba06d14\") " Oct 31 00:01:30.403893 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:30.403863 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bcbadb2f-3539-4d46-9100-2bae9ba06d14-maas-api-openshift-ca" pod "bcbadb2f-3539-4d46-9100-2bae9ba06d14" (UID: "bcbadb2f-3539-4d46-9100-2bae9ba06d14"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:01:30.407484 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:30.407442 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bcbadb2f-3539-4d46-9100-2bae9ba06d14-maas-api-xks-ca" pod "bcbadb2f-3539-4d46-9100-2bae9ba06d14" (UID: "bcbadb2f-3539-4d46-9100-2bae9ba06d14"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:01:30.407578 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:30.407475 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bcbadb2f-3539-4d46-9100-2bae9ba06d14-kube-api-access-xhjb2" pod "bcbadb2f-3539-4d46-9100-2bae9ba06d14" (UID: "bcbadb2f-3539-4d46-9100-2bae9ba06d14"). InnerVolumeSpecName "kube-api-access-xhjb2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:01:30.504937 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:30.504903 4927 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bcbadb2f-3539-4d46-9100-2bae9ba06d14-maas-api-xks-ca\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:01:30.504937 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:30.504937 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xhjb2\" (UniqueName: \"kubernetes.io/projected/bcbadb2f-3539-4d46-9100-2bae9ba06d14-kube-api-access-xhjb2\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:01:30.504937 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:30.504953 4927 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bcbadb2f-3539-4d46-9100-2bae9ba06d14-maas-api-openshift-ca\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:01:31.056803 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:31.056761 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4d752ef-g2sz5" event={"ID":"bcbadb2f-3539-4d46-9100-2bae9ba06d14","Type":"ContainerDied","Data":"971d2bf74f0fd40ac81fdba2d7fbd0e4ecac73e0e37eaec1c5f28b0b1931d055"} Oct 31 00:01:31.057491 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:31.056830 4927 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="971d2bf74f0fd40ac81fdba2d7fbd0e4ecac73e0e37eaec1c5f28b0b1931d055" Oct 31 00:01:31.057491 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:31.056901 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4d752ef-g2sz5" Oct 31 00:01:31.067809 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:31.067772 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-q7dtb" event={"ID":"1f3dbfce-11a6-4176-9a52-c9772a349dbe","Type":"ContainerStarted","Data":"1ddf6df601c51af42c951b2599fe6369fb3899aa2935bec193cde30e0f90c4e9"} Oct 31 00:01:31.067947 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:31.067844 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-q7dtb" event={"ID":"1f3dbfce-11a6-4176-9a52-c9772a349dbe","Type":"ContainerStarted","Data":"8de2217435c525660b5dd89c6a5de7c2d313b103b06c062b85c53b66cb69035e"} Oct 31 00:01:31.205311 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:31.205257 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-q7dtb" podStartSLOduration=2.205242265 podStartE2EDuration="2.205242265s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:01:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:01:31.200685431 +0000 UTC m=+1204.237016272" watchObservedRunningTime="2026-10-31 00:01:31.205242265 +0000 UTC m=+1204.241573135" Oct 31 00:01:31.482093 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:31.482054 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-q7dtb" Oct 31 00:01:31.483589 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:31.483563 4927 patch_prober.go:29] interesting pod/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-q7dtb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body= Oct 31 00:01:31.483698 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:31.483610 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-q7dtb" podUID="1f3dbfce-11a6-4176-9a52-c9772a349dbe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" Oct 31 00:01:32.481917 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:32.481872 4927 patch_prober.go:29] interesting pod/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-q7dtb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body= Oct 31 00:01:32.482362 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:32.481937 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-q7dtb" podUID="1f3dbfce-11a6-4176-9a52-c9772a349dbe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" Oct 31 00:01:32.669175 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:32.669134 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:01:32.823902 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:32.823808 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:01:33.481468 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:33.481427 4927 patch_prober.go:29] interesting pod/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-q7dtb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body= Oct 31 00:01:33.481660 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:33.481485 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-q7dtb" podUID="1f3dbfce-11a6-4176-9a52-c9772a349dbe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" Oct 31 00:01:34.482085 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:34.482048 4927 patch_prober.go:29] interesting pod/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-q7dtb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body= Oct 31 00:01:34.482519 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:34.482108 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-q7dtb" podUID="1f3dbfce-11a6-4176-9a52-c9772a349dbe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" Oct 31 00:01:34.504186 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:34.504153 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:01:35.054790 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:35.054753 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3f61fa33-cbsnw"] Oct 31 00:01:35.105521 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:35.105489 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3f61fa33-cbsnw"] Oct 31 00:01:35.481825 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:35.481791 4927 patch_prober.go:29] interesting pod/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-q7dtb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body= Oct 31 00:01:35.482015 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:35.481851 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-q7dtb" podUID="1f3dbfce-11a6-4176-9a52-c9772a349dbe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" Oct 31 00:01:35.964854 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:35.964815 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6014ecc2-lgxwz"] Oct 31 00:01:35.965946 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:35.965921 4927 cpu_manager.go:414] "containerMap: removing container" podUID="bcbadb2f-3539-4d46-9100-2bae9ba06d14" containerName="revoke-keys" Oct 31 00:01:35.965946 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:35.965946 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bcbadb2f-3539-4d46-9100-2bae9ba06d14" containerName="revoke-keys" Oct 31 00:01:35.966138 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:35.966122 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="bcbadb2f-3539-4d46-9100-2bae9ba06d14" containerName="revoke-keys" Oct 31 00:01:35.970398 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:35.970379 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6014ecc2-lgxwz" Oct 31 00:01:36.048222 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:36.048178 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8djm\" (UniqueName: \"kubernetes.io/projected/51c36120-3c7f-497c-84a2-e23674aae599-kube-api-access-n8djm\") pod \"maas-api-revoke-sub-6014ecc2-lgxwz\" (UID: \"51c36120-3c7f-497c-84a2-e23674aae599\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6014ecc2-lgxwz" Oct 31 00:01:36.048618 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:36.048593 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51c36120-3c7f-497c-84a2-e23674aae599-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6014ecc2-lgxwz\" (UID: \"51c36120-3c7f-497c-84a2-e23674aae599\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6014ecc2-lgxwz" Oct 31 00:01:36.048998 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:36.048974 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51c36120-3c7f-497c-84a2-e23674aae599-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6014ecc2-lgxwz\" (UID: \"51c36120-3c7f-497c-84a2-e23674aae599\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6014ecc2-lgxwz" Oct 31 00:01:36.055184 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:36.055052 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6014ecc2-lgxwz"] Oct 31 00:01:36.065865 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:36.065838 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:01:36.071238 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:36.071217 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4fdzs\"" Oct 31 00:01:36.153957 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:36.153914 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51c36120-3c7f-497c-84a2-e23674aae599-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6014ecc2-lgxwz\" (UID: \"51c36120-3c7f-497c-84a2-e23674aae599\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6014ecc2-lgxwz" Oct 31 00:01:36.154165 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:36.154004 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8djm\" (UniqueName: \"kubernetes.io/projected/51c36120-3c7f-497c-84a2-e23674aae599-kube-api-access-n8djm\") pod \"maas-api-revoke-sub-6014ecc2-lgxwz\" (UID: \"51c36120-3c7f-497c-84a2-e23674aae599\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6014ecc2-lgxwz" Oct 31 00:01:36.154165 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:36.154108 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51c36120-3c7f-497c-84a2-e23674aae599-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6014ecc2-lgxwz\" (UID: \"51c36120-3c7f-497c-84a2-e23674aae599\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6014ecc2-lgxwz" Oct 31 00:01:36.155223 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:36.155188 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51c36120-3c7f-497c-84a2-e23674aae599-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6014ecc2-lgxwz\" (UID: \"51c36120-3c7f-497c-84a2-e23674aae599\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6014ecc2-lgxwz" Oct 31 00:01:36.164963 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:36.164936 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51c36120-3c7f-497c-84a2-e23674aae599-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6014ecc2-lgxwz\" (UID: \"51c36120-3c7f-497c-84a2-e23674aae599\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6014ecc2-lgxwz" Oct 31 00:01:36.217918 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:36.217836 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8djm\" (UniqueName: \"kubernetes.io/projected/51c36120-3c7f-497c-84a2-e23674aae599-kube-api-access-n8djm\") pod \"maas-api-revoke-sub-6014ecc2-lgxwz\" (UID: \"51c36120-3c7f-497c-84a2-e23674aae599\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6014ecc2-lgxwz" Oct 31 00:01:36.284180 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:36.284146 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6014ecc2-lgxwz" Oct 31 00:01:36.481995 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:36.481898 4927 patch_prober.go:29] interesting pod/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-q7dtb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body= Oct 31 00:01:36.481995 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:36.481948 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-q7dtb" podUID="1f3dbfce-11a6-4176-9a52-c9772a349dbe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" Oct 31 00:01:36.685097 ip-10-0-141-180 kubenswrapper[4927]: W1031 00:01:36.685056 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51c36120_3c7f_497c_84a2_e23674aae599.slice/crio-3afb74836e1ad1a44e1c8be9e9c9fc6a08236e590d18b67bbfdeea14d3ff733d WatchSource:0}: Error finding container 3afb74836e1ad1a44e1c8be9e9c9fc6a08236e590d18b67bbfdeea14d3ff733d: Status 404 returned error can't find the container with id 3afb74836e1ad1a44e1c8be9e9c9fc6a08236e590d18b67bbfdeea14d3ff733d Oct 31 00:01:36.685623 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:36.685547 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6014ecc2-lgxwz"] Oct 31 00:01:37.087807 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:37.087774 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e1e0858-406a-4fea-91c4-acf5d505d0b9" path="/var/lib/kubelet/pods/7e1e0858-406a-4fea-91c4-acf5d505d0b9/volumes" Oct 31 00:01:37.177302 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:37.177220 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6014ecc2-lgxwz" event={"ID":"51c36120-3c7f-497c-84a2-e23674aae599","Type":"ContainerStarted","Data":"936c08a97e8b38e293cec6cdba70a9e7d73f7b9df6bdcf07b2dcadfffd793d42"} Oct 31 00:01:37.177302 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:37.177274 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6014ecc2-lgxwz" event={"ID":"51c36120-3c7f-497c-84a2-e23674aae599","Type":"ContainerStarted","Data":"3afb74836e1ad1a44e1c8be9e9c9fc6a08236e590d18b67bbfdeea14d3ff733d"} Oct 31 00:01:37.298630 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:37.298566 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6014ecc2-lgxwz" podStartSLOduration=2.298547518 podStartE2EDuration="2.298547518s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:01:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:01:37.29554178 +0000 UTC m=+1210.331872633" watchObservedRunningTime="2026-10-31 00:01:37.298547518 +0000 UTC m=+1210.334878343" Oct 31 00:01:37.550797 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:37.550687 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-q7dtb" Oct 31 00:01:37.551537 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:37.551505 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-q7dtb" Oct 31 00:01:37.552544 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:37.552523 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-q7dtb" Oct 31 00:01:38.186824 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:38.186792 4927 generic.go:356] "Generic (PLEG): container finished" podID="51c36120-3c7f-497c-84a2-e23674aae599" containerID="936c08a97e8b38e293cec6cdba70a9e7d73f7b9df6bdcf07b2dcadfffd793d42" exitCode=0 Oct 31 00:01:38.187242 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:38.186871 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6014ecc2-lgxwz" event={"ID":"51c36120-3c7f-497c-84a2-e23674aae599","Type":"ContainerDied","Data":"936c08a97e8b38e293cec6cdba70a9e7d73f7b9df6bdcf07b2dcadfffd793d42"} Oct 31 00:01:38.478995 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:38.478924 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ce935ca-nbf9s"] Oct 31 00:01:38.483498 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:38.483479 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ce935ca-nbf9s" Oct 31 00:01:38.543246 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:38.543211 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ce935ca-nbf9s"] Oct 31 00:01:38.590533 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:38.590493 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5aeb8b0-448b-4ead-aab0-5052033ec9f4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3ce935ca-nbf9s\" (UID: \"e5aeb8b0-448b-4ead-aab0-5052033ec9f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ce935ca-nbf9s" Oct 31 00:01:38.590706 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:38.590555 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5aeb8b0-448b-4ead-aab0-5052033ec9f4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3ce935ca-nbf9s\" (UID: \"e5aeb8b0-448b-4ead-aab0-5052033ec9f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ce935ca-nbf9s" Oct 31 00:01:38.590817 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:38.590728 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjnzj\" (UniqueName: \"kubernetes.io/projected/e5aeb8b0-448b-4ead-aab0-5052033ec9f4-kube-api-access-cjnzj\") pod \"maas-api-revoke-sub-3ce935ca-nbf9s\" (UID: \"e5aeb8b0-448b-4ead-aab0-5052033ec9f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ce935ca-nbf9s" Oct 31 00:01:38.692396 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:38.692360 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjnzj\" (UniqueName: \"kubernetes.io/projected/e5aeb8b0-448b-4ead-aab0-5052033ec9f4-kube-api-access-cjnzj\") pod \"maas-api-revoke-sub-3ce935ca-nbf9s\" (UID: \"e5aeb8b0-448b-4ead-aab0-5052033ec9f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ce935ca-nbf9s" Oct 31 00:01:38.692564 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:38.692489 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5aeb8b0-448b-4ead-aab0-5052033ec9f4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3ce935ca-nbf9s\" (UID: \"e5aeb8b0-448b-4ead-aab0-5052033ec9f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ce935ca-nbf9s" Oct 31 00:01:38.692564 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:38.692526 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5aeb8b0-448b-4ead-aab0-5052033ec9f4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3ce935ca-nbf9s\" (UID: \"e5aeb8b0-448b-4ead-aab0-5052033ec9f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ce935ca-nbf9s" Oct 31 00:01:38.693125 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:38.693094 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5aeb8b0-448b-4ead-aab0-5052033ec9f4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3ce935ca-nbf9s\" (UID: \"e5aeb8b0-448b-4ead-aab0-5052033ec9f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ce935ca-nbf9s" Oct 31 00:01:38.694877 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:38.694853 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5aeb8b0-448b-4ead-aab0-5052033ec9f4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3ce935ca-nbf9s\" (UID: \"e5aeb8b0-448b-4ead-aab0-5052033ec9f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ce935ca-nbf9s" Oct 31 00:01:38.754972 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:38.754898 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjnzj\" (UniqueName: \"kubernetes.io/projected/e5aeb8b0-448b-4ead-aab0-5052033ec9f4-kube-api-access-cjnzj\") pod \"maas-api-revoke-sub-3ce935ca-nbf9s\" (UID: \"e5aeb8b0-448b-4ead-aab0-5052033ec9f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ce935ca-nbf9s" Oct 31 00:01:38.797456 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:38.797417 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ce935ca-nbf9s" Oct 31 00:01:39.093806 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:39.093726 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:01:39.206266 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:39.205980 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ce935ca-nbf9s"] Oct 31 00:01:39.401579 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:39.401551 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6014ecc2-lgxwz" Oct 31 00:01:39.505221 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:39.505184 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/51c36120-3c7f-497c-84a2-e23674aae599-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51c36120-3c7f-497c-84a2-e23674aae599-maas-api-xks-ca\") pod \"51c36120-3c7f-497c-84a2-e23674aae599\" (UID: \"51c36120-3c7f-497c-84a2-e23674aae599\") " Oct 31 00:01:39.505466 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:39.505299 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/51c36120-3c7f-497c-84a2-e23674aae599-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51c36120-3c7f-497c-84a2-e23674aae599-maas-api-openshift-ca\") pod \"51c36120-3c7f-497c-84a2-e23674aae599\" (UID: \"51c36120-3c7f-497c-84a2-e23674aae599\") " Oct 31 00:01:39.505568 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:39.505548 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51c36120-3c7f-497c-84a2-e23674aae599-kube-api-access-n8djm\" (UniqueName: \"kubernetes.io/projected/51c36120-3c7f-497c-84a2-e23674aae599-kube-api-access-n8djm\") pod \"51c36120-3c7f-497c-84a2-e23674aae599\" (UID: \"51c36120-3c7f-497c-84a2-e23674aae599\") " Oct 31 00:01:39.505742 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:39.505716 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51c36120-3c7f-497c-84a2-e23674aae599-maas-api-openshift-ca" pod "51c36120-3c7f-497c-84a2-e23674aae599" (UID: "51c36120-3c7f-497c-84a2-e23674aae599"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:01:39.506072 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:39.506046 4927 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51c36120-3c7f-497c-84a2-e23674aae599-maas-api-openshift-ca\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:01:39.507514 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:39.507491 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/51c36120-3c7f-497c-84a2-e23674aae599-maas-api-xks-ca" pod "51c36120-3c7f-497c-84a2-e23674aae599" (UID: "51c36120-3c7f-497c-84a2-e23674aae599"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:01:39.507784 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:39.507763 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51c36120-3c7f-497c-84a2-e23674aae599-kube-api-access-n8djm" pod "51c36120-3c7f-497c-84a2-e23674aae599" (UID: "51c36120-3c7f-497c-84a2-e23674aae599"). InnerVolumeSpecName "kube-api-access-n8djm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:01:39.607810 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:39.607767 4927 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51c36120-3c7f-497c-84a2-e23674aae599-maas-api-xks-ca\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:01:39.607810 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:39.607802 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n8djm\" (UniqueName: \"kubernetes.io/projected/51c36120-3c7f-497c-84a2-e23674aae599-kube-api-access-n8djm\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:01:40.055189 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:40.055098 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-29043c97-pmqhh"] Oct 31 00:01:40.059657 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:40.059625 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-29043c97-pmqhh"] Oct 31 00:01:40.259653 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:40.259627 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6014ecc2-lgxwz" Oct 31 00:01:40.260026 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:40.259771 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6014ecc2-lgxwz" event={"ID":"51c36120-3c7f-497c-84a2-e23674aae599","Type":"ContainerDied","Data":"3afb74836e1ad1a44e1c8be9e9c9fc6a08236e590d18b67bbfdeea14d3ff733d"} Oct 31 00:01:40.260026 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:40.259812 4927 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3afb74836e1ad1a44e1c8be9e9c9fc6a08236e590d18b67bbfdeea14d3ff733d" Oct 31 00:01:40.261460 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:40.261433 4927 generic.go:356] "Generic (PLEG): container finished" podID="e5aeb8b0-448b-4ead-aab0-5052033ec9f4" containerID="8f5773172dce3bbbae288a3ee03ed7287c9f9cf1baa707197487a2fdcf542e2a" exitCode=0 Oct 31 00:01:40.261576 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:40.261469 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ce935ca-nbf9s" event={"ID":"e5aeb8b0-448b-4ead-aab0-5052033ec9f4","Type":"ContainerDied","Data":"8f5773172dce3bbbae288a3ee03ed7287c9f9cf1baa707197487a2fdcf542e2a"} Oct 31 00:01:40.261576 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:40.261529 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ce935ca-nbf9s" event={"ID":"e5aeb8b0-448b-4ead-aab0-5052033ec9f4","Type":"ContainerStarted","Data":"47a1eefce141708543c6cb45cd6df4d7175905b56046f1500c6b6274444ecee5"} Oct 31 00:01:41.087680 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:41.087647 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba94b314-c7ea-4a07-9084-98f7a36edc1a" path="/var/lib/kubelet/pods/ba94b314-c7ea-4a07-9084-98f7a36edc1a/volumes" Oct 31 00:01:41.446275 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:41.446241 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ce935ca-nbf9s" Oct 31 00:01:41.634127 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:41.634097 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5aeb8b0-448b-4ead-aab0-5052033ec9f4-kube-api-access-cjnzj\" (UniqueName: \"kubernetes.io/projected/e5aeb8b0-448b-4ead-aab0-5052033ec9f4-kube-api-access-cjnzj\") pod \"e5aeb8b0-448b-4ead-aab0-5052033ec9f4\" (UID: \"e5aeb8b0-448b-4ead-aab0-5052033ec9f4\") " Oct 31 00:01:41.634286 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:41.634257 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e5aeb8b0-448b-4ead-aab0-5052033ec9f4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5aeb8b0-448b-4ead-aab0-5052033ec9f4-maas-api-openshift-ca\") pod \"e5aeb8b0-448b-4ead-aab0-5052033ec9f4\" (UID: \"e5aeb8b0-448b-4ead-aab0-5052033ec9f4\") " Oct 31 00:01:41.634416 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:41.634395 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e5aeb8b0-448b-4ead-aab0-5052033ec9f4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5aeb8b0-448b-4ead-aab0-5052033ec9f4-maas-api-xks-ca\") pod \"e5aeb8b0-448b-4ead-aab0-5052033ec9f4\" (UID: \"e5aeb8b0-448b-4ead-aab0-5052033ec9f4\") " Oct 31 00:01:41.634702 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:41.634671 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5aeb8b0-448b-4ead-aab0-5052033ec9f4-maas-api-openshift-ca" pod "e5aeb8b0-448b-4ead-aab0-5052033ec9f4" (UID: "e5aeb8b0-448b-4ead-aab0-5052033ec9f4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:01:41.634983 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:41.634966 4927 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5aeb8b0-448b-4ead-aab0-5052033ec9f4-maas-api-openshift-ca\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:01:41.636314 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:41.636292 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5aeb8b0-448b-4ead-aab0-5052033ec9f4-kube-api-access-cjnzj" pod "e5aeb8b0-448b-4ead-aab0-5052033ec9f4" (UID: "e5aeb8b0-448b-4ead-aab0-5052033ec9f4"). InnerVolumeSpecName "kube-api-access-cjnzj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:01:41.636435 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:41.636311 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e5aeb8b0-448b-4ead-aab0-5052033ec9f4-maas-api-xks-ca" pod "e5aeb8b0-448b-4ead-aab0-5052033ec9f4" (UID: "e5aeb8b0-448b-4ead-aab0-5052033ec9f4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:01:41.736934 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:41.736845 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cjnzj\" (UniqueName: \"kubernetes.io/projected/e5aeb8b0-448b-4ead-aab0-5052033ec9f4-kube-api-access-cjnzj\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:01:41.736934 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:41.736882 4927 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5aeb8b0-448b-4ead-aab0-5052033ec9f4-maas-api-xks-ca\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:01:42.280583 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:42.280540 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ce935ca-nbf9s" event={"ID":"e5aeb8b0-448b-4ead-aab0-5052033ec9f4","Type":"ContainerDied","Data":"47a1eefce141708543c6cb45cd6df4d7175905b56046f1500c6b6274444ecee5"} Oct 31 00:01:42.280583 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:42.280567 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ce935ca-nbf9s" Oct 31 00:01:42.280829 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:42.280597 4927 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="47a1eefce141708543c6cb45cd6df4d7175905b56046f1500c6b6274444ecee5" Oct 31 00:01:43.676408 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:43.676373 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:01:44.584230 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:44.584197 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e9419c8d-xp797"] Oct 31 00:01:44.584869 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:44.584854 4927 cpu_manager.go:414] "containerMap: removing container" podUID="e5aeb8b0-448b-4ead-aab0-5052033ec9f4" containerName="revoke-keys" Oct 31 00:01:44.584914 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:44.584871 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5aeb8b0-448b-4ead-aab0-5052033ec9f4" containerName="revoke-keys" Oct 31 00:01:44.584914 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:44.584882 4927 cpu_manager.go:414] "containerMap: removing container" podUID="51c36120-3c7f-497c-84a2-e23674aae599" containerName="revoke-keys" Oct 31 00:01:44.584914 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:44.584887 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51c36120-3c7f-497c-84a2-e23674aae599" containerName="revoke-keys" Oct 31 00:01:44.585030 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:44.585020 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5aeb8b0-448b-4ead-aab0-5052033ec9f4" containerName="revoke-keys" Oct 31 00:01:44.585064 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:44.585032 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="51c36120-3c7f-497c-84a2-e23674aae599" containerName="revoke-keys" Oct 31 00:01:44.588631 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:44.588615 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9419c8d-xp797" Oct 31 00:01:44.647289 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:44.647252 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e9419c8d-xp797"] Oct 31 00:01:44.675049 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:44.675017 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa9d6354-c4e7-40fd-b163-103d75f0e34e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e9419c8d-xp797\" (UID: \"fa9d6354-c4e7-40fd-b163-103d75f0e34e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9419c8d-xp797" Oct 31 00:01:44.675217 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:44.675185 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa9d6354-c4e7-40fd-b163-103d75f0e34e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e9419c8d-xp797\" (UID: \"fa9d6354-c4e7-40fd-b163-103d75f0e34e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9419c8d-xp797" Oct 31 00:01:44.675422 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:44.675399 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ntttc\" (UniqueName: \"kubernetes.io/projected/fa9d6354-c4e7-40fd-b163-103d75f0e34e-kube-api-access-ntttc\") pod \"maas-api-revoke-sub-e9419c8d-xp797\" (UID: \"fa9d6354-c4e7-40fd-b163-103d75f0e34e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9419c8d-xp797" Oct 31 00:01:44.692132 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:44.692102 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:01:44.692589 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:44.692188 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4fdzs\"" Oct 31 00:01:44.777028 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:44.776990 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ntttc\" (UniqueName: \"kubernetes.io/projected/fa9d6354-c4e7-40fd-b163-103d75f0e34e-kube-api-access-ntttc\") pod \"maas-api-revoke-sub-e9419c8d-xp797\" (UID: \"fa9d6354-c4e7-40fd-b163-103d75f0e34e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9419c8d-xp797" Oct 31 00:01:44.777243 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:44.777077 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa9d6354-c4e7-40fd-b163-103d75f0e34e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e9419c8d-xp797\" (UID: \"fa9d6354-c4e7-40fd-b163-103d75f0e34e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9419c8d-xp797" Oct 31 00:01:44.777243 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:44.777150 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa9d6354-c4e7-40fd-b163-103d75f0e34e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e9419c8d-xp797\" (UID: \"fa9d6354-c4e7-40fd-b163-103d75f0e34e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9419c8d-xp797" Oct 31 00:01:44.777767 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:44.777736 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa9d6354-c4e7-40fd-b163-103d75f0e34e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e9419c8d-xp797\" (UID: \"fa9d6354-c4e7-40fd-b163-103d75f0e34e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9419c8d-xp797" Oct 31 00:01:44.779621 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:44.779587 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa9d6354-c4e7-40fd-b163-103d75f0e34e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e9419c8d-xp797\" (UID: \"fa9d6354-c4e7-40fd-b163-103d75f0e34e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9419c8d-xp797" Oct 31 00:01:44.839773 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:44.839701 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ntttc\" (UniqueName: \"kubernetes.io/projected/fa9d6354-c4e7-40fd-b163-103d75f0e34e-kube-api-access-ntttc\") pod \"maas-api-revoke-sub-e9419c8d-xp797\" (UID: \"fa9d6354-c4e7-40fd-b163-103d75f0e34e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9419c8d-xp797" Oct 31 00:01:44.906886 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:44.906852 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9419c8d-xp797" Oct 31 00:01:45.056716 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:45.056682 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-719bfdf0-t6rgr"] Oct 31 00:01:45.063661 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:45.063634 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-719bfdf0-t6rgr"] Oct 31 00:01:45.086813 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:45.086782 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1385f845-a8bf-46a4-8c44-1974c26c0c65" path="/var/lib/kubelet/pods/1385f845-a8bf-46a4-8c44-1974c26c0c65/volumes" Oct 31 00:01:45.290459 ip-10-0-141-180 kubenswrapper[4927]: W1031 00:01:45.290432 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa9d6354_c4e7_40fd_b163_103d75f0e34e.slice/crio-84c8dc0afe1b0757ed2cec3e019dacf03fb19f53ad90b6cfdc574b65fafcd54b WatchSource:0}: Error finding container 84c8dc0afe1b0757ed2cec3e019dacf03fb19f53ad90b6cfdc574b65fafcd54b: Status 404 returned error can't find the container with id 84c8dc0afe1b0757ed2cec3e019dacf03fb19f53ad90b6cfdc574b65fafcd54b Oct 31 00:01:45.292193 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:45.292149 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e9419c8d-xp797"] Oct 31 00:01:45.307171 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:45.307141 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9419c8d-xp797" event={"ID":"fa9d6354-c4e7-40fd-b163-103d75f0e34e","Type":"ContainerStarted","Data":"84c8dc0afe1b0757ed2cec3e019dacf03fb19f53ad90b6cfdc574b65fafcd54b"} Oct 31 00:01:45.479486 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:45.479450 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-j2j46"] Oct 31 00:01:45.485058 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:45.485039 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-j2j46" Oct 31 00:01:45.584828 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:45.584748 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-j2j46"] Oct 31 00:01:45.588166 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:45.588126 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jtdzk\" (UniqueName: \"kubernetes.io/projected/a0f800b8-0d6f-484a-bfa8-e71b15ce18e5-kube-api-access-jtdzk\") pod \"e2e-worker-w1-5c3374-openshift-default-d6ff699c7-j2j46\" (UID: \"a0f800b8-0d6f-484a-bfa8-e71b15ce18e5\") " pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-j2j46" Oct 31 00:01:45.588322 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:45.588189 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0f800b8-0d6f-484a-bfa8-e71b15ce18e5-credential-socket\") pod \"e2e-worker-w1-5c3374-openshift-default-d6ff699c7-j2j46\" (UID: \"a0f800b8-0d6f-484a-bfa8-e71b15ce18e5\") " pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-j2j46" Oct 31 00:01:45.588322 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:45.588259 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0f800b8-0d6f-484a-bfa8-e71b15ce18e5-istio-token\") pod \"e2e-worker-w1-5c3374-openshift-default-d6ff699c7-j2j46\" (UID: \"a0f800b8-0d6f-484a-bfa8-e71b15ce18e5\") " pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-j2j46" Oct 31 00:01:45.588461 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:45.588383 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0f800b8-0d6f-484a-bfa8-e71b15ce18e5-workload-socket\") pod \"e2e-worker-w1-5c3374-openshift-default-d6ff699c7-j2j46\" (UID: \"a0f800b8-0d6f-484a-bfa8-e71b15ce18e5\") " pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-j2j46" Oct 31 00:01:45.588461 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:45.588436 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0f800b8-0d6f-484a-bfa8-e71b15ce18e5-istio-podinfo\") pod \"e2e-worker-w1-5c3374-openshift-default-d6ff699c7-j2j46\" (UID: \"a0f800b8-0d6f-484a-bfa8-e71b15ce18e5\") " pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-j2j46" Oct 31 00:01:45.588599 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:45.588491 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0f800b8-0d6f-484a-bfa8-e71b15ce18e5-workload-certs\") pod \"e2e-worker-w1-5c3374-openshift-default-d6ff699c7-j2j46\" (UID: \"a0f800b8-0d6f-484a-bfa8-e71b15ce18e5\") " pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-j2j46" Oct 31 00:01:45.588599 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:45.588516 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0f800b8-0d6f-484a-bfa8-e71b15ce18e5-istiod-ca-cert\") pod \"e2e-worker-w1-5c3374-openshift-default-d6ff699c7-j2j46\" (UID: \"a0f800b8-0d6f-484a-bfa8-e71b15ce18e5\") " pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-j2j46" Oct 31 00:01:45.588599 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:45.588565 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0f800b8-0d6f-484a-bfa8-e71b15ce18e5-istio-data\") pod \"e2e-worker-w1-5c3374-openshift-default-d6ff699c7-j2j46\" (UID: \"a0f800b8-0d6f-484a-bfa8-e71b15ce18e5\") " pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-j2j46" Oct 31 00:01:45.588782 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:45.588754 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0f800b8-0d6f-484a-bfa8-e71b15ce18e5-istio-envoy\") pod \"e2e-worker-w1-5c3374-openshift-default-d6ff699c7-j2j46\" (UID: \"a0f800b8-0d6f-484a-bfa8-e71b15ce18e5\") " pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-j2j46" Oct 31 00:01:45.690003 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:45.689967 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0f800b8-0d6f-484a-bfa8-e71b15ce18e5-istio-envoy\") pod \"e2e-worker-w1-5c3374-openshift-default-d6ff699c7-j2j46\" (UID: \"a0f800b8-0d6f-484a-bfa8-e71b15ce18e5\") " pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-j2j46" Oct 31 00:01:45.690199 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:45.690132 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jtdzk\" (UniqueName: \"kubernetes.io/projected/a0f800b8-0d6f-484a-bfa8-e71b15ce18e5-kube-api-access-jtdzk\") pod \"e2e-worker-w1-5c3374-openshift-default-d6ff699c7-j2j46\" (UID: \"a0f800b8-0d6f-484a-bfa8-e71b15ce18e5\") " pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-j2j46" Oct 31 00:01:45.690199 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:45.690189 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0f800b8-0d6f-484a-bfa8-e71b15ce18e5-credential-socket\") pod \"e2e-worker-w1-5c3374-openshift-default-d6ff699c7-j2j46\" (UID: \"a0f800b8-0d6f-484a-bfa8-e71b15ce18e5\") " pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-j2j46" Oct 31 00:01:45.690368 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:45.690264 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0f800b8-0d6f-484a-bfa8-e71b15ce18e5-istio-token\") pod \"e2e-worker-w1-5c3374-openshift-default-d6ff699c7-j2j46\" (UID: \"a0f800b8-0d6f-484a-bfa8-e71b15ce18e5\") " pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-j2j46" Oct 31 00:01:45.690452 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:45.690430 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0f800b8-0d6f-484a-bfa8-e71b15ce18e5-workload-socket\") pod \"e2e-worker-w1-5c3374-openshift-default-d6ff699c7-j2j46\" (UID: \"a0f800b8-0d6f-484a-bfa8-e71b15ce18e5\") " pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-j2j46" Oct 31 00:01:45.690525 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:45.690496 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0f800b8-0d6f-484a-bfa8-e71b15ce18e5-istio-podinfo\") pod \"e2e-worker-w1-5c3374-openshift-default-d6ff699c7-j2j46\" (UID: \"a0f800b8-0d6f-484a-bfa8-e71b15ce18e5\") " pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-j2j46" Oct 31 00:01:45.690576 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:45.690556 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0f800b8-0d6f-484a-bfa8-e71b15ce18e5-workload-certs\") pod \"e2e-worker-w1-5c3374-openshift-default-d6ff699c7-j2j46\" (UID: \"a0f800b8-0d6f-484a-bfa8-e71b15ce18e5\") " pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-j2j46" Oct 31 00:01:45.690641 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:45.690581 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0f800b8-0d6f-484a-bfa8-e71b15ce18e5-istiod-ca-cert\") pod \"e2e-worker-w1-5c3374-openshift-default-d6ff699c7-j2j46\" (UID: \"a0f800b8-0d6f-484a-bfa8-e71b15ce18e5\") " pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-j2j46" Oct 31 00:01:45.690641 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:45.690616 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0f800b8-0d6f-484a-bfa8-e71b15ce18e5-istio-data\") pod \"e2e-worker-w1-5c3374-openshift-default-d6ff699c7-j2j46\" (UID: \"a0f800b8-0d6f-484a-bfa8-e71b15ce18e5\") " pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-j2j46" Oct 31 00:01:45.690743 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:45.690647 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0f800b8-0d6f-484a-bfa8-e71b15ce18e5-credential-socket\") pod \"e2e-worker-w1-5c3374-openshift-default-d6ff699c7-j2j46\" (UID: \"a0f800b8-0d6f-484a-bfa8-e71b15ce18e5\") " pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-j2j46" Oct 31 00:01:45.690927 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:45.690861 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0f800b8-0d6f-484a-bfa8-e71b15ce18e5-workload-socket\") pod \"e2e-worker-w1-5c3374-openshift-default-d6ff699c7-j2j46\" (UID: \"a0f800b8-0d6f-484a-bfa8-e71b15ce18e5\") " pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-j2j46" Oct 31 00:01:45.691028 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:45.690963 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0f800b8-0d6f-484a-bfa8-e71b15ce18e5-workload-certs\") pod \"e2e-worker-w1-5c3374-openshift-default-d6ff699c7-j2j46\" (UID: \"a0f800b8-0d6f-484a-bfa8-e71b15ce18e5\") " pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-j2j46" Oct 31 00:01:45.691076 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:45.691031 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0f800b8-0d6f-484a-bfa8-e71b15ce18e5-istio-data\") pod \"e2e-worker-w1-5c3374-openshift-default-d6ff699c7-j2j46\" (UID: \"a0f800b8-0d6f-484a-bfa8-e71b15ce18e5\") " pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-j2j46" Oct 31 00:01:45.691389 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:45.691371 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0f800b8-0d6f-484a-bfa8-e71b15ce18e5-istiod-ca-cert\") pod \"e2e-worker-w1-5c3374-openshift-default-d6ff699c7-j2j46\" (UID: \"a0f800b8-0d6f-484a-bfa8-e71b15ce18e5\") " pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-j2j46" Oct 31 00:01:45.692453 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:45.692437 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0f800b8-0d6f-484a-bfa8-e71b15ce18e5-istio-envoy\") pod \"e2e-worker-w1-5c3374-openshift-default-d6ff699c7-j2j46\" (UID: \"a0f800b8-0d6f-484a-bfa8-e71b15ce18e5\") " pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-j2j46" Oct 31 00:01:45.692975 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:45.692953 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0f800b8-0d6f-484a-bfa8-e71b15ce18e5-istio-podinfo\") pod \"e2e-worker-w1-5c3374-openshift-default-d6ff699c7-j2j46\" (UID: \"a0f800b8-0d6f-484a-bfa8-e71b15ce18e5\") " pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-j2j46" Oct 31 00:01:45.753030 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:45.752998 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0f800b8-0d6f-484a-bfa8-e71b15ce18e5-istio-token\") pod \"e2e-worker-w1-5c3374-openshift-default-d6ff699c7-j2j46\" (UID: \"a0f800b8-0d6f-484a-bfa8-e71b15ce18e5\") " pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-j2j46" Oct 31 00:01:45.753327 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:45.753307 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jtdzk\" (UniqueName: \"kubernetes.io/projected/a0f800b8-0d6f-484a-bfa8-e71b15ce18e5-kube-api-access-jtdzk\") pod \"e2e-worker-w1-5c3374-openshift-default-d6ff699c7-j2j46\" (UID: \"a0f800b8-0d6f-484a-bfa8-e71b15ce18e5\") " pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-j2j46" Oct 31 00:01:45.803438 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:45.803401 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-j2j46" Oct 31 00:01:46.245985 ip-10-0-141-180 kubenswrapper[4927]: W1031 00:01:46.245946 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0f800b8_0d6f_484a_bfa8_e71b15ce18e5.slice/crio-30a7626bc54dd27455f33b97c63d4bc335b342a47367a0999567799083941a75 WatchSource:0}: Error finding container 30a7626bc54dd27455f33b97c63d4bc335b342a47367a0999567799083941a75: Status 404 returned error can't find the container with id 30a7626bc54dd27455f33b97c63d4bc335b342a47367a0999567799083941a75 Oct 31 00:01:46.247700 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:46.247675 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-j2j46"] Oct 31 00:01:46.249326 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:46.249275 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:01:46.249447 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:46.249382 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:01:46.249447 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:46.249428 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:01:46.321480 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:46.321443 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9419c8d-xp797" event={"ID":"fa9d6354-c4e7-40fd-b163-103d75f0e34e","Type":"ContainerStarted","Data":"b6e2f488c4668340ae0be6ecf66735b3596aa898b457aa664c61c51296804f77"} Oct 31 00:01:46.324230 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:46.324201 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-j2j46" event={"ID":"a0f800b8-0d6f-484a-bfa8-e71b15ce18e5","Type":"ContainerStarted","Data":"30a7626bc54dd27455f33b97c63d4bc335b342a47367a0999567799083941a75"} Oct 31 00:01:46.451917 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:46.451863 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9419c8d-xp797" podStartSLOduration=2.45184719 podStartE2EDuration="2.45184719s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:01:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:01:46.448177732 +0000 UTC m=+1219.484508565" watchObservedRunningTime="2026-10-31 00:01:46.45184719 +0000 UTC m=+1219.488178011" Oct 31 00:01:47.345295 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:47.345262 4927 generic.go:356] "Generic (PLEG): container finished" podID="fa9d6354-c4e7-40fd-b163-103d75f0e34e" containerID="b6e2f488c4668340ae0be6ecf66735b3596aa898b457aa664c61c51296804f77" exitCode=0 Oct 31 00:01:47.345778 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:47.345378 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9419c8d-xp797" event={"ID":"fa9d6354-c4e7-40fd-b163-103d75f0e34e","Type":"ContainerDied","Data":"b6e2f488c4668340ae0be6ecf66735b3596aa898b457aa664c61c51296804f77"} Oct 31 00:01:47.347285 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:47.347259 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-j2j46" event={"ID":"a0f800b8-0d6f-484a-bfa8-e71b15ce18e5","Type":"ContainerStarted","Data":"5829cc9dd4c653da767410b7e21bd6de3da2f4239d4b4bc49bc4b779f2871f79"} Oct 31 00:01:47.597452 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:47.597343 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-j2j46" podStartSLOduration=2.597312321 podStartE2EDuration="2.597312321s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:01:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:01:47.592816213 +0000 UTC m=+1220.629147059" watchObservedRunningTime="2026-10-31 00:01:47.597312321 +0000 UTC m=+1220.633643150" Oct 31 00:01:47.804452 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:47.804416 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-j2j46" Oct 31 00:01:47.805980 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:47.805954 4927 patch_prober.go:29] interesting pod/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-j2j46 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Oct 31 00:01:47.806115 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:47.806006 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-j2j46" podUID="a0f800b8-0d6f-484a-bfa8-e71b15ce18e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Oct 31 00:01:48.471187 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:48.471150 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:01:48.747035 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:48.747004 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9419c8d-xp797" Oct 31 00:01:48.756032 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:48.756001 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fa9d6354-c4e7-40fd-b163-103d75f0e34e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa9d6354-c4e7-40fd-b163-103d75f0e34e-maas-api-xks-ca\") pod \"fa9d6354-c4e7-40fd-b163-103d75f0e34e\" (UID: \"fa9d6354-c4e7-40fd-b163-103d75f0e34e\") " Oct 31 00:01:48.756229 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:48.756211 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa9d6354-c4e7-40fd-b163-103d75f0e34e-kube-api-access-ntttc\" (UniqueName: \"kubernetes.io/projected/fa9d6354-c4e7-40fd-b163-103d75f0e34e-kube-api-access-ntttc\") pod \"fa9d6354-c4e7-40fd-b163-103d75f0e34e\" (UID: \"fa9d6354-c4e7-40fd-b163-103d75f0e34e\") " Oct 31 00:01:48.756323 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:48.756309 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa9d6354-c4e7-40fd-b163-103d75f0e34e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa9d6354-c4e7-40fd-b163-103d75f0e34e-maas-api-openshift-ca\") pod \"fa9d6354-c4e7-40fd-b163-103d75f0e34e\" (UID: \"fa9d6354-c4e7-40fd-b163-103d75f0e34e\") " Oct 31 00:01:48.756673 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:48.756647 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa9d6354-c4e7-40fd-b163-103d75f0e34e-maas-api-openshift-ca" pod "fa9d6354-c4e7-40fd-b163-103d75f0e34e" (UID: "fa9d6354-c4e7-40fd-b163-103d75f0e34e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:01:48.756856 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:48.756841 4927 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa9d6354-c4e7-40fd-b163-103d75f0e34e-maas-api-openshift-ca\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:01:48.758087 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:48.758066 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa9d6354-c4e7-40fd-b163-103d75f0e34e-maas-api-xks-ca" pod "fa9d6354-c4e7-40fd-b163-103d75f0e34e" (UID: "fa9d6354-c4e7-40fd-b163-103d75f0e34e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:01:48.758239 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:48.758217 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa9d6354-c4e7-40fd-b163-103d75f0e34e-kube-api-access-ntttc" pod "fa9d6354-c4e7-40fd-b163-103d75f0e34e" (UID: "fa9d6354-c4e7-40fd-b163-103d75f0e34e"). InnerVolumeSpecName "kube-api-access-ntttc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:01:48.804603 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:48.804570 4927 patch_prober.go:29] interesting pod/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-j2j46 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Oct 31 00:01:48.804756 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:48.804626 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-j2j46" podUID="a0f800b8-0d6f-484a-bfa8-e71b15ce18e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Oct 31 00:01:48.858503 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:48.858469 4927 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa9d6354-c4e7-40fd-b163-103d75f0e34e-maas-api-xks-ca\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:01:48.858503 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:48.858497 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ntttc\" (UniqueName: \"kubernetes.io/projected/fa9d6354-c4e7-40fd-b163-103d75f0e34e-kube-api-access-ntttc\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:01:49.364229 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:49.364139 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9419c8d-xp797" Oct 31 00:01:49.364229 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:49.364159 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9419c8d-xp797" event={"ID":"fa9d6354-c4e7-40fd-b163-103d75f0e34e","Type":"ContainerDied","Data":"84c8dc0afe1b0757ed2cec3e019dacf03fb19f53ad90b6cfdc574b65fafcd54b"} Oct 31 00:01:49.364229 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:49.364215 4927 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="84c8dc0afe1b0757ed2cec3e019dacf03fb19f53ad90b6cfdc574b65fafcd54b" Oct 31 00:01:49.678807 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:01:49.678773 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sqv69" containerName="extractor" Oct 31 00:01:49.804617 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:49.804578 4927 patch_prober.go:29] interesting pod/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-j2j46 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Oct 31 00:01:49.804765 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:49.804640 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-j2j46" podUID="a0f800b8-0d6f-484a-bfa8-e71b15ce18e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Oct 31 00:01:50.804372 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:50.804325 4927 patch_prober.go:29] interesting pod/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-j2j46 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Oct 31 00:01:50.804748 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:50.804389 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-j2j46" podUID="a0f800b8-0d6f-484a-bfa8-e71b15ce18e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Oct 31 00:01:51.804195 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:51.804153 4927 patch_prober.go:29] interesting pod/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-j2j46 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Oct 31 00:01:51.804434 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:51.804218 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-j2j46" podUID="a0f800b8-0d6f-484a-bfa8-e71b15ce18e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Oct 31 00:01:52.803826 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:52.803787 4927 patch_prober.go:29] interesting pod/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-j2j46 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Oct 31 00:01:52.804019 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:52.803856 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-j2j46" podUID="a0f800b8-0d6f-484a-bfa8-e71b15ce18e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Oct 31 00:01:53.305990 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:53.305948 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:01:53.809559 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:53.809528 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-j2j46" Oct 31 00:01:53.810042 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:53.810014 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-j2j46" Oct 31 00:01:53.810757 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:53.810735 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-j2j46" Oct 31 00:01:57.887250 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:01:57.887219 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:02:01.054043 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:01.054003 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d5cc73ad-pf2kb"] Oct 31 00:02:01.101768 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:01.101731 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:02:01.104932 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:01.104906 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d5cc73ad-pf2kb"] Oct 31 00:02:01.255174 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:01.255117 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:02:02.064405 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:02.064235 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c43c9a94-nxv5l"] Oct 31 00:02:02.113719 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:02.113676 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c43c9a94-nxv5l"] Oct 31 00:02:02.519819 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:02.519728 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:02:03.056383 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:03.056340 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-20c2fd60-6jhdm"] Oct 31 00:02:03.089497 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:03.089461 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c6b629f-f032-469a-a06f-399b6a1a19c8" path="/var/lib/kubelet/pods/1c6b629f-f032-469a-a06f-399b6a1a19c8/volumes" Oct 31 00:02:03.090004 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:03.089987 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="833799d2-5e63-4445-9e34-42bbb7e13359" path="/var/lib/kubelet/pods/833799d2-5e63-4445-9e34-42bbb7e13359/volumes" Oct 31 00:02:03.105975 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:03.105934 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-20c2fd60-6jhdm"] Oct 31 00:02:04.667358 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:04.663593 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-27e82b/distinct-w2-27e82b-kserve-75f475fcd9-mccsz" podUID="4ba8ced4-e93f-46ac-87c5-3d0b7fadc02d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:02:05.209729 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:05.209674 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1aa39837-dbe0-4478-a514-b4afc328c705" path="/var/lib/kubelet/pods/1aa39837-dbe0-4478-a514-b4afc328c705/volumes" Oct 31 00:02:06.829551 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:06.829513 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6d701af0-6r8hs"] Oct 31 00:02:06.830400 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:06.830378 4927 cpu_manager.go:414] "containerMap: removing container" podUID="fa9d6354-c4e7-40fd-b163-103d75f0e34e" containerName="revoke-keys" Oct 31 00:02:06.830400 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:06.830400 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa9d6354-c4e7-40fd-b163-103d75f0e34e" containerName="revoke-keys" Oct 31 00:02:06.830612 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:06.830598 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa9d6354-c4e7-40fd-b163-103d75f0e34e" containerName="revoke-keys" Oct 31 00:02:06.835639 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:06.835616 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d701af0-6r8hs" Oct 31 00:02:06.891656 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:06.891625 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6d701af0-6r8hs"] Oct 31 00:02:06.911477 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:06.911436 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8vwq\" (UniqueName: \"kubernetes.io/projected/fdd8d855-2e0f-4843-9705-dc15f15625d1-kube-api-access-g8vwq\") pod \"maas-api-revoke-sub-6d701af0-6r8hs\" (UID: \"fdd8d855-2e0f-4843-9705-dc15f15625d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d701af0-6r8hs" Oct 31 00:02:06.911637 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:06.911595 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fdd8d855-2e0f-4843-9705-dc15f15625d1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6d701af0-6r8hs\" (UID: \"fdd8d855-2e0f-4843-9705-dc15f15625d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d701af0-6r8hs" Oct 31 00:02:06.911717 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:06.911700 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fdd8d855-2e0f-4843-9705-dc15f15625d1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6d701af0-6r8hs\" (UID: \"fdd8d855-2e0f-4843-9705-dc15f15625d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d701af0-6r8hs" Oct 31 00:02:06.936707 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:06.936678 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:02:06.936896 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:06.936878 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4fdzs\"" Oct 31 00:02:07.012800 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:07.012760 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fdd8d855-2e0f-4843-9705-dc15f15625d1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6d701af0-6r8hs\" (UID: \"fdd8d855-2e0f-4843-9705-dc15f15625d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d701af0-6r8hs" Oct 31 00:02:07.012981 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:07.012836 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fdd8d855-2e0f-4843-9705-dc15f15625d1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6d701af0-6r8hs\" (UID: \"fdd8d855-2e0f-4843-9705-dc15f15625d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d701af0-6r8hs" Oct 31 00:02:07.012981 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:07.012964 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8vwq\" (UniqueName: \"kubernetes.io/projected/fdd8d855-2e0f-4843-9705-dc15f15625d1-kube-api-access-g8vwq\") pod \"maas-api-revoke-sub-6d701af0-6r8hs\" (UID: \"fdd8d855-2e0f-4843-9705-dc15f15625d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d701af0-6r8hs" Oct 31 00:02:07.013869 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:07.013836 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fdd8d855-2e0f-4843-9705-dc15f15625d1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6d701af0-6r8hs\" (UID: \"fdd8d855-2e0f-4843-9705-dc15f15625d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d701af0-6r8hs" Oct 31 00:02:07.015358 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:07.015315 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fdd8d855-2e0f-4843-9705-dc15f15625d1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6d701af0-6r8hs\" (UID: \"fdd8d855-2e0f-4843-9705-dc15f15625d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d701af0-6r8hs" Oct 31 00:02:07.076093 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:07.076059 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8vwq\" (UniqueName: \"kubernetes.io/projected/fdd8d855-2e0f-4843-9705-dc15f15625d1-kube-api-access-g8vwq\") pod \"maas-api-revoke-sub-6d701af0-6r8hs\" (UID: \"fdd8d855-2e0f-4843-9705-dc15f15625d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d701af0-6r8hs" Oct 31 00:02:07.153740 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:07.153652 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d701af0-6r8hs" Oct 31 00:02:07.384569 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:07.384527 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4143d909-tk5lx"] Oct 31 00:02:07.390083 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:07.390057 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4143d909-tk5lx" Oct 31 00:02:07.418366 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:07.418306 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xw7dh\" (UniqueName: \"kubernetes.io/projected/f4cd0f96-bac0-49bc-a27f-4c2ae5e7aaf7-kube-api-access-xw7dh\") pod \"maas-api-revoke-sub-4143d909-tk5lx\" (UID: \"f4cd0f96-bac0-49bc-a27f-4c2ae5e7aaf7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4143d909-tk5lx" Oct 31 00:02:07.418508 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:07.418435 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4cd0f96-bac0-49bc-a27f-4c2ae5e7aaf7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4143d909-tk5lx\" (UID: \"f4cd0f96-bac0-49bc-a27f-4c2ae5e7aaf7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4143d909-tk5lx" Oct 31 00:02:07.418661 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:07.418642 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4cd0f96-bac0-49bc-a27f-4c2ae5e7aaf7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4143d909-tk5lx\" (UID: \"f4cd0f96-bac0-49bc-a27f-4c2ae5e7aaf7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4143d909-tk5lx" Oct 31 00:02:07.452431 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:07.452397 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4143d909-tk5lx"] Oct 31 00:02:07.519699 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:07.519663 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4cd0f96-bac0-49bc-a27f-4c2ae5e7aaf7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4143d909-tk5lx\" (UID: \"f4cd0f96-bac0-49bc-a27f-4c2ae5e7aaf7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4143d909-tk5lx" Oct 31 00:02:07.519883 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:07.519812 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xw7dh\" (UniqueName: \"kubernetes.io/projected/f4cd0f96-bac0-49bc-a27f-4c2ae5e7aaf7-kube-api-access-xw7dh\") pod \"maas-api-revoke-sub-4143d909-tk5lx\" (UID: \"f4cd0f96-bac0-49bc-a27f-4c2ae5e7aaf7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4143d909-tk5lx" Oct 31 00:02:07.519883 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:07.519865 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4cd0f96-bac0-49bc-a27f-4c2ae5e7aaf7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4143d909-tk5lx\" (UID: \"f4cd0f96-bac0-49bc-a27f-4c2ae5e7aaf7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4143d909-tk5lx" Oct 31 00:02:07.520526 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:07.520494 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4cd0f96-bac0-49bc-a27f-4c2ae5e7aaf7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4143d909-tk5lx\" (UID: \"f4cd0f96-bac0-49bc-a27f-4c2ae5e7aaf7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4143d909-tk5lx" Oct 31 00:02:07.522398 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:07.522367 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4cd0f96-bac0-49bc-a27f-4c2ae5e7aaf7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4143d909-tk5lx\" (UID: \"f4cd0f96-bac0-49bc-a27f-4c2ae5e7aaf7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4143d909-tk5lx" Oct 31 00:02:07.564791 ip-10-0-141-180 kubenswrapper[4927]: W1031 00:02:07.564765 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdd8d855_2e0f_4843_9705_dc15f15625d1.slice/crio-b6f16a3ea9ab47b59e5d10e26ee9b5f412fbb0be11efb8ce0866944d040fd64e WatchSource:0}: Error finding container b6f16a3ea9ab47b59e5d10e26ee9b5f412fbb0be11efb8ce0866944d040fd64e: Status 404 returned error can't find the container with id b6f16a3ea9ab47b59e5d10e26ee9b5f412fbb0be11efb8ce0866944d040fd64e Oct 31 00:02:07.565803 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:07.565776 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6d701af0-6r8hs"] Oct 31 00:02:07.583134 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:07.583105 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xw7dh\" (UniqueName: \"kubernetes.io/projected/f4cd0f96-bac0-49bc-a27f-4c2ae5e7aaf7-kube-api-access-xw7dh\") pod \"maas-api-revoke-sub-4143d909-tk5lx\" (UID: \"f4cd0f96-bac0-49bc-a27f-4c2ae5e7aaf7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4143d909-tk5lx" Oct 31 00:02:07.705620 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:07.705545 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4143d909-tk5lx" Oct 31 00:02:07.732633 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:07.732604 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:02:07.809501 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:07.780369 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d701af0-6r8hs" event={"ID":"fdd8d855-2e0f-4843-9705-dc15f15625d1","Type":"ContainerStarted","Data":"b6f16a3ea9ab47b59e5d10e26ee9b5f412fbb0be11efb8ce0866944d040fd64e"} Oct 31 00:02:08.100118 ip-10-0-141-180 kubenswrapper[4927]: W1031 00:02:08.100083 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4cd0f96_bac0_49bc_a27f_4c2ae5e7aaf7.slice/crio-6ba08ea9f2e144416f6909fc40a6fdb8fda27be3d1bdaa86e2c41adc43d7b5cc WatchSource:0}: Error finding container 6ba08ea9f2e144416f6909fc40a6fdb8fda27be3d1bdaa86e2c41adc43d7b5cc: Status 404 returned error can't find the container with id 6ba08ea9f2e144416f6909fc40a6fdb8fda27be3d1bdaa86e2c41adc43d7b5cc Oct 31 00:02:08.100967 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:08.100939 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4143d909-tk5lx"] Oct 31 00:02:08.790442 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:08.790315 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4143d909-tk5lx" event={"ID":"f4cd0f96-bac0-49bc-a27f-4c2ae5e7aaf7","Type":"ContainerStarted","Data":"22a2be1a89949d0a19071805fa1a872deafb7323b2455d4b3619589d1a246af5"} Oct 31 00:02:08.790621 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:08.790444 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4143d909-tk5lx" event={"ID":"f4cd0f96-bac0-49bc-a27f-4c2ae5e7aaf7","Type":"ContainerStarted","Data":"6ba08ea9f2e144416f6909fc40a6fdb8fda27be3d1bdaa86e2c41adc43d7b5cc"} Oct 31 00:02:08.791988 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:08.791961 4927 generic.go:356] "Generic (PLEG): container finished" podID="fdd8d855-2e0f-4843-9705-dc15f15625d1" containerID="a0e7e4d9c971111513db178ecf9667f819eb1c765085a9019ad15ca0ca63bea2" exitCode=0 Oct 31 00:02:08.792106 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:08.792038 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d701af0-6r8hs" event={"ID":"fdd8d855-2e0f-4843-9705-dc15f15625d1","Type":"ContainerDied","Data":"a0e7e4d9c971111513db178ecf9667f819eb1c765085a9019ad15ca0ca63bea2"} Oct 31 00:02:08.911396 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:08.910782 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4143d909-tk5lx" podStartSLOduration=1.910765712 podStartE2EDuration="1.910765712s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:02:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:02:08.910498162 +0000 UTC m=+1241.946828986" watchObservedRunningTime="2026-10-31 00:02:08.910765712 +0000 UTC m=+1241.947096536" Oct 31 00:02:09.802292 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:09.802256 4927 generic.go:356] "Generic (PLEG): container finished" podID="f4cd0f96-bac0-49bc-a27f-4c2ae5e7aaf7" containerID="22a2be1a89949d0a19071805fa1a872deafb7323b2455d4b3619589d1a246af5" exitCode=0 Oct 31 00:02:09.802820 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:09.802356 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4143d909-tk5lx" event={"ID":"f4cd0f96-bac0-49bc-a27f-4c2ae5e7aaf7","Type":"ContainerDied","Data":"22a2be1a89949d0a19071805fa1a872deafb7323b2455d4b3619589d1a246af5"} Oct 31 00:02:10.008047 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:10.008012 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d701af0-6r8hs" Oct 31 00:02:10.063258 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:10.063175 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fdd8d855-2e0f-4843-9705-dc15f15625d1-kube-api-access-g8vwq\" (UniqueName: \"kubernetes.io/projected/fdd8d855-2e0f-4843-9705-dc15f15625d1-kube-api-access-g8vwq\") pod \"fdd8d855-2e0f-4843-9705-dc15f15625d1\" (UID: \"fdd8d855-2e0f-4843-9705-dc15f15625d1\") " Oct 31 00:02:10.063258 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:10.063230 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fdd8d855-2e0f-4843-9705-dc15f15625d1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fdd8d855-2e0f-4843-9705-dc15f15625d1-maas-api-openshift-ca\") pod \"fdd8d855-2e0f-4843-9705-dc15f15625d1\" (UID: \"fdd8d855-2e0f-4843-9705-dc15f15625d1\") " Oct 31 00:02:10.063578 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:10.063460 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fdd8d855-2e0f-4843-9705-dc15f15625d1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fdd8d855-2e0f-4843-9705-dc15f15625d1-maas-api-xks-ca\") pod \"fdd8d855-2e0f-4843-9705-dc15f15625d1\" (UID: \"fdd8d855-2e0f-4843-9705-dc15f15625d1\") " Oct 31 00:02:10.063735 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:10.063709 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fdd8d855-2e0f-4843-9705-dc15f15625d1-maas-api-openshift-ca" pod "fdd8d855-2e0f-4843-9705-dc15f15625d1" (UID: "fdd8d855-2e0f-4843-9705-dc15f15625d1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:02:10.064095 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:10.064071 4927 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fdd8d855-2e0f-4843-9705-dc15f15625d1-maas-api-openshift-ca\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:02:10.065580 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:10.065550 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fdd8d855-2e0f-4843-9705-dc15f15625d1-maas-api-xks-ca" pod "fdd8d855-2e0f-4843-9705-dc15f15625d1" (UID: "fdd8d855-2e0f-4843-9705-dc15f15625d1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:02:10.065672 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:10.065637 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fdd8d855-2e0f-4843-9705-dc15f15625d1-kube-api-access-g8vwq" pod "fdd8d855-2e0f-4843-9705-dc15f15625d1" (UID: "fdd8d855-2e0f-4843-9705-dc15f15625d1"). InnerVolumeSpecName "kube-api-access-g8vwq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:02:10.068960 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:10.068936 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-56af7078-tvhbk"] Oct 31 00:02:10.085641 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:10.080695 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-56af7078-tvhbk"] Oct 31 00:02:10.165450 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:10.165412 4927 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fdd8d855-2e0f-4843-9705-dc15f15625d1-maas-api-xks-ca\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:02:10.165450 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:10.165449 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g8vwq\" (UniqueName: \"kubernetes.io/projected/fdd8d855-2e0f-4843-9705-dc15f15625d1-kube-api-access-g8vwq\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:02:10.778867 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:10.778834 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b383413a-nvgmn"] Oct 31 00:02:10.780006 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:10.779973 4927 cpu_manager.go:414] "containerMap: removing container" podUID="fdd8d855-2e0f-4843-9705-dc15f15625d1" containerName="revoke-keys" Oct 31 00:02:10.780006 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:10.779999 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fdd8d855-2e0f-4843-9705-dc15f15625d1" containerName="revoke-keys" Oct 31 00:02:10.780235 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:10.780220 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="fdd8d855-2e0f-4843-9705-dc15f15625d1" containerName="revoke-keys" Oct 31 00:02:10.784273 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:10.784255 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b383413a-nvgmn" Oct 31 00:02:10.815390 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:10.815361 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d701af0-6r8hs" Oct 31 00:02:10.815836 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:10.815386 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d701af0-6r8hs" event={"ID":"fdd8d855-2e0f-4843-9705-dc15f15625d1","Type":"ContainerDied","Data":"b6f16a3ea9ab47b59e5d10e26ee9b5f412fbb0be11efb8ce0866944d040fd64e"} Oct 31 00:02:10.815836 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:10.815435 4927 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b6f16a3ea9ab47b59e5d10e26ee9b5f412fbb0be11efb8ce0866944d040fd64e" Oct 31 00:02:10.841884 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:10.841804 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b383413a-nvgmn"] Oct 31 00:02:10.875302 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:10.875258 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29750093-15c3-4899-a7b2-b8f46713c2ee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b383413a-nvgmn\" (UID: \"29750093-15c3-4899-a7b2-b8f46713c2ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b383413a-nvgmn" Oct 31 00:02:10.875825 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:10.875798 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29750093-15c3-4899-a7b2-b8f46713c2ee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b383413a-nvgmn\" (UID: \"29750093-15c3-4899-a7b2-b8f46713c2ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b383413a-nvgmn" Oct 31 00:02:10.876010 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:10.875992 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9c9j\" (UniqueName: \"kubernetes.io/projected/29750093-15c3-4899-a7b2-b8f46713c2ee-kube-api-access-v9c9j\") pod \"maas-api-revoke-sub-b383413a-nvgmn\" (UID: \"29750093-15c3-4899-a7b2-b8f46713c2ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b383413a-nvgmn" Oct 31 00:02:10.978003 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:10.977976 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29750093-15c3-4899-a7b2-b8f46713c2ee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b383413a-nvgmn\" (UID: \"29750093-15c3-4899-a7b2-b8f46713c2ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b383413a-nvgmn" Oct 31 00:02:10.978104 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:10.978013 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9c9j\" (UniqueName: \"kubernetes.io/projected/29750093-15c3-4899-a7b2-b8f46713c2ee-kube-api-access-v9c9j\") pod \"maas-api-revoke-sub-b383413a-nvgmn\" (UID: \"29750093-15c3-4899-a7b2-b8f46713c2ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b383413a-nvgmn" Oct 31 00:02:10.978173 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:10.978124 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29750093-15c3-4899-a7b2-b8f46713c2ee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b383413a-nvgmn\" (UID: \"29750093-15c3-4899-a7b2-b8f46713c2ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b383413a-nvgmn" Oct 31 00:02:10.978842 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:10.978816 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29750093-15c3-4899-a7b2-b8f46713c2ee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b383413a-nvgmn\" (UID: \"29750093-15c3-4899-a7b2-b8f46713c2ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b383413a-nvgmn" Oct 31 00:02:10.980319 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:10.980297 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29750093-15c3-4899-a7b2-b8f46713c2ee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b383413a-nvgmn\" (UID: \"29750093-15c3-4899-a7b2-b8f46713c2ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b383413a-nvgmn" Oct 31 00:02:11.009653 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:11.009627 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4143d909-tk5lx" Oct 31 00:02:11.041384 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:11.041285 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9c9j\" (UniqueName: \"kubernetes.io/projected/29750093-15c3-4899-a7b2-b8f46713c2ee-kube-api-access-v9c9j\") pod \"maas-api-revoke-sub-b383413a-nvgmn\" (UID: \"29750093-15c3-4899-a7b2-b8f46713c2ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b383413a-nvgmn" Oct 31 00:02:11.079394 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:11.079365 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f4cd0f96-bac0-49bc-a27f-4c2ae5e7aaf7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4cd0f96-bac0-49bc-a27f-4c2ae5e7aaf7-maas-api-xks-ca\") pod \"f4cd0f96-bac0-49bc-a27f-4c2ae5e7aaf7\" (UID: \"f4cd0f96-bac0-49bc-a27f-4c2ae5e7aaf7\") " Oct 31 00:02:11.079572 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:11.079460 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f4cd0f96-bac0-49bc-a27f-4c2ae5e7aaf7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4cd0f96-bac0-49bc-a27f-4c2ae5e7aaf7-maas-api-openshift-ca\") pod \"f4cd0f96-bac0-49bc-a27f-4c2ae5e7aaf7\" (UID: \"f4cd0f96-bac0-49bc-a27f-4c2ae5e7aaf7\") " Oct 31 00:02:11.079572 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:11.079508 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4cd0f96-bac0-49bc-a27f-4c2ae5e7aaf7-kube-api-access-xw7dh\" (UniqueName: \"kubernetes.io/projected/f4cd0f96-bac0-49bc-a27f-4c2ae5e7aaf7-kube-api-access-xw7dh\") pod \"f4cd0f96-bac0-49bc-a27f-4c2ae5e7aaf7\" (UID: \"f4cd0f96-bac0-49bc-a27f-4c2ae5e7aaf7\") " Oct 31 00:02:11.079863 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:11.079837 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4cd0f96-bac0-49bc-a27f-4c2ae5e7aaf7-maas-api-openshift-ca" pod "f4cd0f96-bac0-49bc-a27f-4c2ae5e7aaf7" (UID: "f4cd0f96-bac0-49bc-a27f-4c2ae5e7aaf7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:02:11.080020 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:11.080004 4927 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4cd0f96-bac0-49bc-a27f-4c2ae5e7aaf7-maas-api-openshift-ca\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:02:11.081588 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:11.081562 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f4cd0f96-bac0-49bc-a27f-4c2ae5e7aaf7-maas-api-xks-ca" pod "f4cd0f96-bac0-49bc-a27f-4c2ae5e7aaf7" (UID: "f4cd0f96-bac0-49bc-a27f-4c2ae5e7aaf7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:02:11.081759 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:11.081738 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4cd0f96-bac0-49bc-a27f-4c2ae5e7aaf7-kube-api-access-xw7dh" pod "f4cd0f96-bac0-49bc-a27f-4c2ae5e7aaf7" (UID: "f4cd0f96-bac0-49bc-a27f-4c2ae5e7aaf7"). InnerVolumeSpecName "kube-api-access-xw7dh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:02:11.091227 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:11.091199 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1cb89f94-8f2a-4cd7-b1da-509f7d5f883b" path="/var/lib/kubelet/pods/1cb89f94-8f2a-4cd7-b1da-509f7d5f883b/volumes" Oct 31 00:02:11.098992 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:11.098969 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b383413a-nvgmn" Oct 31 00:02:11.182684 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:11.182632 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xw7dh\" (UniqueName: \"kubernetes.io/projected/f4cd0f96-bac0-49bc-a27f-4c2ae5e7aaf7-kube-api-access-xw7dh\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:02:11.182818 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:11.182690 4927 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4cd0f96-bac0-49bc-a27f-4c2ae5e7aaf7-maas-api-xks-ca\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:02:11.521747 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:11.521703 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b383413a-nvgmn"] Oct 31 00:02:11.692283 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:11.692253 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c81c200d-pjrnb"] Oct 31 00:02:11.692951 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:11.692935 4927 cpu_manager.go:414] "containerMap: removing container" podUID="f4cd0f96-bac0-49bc-a27f-4c2ae5e7aaf7" containerName="revoke-keys" Oct 31 00:02:11.693001 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:11.692952 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4cd0f96-bac0-49bc-a27f-4c2ae5e7aaf7" containerName="revoke-keys" Oct 31 00:02:11.693070 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:11.693062 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4cd0f96-bac0-49bc-a27f-4c2ae5e7aaf7" containerName="revoke-keys" Oct 31 00:02:11.697902 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:11.697885 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c81c200d-pjrnb" Oct 31 00:02:11.760870 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:11.760840 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c81c200d-pjrnb"] Oct 31 00:02:11.791373 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:11.791258 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a54b0594-3037-4e79-83bf-f9e13b14d204-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c81c200d-pjrnb\" (UID: \"a54b0594-3037-4e79-83bf-f9e13b14d204\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c81c200d-pjrnb" Oct 31 00:02:11.791541 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:11.791457 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7c9wr\" (UniqueName: \"kubernetes.io/projected/a54b0594-3037-4e79-83bf-f9e13b14d204-kube-api-access-7c9wr\") pod \"maas-api-revoke-sub-c81c200d-pjrnb\" (UID: \"a54b0594-3037-4e79-83bf-f9e13b14d204\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c81c200d-pjrnb" Oct 31 00:02:11.791728 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:11.791704 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a54b0594-3037-4e79-83bf-f9e13b14d204-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c81c200d-pjrnb\" (UID: \"a54b0594-3037-4e79-83bf-f9e13b14d204\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c81c200d-pjrnb" Oct 31 00:02:11.828269 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:11.828244 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4143d909-tk5lx" Oct 31 00:02:11.828269 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:11.828251 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4143d909-tk5lx" event={"ID":"f4cd0f96-bac0-49bc-a27f-4c2ae5e7aaf7","Type":"ContainerDied","Data":"6ba08ea9f2e144416f6909fc40a6fdb8fda27be3d1bdaa86e2c41adc43d7b5cc"} Oct 31 00:02:11.829112 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:11.828300 4927 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6ba08ea9f2e144416f6909fc40a6fdb8fda27be3d1bdaa86e2c41adc43d7b5cc" Oct 31 00:02:11.830429 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:11.830397 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b383413a-nvgmn" event={"ID":"29750093-15c3-4899-a7b2-b8f46713c2ee","Type":"ContainerStarted","Data":"f0706429e897ef20d2060797cab454eae2f9375a03b9b7984a56bff145cb786a"} Oct 31 00:02:11.892921 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:11.892890 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a54b0594-3037-4e79-83bf-f9e13b14d204-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c81c200d-pjrnb\" (UID: \"a54b0594-3037-4e79-83bf-f9e13b14d204\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c81c200d-pjrnb" Oct 31 00:02:11.893097 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:11.892946 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a54b0594-3037-4e79-83bf-f9e13b14d204-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c81c200d-pjrnb\" (UID: \"a54b0594-3037-4e79-83bf-f9e13b14d204\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c81c200d-pjrnb" Oct 31 00:02:11.893097 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:11.893029 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7c9wr\" (UniqueName: \"kubernetes.io/projected/a54b0594-3037-4e79-83bf-f9e13b14d204-kube-api-access-7c9wr\") pod \"maas-api-revoke-sub-c81c200d-pjrnb\" (UID: \"a54b0594-3037-4e79-83bf-f9e13b14d204\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c81c200d-pjrnb" Oct 31 00:02:11.893588 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:11.893569 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a54b0594-3037-4e79-83bf-f9e13b14d204-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c81c200d-pjrnb\" (UID: \"a54b0594-3037-4e79-83bf-f9e13b14d204\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c81c200d-pjrnb" Oct 31 00:02:11.895218 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:11.895200 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a54b0594-3037-4e79-83bf-f9e13b14d204-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c81c200d-pjrnb\" (UID: \"a54b0594-3037-4e79-83bf-f9e13b14d204\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c81c200d-pjrnb" Oct 31 00:02:11.955287 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:11.955257 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7c9wr\" (UniqueName: \"kubernetes.io/projected/a54b0594-3037-4e79-83bf-f9e13b14d204-kube-api-access-7c9wr\") pod \"maas-api-revoke-sub-c81c200d-pjrnb\" (UID: \"a54b0594-3037-4e79-83bf-f9e13b14d204\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c81c200d-pjrnb" Oct 31 00:02:11.959626 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:11.959586 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b383413a-nvgmn" podStartSLOduration=1.959572744 podStartE2EDuration="1.959572744s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:02:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:02:11.951567771 +0000 UTC m=+1244.987898598" watchObservedRunningTime="2026-10-31 00:02:11.959572744 +0000 UTC m=+1244.995903566" Oct 31 00:02:12.015391 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:12.015361 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c81c200d-pjrnb" Oct 31 00:02:12.393952 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:12.393865 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:02:12.406682 ip-10-0-141-180 kubenswrapper[4927]: W1031 00:02:12.406649 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda54b0594_3037_4e79_83bf_f9e13b14d204.slice/crio-25c7b7840ffdf4cd70470101a906049d377784c13f45dccb9b6c17aac68bfa05 WatchSource:0}: Error finding container 25c7b7840ffdf4cd70470101a906049d377784c13f45dccb9b6c17aac68bfa05: Status 404 returned error can't find the container with id 25c7b7840ffdf4cd70470101a906049d377784c13f45dccb9b6c17aac68bfa05 Oct 31 00:02:12.407415 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:12.407392 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c81c200d-pjrnb"] Oct 31 00:02:12.841762 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:12.841731 4927 generic.go:356] "Generic (PLEG): container finished" podID="29750093-15c3-4899-a7b2-b8f46713c2ee" containerID="db9f426aad062a104cc9fe6aba026c4c50a645af3c2f97bfc927c88bd016e37a" exitCode=0 Oct 31 00:02:12.842130 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:12.841824 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b383413a-nvgmn" event={"ID":"29750093-15c3-4899-a7b2-b8f46713c2ee","Type":"ContainerDied","Data":"db9f426aad062a104cc9fe6aba026c4c50a645af3c2f97bfc927c88bd016e37a"} Oct 31 00:02:12.843803 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:12.843773 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c81c200d-pjrnb" event={"ID":"a54b0594-3037-4e79-83bf-f9e13b14d204","Type":"ContainerStarted","Data":"7b5aa63c0b6630abe4b023572b18a1858f2ec88be22b58c0ea310439124e1f58"} Oct 31 00:02:12.843929 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:12.843818 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c81c200d-pjrnb" event={"ID":"a54b0594-3037-4e79-83bf-f9e13b14d204","Type":"ContainerStarted","Data":"25c7b7840ffdf4cd70470101a906049d377784c13f45dccb9b6c17aac68bfa05"} Oct 31 00:02:13.086690 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:13.086594 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c81c200d-pjrnb" podStartSLOduration=2.086579018 podStartE2EDuration="2.086579018s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:02:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:02:13.080309274 +0000 UTC m=+1246.116640096" watchObservedRunningTime="2026-10-31 00:02:13.086579018 +0000 UTC m=+1246.122909839" Oct 31 00:02:13.854393 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:13.854317 4927 generic.go:356] "Generic (PLEG): container finished" podID="a54b0594-3037-4e79-83bf-f9e13b14d204" containerID="7b5aa63c0b6630abe4b023572b18a1858f2ec88be22b58c0ea310439124e1f58" exitCode=0 Oct 31 00:02:13.854393 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:13.854375 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c81c200d-pjrnb" event={"ID":"a54b0594-3037-4e79-83bf-f9e13b14d204","Type":"ContainerDied","Data":"7b5aa63c0b6630abe4b023572b18a1858f2ec88be22b58c0ea310439124e1f58"} Oct 31 00:02:14.024642 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:14.024611 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b383413a-nvgmn" Oct 31 00:02:14.122469 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:14.122378 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29750093-15c3-4899-a7b2-b8f46713c2ee-kube-api-access-v9c9j\" (UniqueName: \"kubernetes.io/projected/29750093-15c3-4899-a7b2-b8f46713c2ee-kube-api-access-v9c9j\") pod \"29750093-15c3-4899-a7b2-b8f46713c2ee\" (UID: \"29750093-15c3-4899-a7b2-b8f46713c2ee\") " Oct 31 00:02:14.122469 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:14.122444 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/29750093-15c3-4899-a7b2-b8f46713c2ee-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29750093-15c3-4899-a7b2-b8f46713c2ee-maas-api-xks-ca\") pod \"29750093-15c3-4899-a7b2-b8f46713c2ee\" (UID: \"29750093-15c3-4899-a7b2-b8f46713c2ee\") " Oct 31 00:02:14.122702 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:14.122553 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/29750093-15c3-4899-a7b2-b8f46713c2ee-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29750093-15c3-4899-a7b2-b8f46713c2ee-maas-api-openshift-ca\") pod \"29750093-15c3-4899-a7b2-b8f46713c2ee\" (UID: \"29750093-15c3-4899-a7b2-b8f46713c2ee\") " Oct 31 00:02:14.122992 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:14.122963 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29750093-15c3-4899-a7b2-b8f46713c2ee-maas-api-openshift-ca" pod "29750093-15c3-4899-a7b2-b8f46713c2ee" (UID: "29750093-15c3-4899-a7b2-b8f46713c2ee"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:02:14.123173 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:14.123154 4927 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29750093-15c3-4899-a7b2-b8f46713c2ee-maas-api-openshift-ca\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:02:14.124676 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:14.124648 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29750093-15c3-4899-a7b2-b8f46713c2ee-kube-api-access-v9c9j" pod "29750093-15c3-4899-a7b2-b8f46713c2ee" (UID: "29750093-15c3-4899-a7b2-b8f46713c2ee"). InnerVolumeSpecName "kube-api-access-v9c9j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:02:14.124760 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:14.124707 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29750093-15c3-4899-a7b2-b8f46713c2ee-maas-api-xks-ca" pod "29750093-15c3-4899-a7b2-b8f46713c2ee" (UID: "29750093-15c3-4899-a7b2-b8f46713c2ee"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:02:14.224861 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:14.224828 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v9c9j\" (UniqueName: \"kubernetes.io/projected/29750093-15c3-4899-a7b2-b8f46713c2ee-kube-api-access-v9c9j\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:02:14.224861 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:14.224859 4927 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29750093-15c3-4899-a7b2-b8f46713c2ee-maas-api-xks-ca\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:02:14.864362 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:14.864317 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b383413a-nvgmn" Oct 31 00:02:14.864777 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:14.864313 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b383413a-nvgmn" event={"ID":"29750093-15c3-4899-a7b2-b8f46713c2ee","Type":"ContainerDied","Data":"f0706429e897ef20d2060797cab454eae2f9375a03b9b7984a56bff145cb786a"} Oct 31 00:02:14.864777 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:14.864435 4927 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f0706429e897ef20d2060797cab454eae2f9375a03b9b7984a56bff145cb786a" Oct 31 00:02:15.032560 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:15.032531 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c81c200d-pjrnb" Oct 31 00:02:15.136390 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:15.136312 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a54b0594-3037-4e79-83bf-f9e13b14d204-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a54b0594-3037-4e79-83bf-f9e13b14d204-maas-api-xks-ca\") pod \"a54b0594-3037-4e79-83bf-f9e13b14d204\" (UID: \"a54b0594-3037-4e79-83bf-f9e13b14d204\") " Oct 31 00:02:15.136516 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:15.136436 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a54b0594-3037-4e79-83bf-f9e13b14d204-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a54b0594-3037-4e79-83bf-f9e13b14d204-maas-api-openshift-ca\") pod \"a54b0594-3037-4e79-83bf-f9e13b14d204\" (UID: \"a54b0594-3037-4e79-83bf-f9e13b14d204\") " Oct 31 00:02:15.136569 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:15.136529 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a54b0594-3037-4e79-83bf-f9e13b14d204-kube-api-access-7c9wr\" (UniqueName: \"kubernetes.io/projected/a54b0594-3037-4e79-83bf-f9e13b14d204-kube-api-access-7c9wr\") pod \"a54b0594-3037-4e79-83bf-f9e13b14d204\" (UID: \"a54b0594-3037-4e79-83bf-f9e13b14d204\") " Oct 31 00:02:15.136778 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:15.136753 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a54b0594-3037-4e79-83bf-f9e13b14d204-maas-api-openshift-ca" pod "a54b0594-3037-4e79-83bf-f9e13b14d204" (UID: "a54b0594-3037-4e79-83bf-f9e13b14d204"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:02:15.137168 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:15.137144 4927 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a54b0594-3037-4e79-83bf-f9e13b14d204-maas-api-openshift-ca\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:02:15.138574 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:15.138550 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a54b0594-3037-4e79-83bf-f9e13b14d204-maas-api-xks-ca" pod "a54b0594-3037-4e79-83bf-f9e13b14d204" (UID: "a54b0594-3037-4e79-83bf-f9e13b14d204"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:02:15.138666 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:15.138648 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a54b0594-3037-4e79-83bf-f9e13b14d204-kube-api-access-7c9wr" pod "a54b0594-3037-4e79-83bf-f9e13b14d204" (UID: "a54b0594-3037-4e79-83bf-f9e13b14d204"). InnerVolumeSpecName "kube-api-access-7c9wr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:02:15.238368 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:15.238319 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7c9wr\" (UniqueName: \"kubernetes.io/projected/a54b0594-3037-4e79-83bf-f9e13b14d204-kube-api-access-7c9wr\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:02:15.238368 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:15.238361 4927 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a54b0594-3037-4e79-83bf-f9e13b14d204-maas-api-xks-ca\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:02:15.875144 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:15.875111 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c81c200d-pjrnb" Oct 31 00:02:15.875144 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:15.875127 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c81c200d-pjrnb" event={"ID":"a54b0594-3037-4e79-83bf-f9e13b14d204","Type":"ContainerDied","Data":"25c7b7840ffdf4cd70470101a906049d377784c13f45dccb9b6c17aac68bfa05"} Oct 31 00:02:15.875573 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:15.875171 4927 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="25c7b7840ffdf4cd70470101a906049d377784c13f45dccb9b6c17aac68bfa05" Oct 31 00:02:16.995847 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:16.995813 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:02:18.055299 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:18.055260 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-afa95927-2tlqf"] Oct 31 00:02:18.106317 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:18.106281 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-afa95927-2tlqf"] Oct 31 00:02:18.110884 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:18.110856 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-50b468fd-749rv"] Oct 31 00:02:18.114757 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:18.114738 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-50b468fd-749rv"] Oct 31 00:02:19.087321 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:19.087286 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="15ba1fff-a83b-41e6-9e54-3ce2d7417014" path="/var/lib/kubelet/pods/15ba1fff-a83b-41e6-9e54-3ce2d7417014/volumes" Oct 31 00:02:19.087993 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:19.087977 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="22c6669d-27fa-4841-a900-d4f70fbb9f2c" path="/var/lib/kubelet/pods/22c6669d-27fa-4841-a900-d4f70fbb9f2c/volumes" Oct 31 00:02:19.678374 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:02:19.678306 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sqv69" containerName="extractor" Oct 31 00:02:21.598212 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:21.598182 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:02:21.796303 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:21.796269 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-09732b29-8wg8w"] Oct 31 00:02:21.796957 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:21.796942 4927 cpu_manager.go:414] "containerMap: removing container" podUID="a54b0594-3037-4e79-83bf-f9e13b14d204" containerName="revoke-keys" Oct 31 00:02:21.797005 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:21.796958 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a54b0594-3037-4e79-83bf-f9e13b14d204" containerName="revoke-keys" Oct 31 00:02:21.797005 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:21.796980 4927 cpu_manager.go:414] "containerMap: removing container" podUID="29750093-15c3-4899-a7b2-b8f46713c2ee" containerName="revoke-keys" Oct 31 00:02:21.797005 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:21.796985 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29750093-15c3-4899-a7b2-b8f46713c2ee" containerName="revoke-keys" Oct 31 00:02:21.797094 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:21.797075 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="29750093-15c3-4899-a7b2-b8f46713c2ee" containerName="revoke-keys" Oct 31 00:02:21.797094 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:21.797093 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="a54b0594-3037-4e79-83bf-f9e13b14d204" containerName="revoke-keys" Oct 31 00:02:21.802091 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:21.802072 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09732b29-8wg8w" Oct 31 00:02:21.857430 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:21.857357 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-09732b29-8wg8w"] Oct 31 00:02:21.902051 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:21.902024 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:02:21.902051 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:21.902032 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4fdzs\"" Oct 31 00:02:21.943016 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:21.942971 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjtrf\" (UniqueName: \"kubernetes.io/projected/5ed957ff-0566-4f8a-a9fe-21a5a6b428e9-kube-api-access-bjtrf\") pod \"maas-api-revoke-sub-09732b29-8wg8w\" (UID: \"5ed957ff-0566-4f8a-a9fe-21a5a6b428e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09732b29-8wg8w" Oct 31 00:02:21.943174 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:21.943138 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ed957ff-0566-4f8a-a9fe-21a5a6b428e9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-09732b29-8wg8w\" (UID: \"5ed957ff-0566-4f8a-a9fe-21a5a6b428e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09732b29-8wg8w" Oct 31 00:02:21.943224 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:21.943173 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ed957ff-0566-4f8a-a9fe-21a5a6b428e9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-09732b29-8wg8w\" (UID: \"5ed957ff-0566-4f8a-a9fe-21a5a6b428e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09732b29-8wg8w" Oct 31 00:02:22.044132 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:22.044098 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjtrf\" (UniqueName: \"kubernetes.io/projected/5ed957ff-0566-4f8a-a9fe-21a5a6b428e9-kube-api-access-bjtrf\") pod \"maas-api-revoke-sub-09732b29-8wg8w\" (UID: \"5ed957ff-0566-4f8a-a9fe-21a5a6b428e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09732b29-8wg8w" Oct 31 00:02:22.044310 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:22.044190 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ed957ff-0566-4f8a-a9fe-21a5a6b428e9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-09732b29-8wg8w\" (UID: \"5ed957ff-0566-4f8a-a9fe-21a5a6b428e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09732b29-8wg8w" Oct 31 00:02:22.044310 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:22.044214 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ed957ff-0566-4f8a-a9fe-21a5a6b428e9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-09732b29-8wg8w\" (UID: \"5ed957ff-0566-4f8a-a9fe-21a5a6b428e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09732b29-8wg8w" Oct 31 00:02:22.044894 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:22.044870 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ed957ff-0566-4f8a-a9fe-21a5a6b428e9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-09732b29-8wg8w\" (UID: \"5ed957ff-0566-4f8a-a9fe-21a5a6b428e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09732b29-8wg8w" Oct 31 00:02:22.046574 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:22.046554 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ed957ff-0566-4f8a-a9fe-21a5a6b428e9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-09732b29-8wg8w\" (UID: \"5ed957ff-0566-4f8a-a9fe-21a5a6b428e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09732b29-8wg8w" Oct 31 00:02:22.106848 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:22.106815 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjtrf\" (UniqueName: \"kubernetes.io/projected/5ed957ff-0566-4f8a-a9fe-21a5a6b428e9-kube-api-access-bjtrf\") pod \"maas-api-revoke-sub-09732b29-8wg8w\" (UID: \"5ed957ff-0566-4f8a-a9fe-21a5a6b428e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09732b29-8wg8w" Oct 31 00:02:22.118543 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:22.118484 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09732b29-8wg8w" Oct 31 00:02:22.526672 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:22.526637 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-09732b29-8wg8w"] Oct 31 00:02:22.938773 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:22.938739 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09732b29-8wg8w" event={"ID":"5ed957ff-0566-4f8a-a9fe-21a5a6b428e9","Type":"ContainerStarted","Data":"ebc28877756ebf7357a2d3a230972d1a059bffa2cb7314187f02786d10711f75"} Oct 31 00:02:22.939192 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:22.938797 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09732b29-8wg8w" event={"ID":"5ed957ff-0566-4f8a-a9fe-21a5a6b428e9","Type":"ContainerStarted","Data":"a4869825affbceef5c6121fd7d357dbe90b1b108f57e96a46d07d11511d1983e"} Oct 31 00:02:23.060956 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:23.060906 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09732b29-8wg8w" podStartSLOduration=2.060894284 podStartE2EDuration="2.060894284s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:02:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:02:23.059267248 +0000 UTC m=+1256.095598070" watchObservedRunningTime="2026-10-31 00:02:23.060894284 +0000 UTC m=+1256.097225115" Oct 31 00:02:23.952975 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:23.952939 4927 generic.go:356] "Generic (PLEG): container finished" podID="5ed957ff-0566-4f8a-a9fe-21a5a6b428e9" containerID="ebc28877756ebf7357a2d3a230972d1a059bffa2cb7314187f02786d10711f75" exitCode=0 Oct 31 00:02:23.953453 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:23.953025 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09732b29-8wg8w" event={"ID":"5ed957ff-0566-4f8a-a9fe-21a5a6b428e9","Type":"ContainerDied","Data":"ebc28877756ebf7357a2d3a230972d1a059bffa2cb7314187f02786d10711f75"} Oct 31 00:02:24.173414 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:24.173382 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:02:24.368963 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:24.368881 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:02:25.138282 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:25.138254 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09732b29-8wg8w" Oct 31 00:02:25.294448 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:25.294368 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ed957ff-0566-4f8a-a9fe-21a5a6b428e9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ed957ff-0566-4f8a-a9fe-21a5a6b428e9-maas-api-openshift-ca\") pod \"5ed957ff-0566-4f8a-a9fe-21a5a6b428e9\" (UID: \"5ed957ff-0566-4f8a-a9fe-21a5a6b428e9\") " Oct 31 00:02:25.294654 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:25.294634 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ed957ff-0566-4f8a-a9fe-21a5a6b428e9-kube-api-access-bjtrf\" (UniqueName: \"kubernetes.io/projected/5ed957ff-0566-4f8a-a9fe-21a5a6b428e9-kube-api-access-bjtrf\") pod \"5ed957ff-0566-4f8a-a9fe-21a5a6b428e9\" (UID: \"5ed957ff-0566-4f8a-a9fe-21a5a6b428e9\") " Oct 31 00:02:25.294717 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:25.294681 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5ed957ff-0566-4f8a-a9fe-21a5a6b428e9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ed957ff-0566-4f8a-a9fe-21a5a6b428e9-maas-api-xks-ca\") pod \"5ed957ff-0566-4f8a-a9fe-21a5a6b428e9\" (UID: \"5ed957ff-0566-4f8a-a9fe-21a5a6b428e9\") " Oct 31 00:02:25.294717 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:25.294694 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ed957ff-0566-4f8a-a9fe-21a5a6b428e9-maas-api-openshift-ca" pod "5ed957ff-0566-4f8a-a9fe-21a5a6b428e9" (UID: "5ed957ff-0566-4f8a-a9fe-21a5a6b428e9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:02:25.295452 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:25.295426 4927 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ed957ff-0566-4f8a-a9fe-21a5a6b428e9-maas-api-openshift-ca\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:02:25.297238 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:25.297210 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ed957ff-0566-4f8a-a9fe-21a5a6b428e9-kube-api-access-bjtrf" pod "5ed957ff-0566-4f8a-a9fe-21a5a6b428e9" (UID: "5ed957ff-0566-4f8a-a9fe-21a5a6b428e9"). InnerVolumeSpecName "kube-api-access-bjtrf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:02:25.298460 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:25.298430 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ed957ff-0566-4f8a-a9fe-21a5a6b428e9-maas-api-xks-ca" pod "5ed957ff-0566-4f8a-a9fe-21a5a6b428e9" (UID: "5ed957ff-0566-4f8a-a9fe-21a5a6b428e9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:02:25.396910 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:25.396872 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bjtrf\" (UniqueName: \"kubernetes.io/projected/5ed957ff-0566-4f8a-a9fe-21a5a6b428e9-kube-api-access-bjtrf\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:02:25.396910 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:25.396906 4927 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ed957ff-0566-4f8a-a9fe-21a5a6b428e9-maas-api-xks-ca\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:02:25.977916 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:25.977867 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09732b29-8wg8w" event={"ID":"5ed957ff-0566-4f8a-a9fe-21a5a6b428e9","Type":"ContainerDied","Data":"a4869825affbceef5c6121fd7d357dbe90b1b108f57e96a46d07d11511d1983e"} Oct 31 00:02:25.978097 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:25.977980 4927 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a4869825affbceef5c6121fd7d357dbe90b1b108f57e96a46d07d11511d1983e" Oct 31 00:02:25.978097 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:25.978065 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09732b29-8wg8w" Oct 31 00:02:26.053499 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:26.053466 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b9c40b0e-8lmzb"] Oct 31 00:02:26.097666 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:26.097624 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b9c40b0e-8lmzb"] Oct 31 00:02:26.135988 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:26.135954 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01559921-kq4sn"] Oct 31 00:02:26.136662 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:26.136641 4927 cpu_manager.go:414] "containerMap: removing container" podUID="5ed957ff-0566-4f8a-a9fe-21a5a6b428e9" containerName="revoke-keys" Oct 31 00:02:26.136662 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:26.136658 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ed957ff-0566-4f8a-a9fe-21a5a6b428e9" containerName="revoke-keys" Oct 31 00:02:26.136849 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:26.136782 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ed957ff-0566-4f8a-a9fe-21a5a6b428e9" containerName="revoke-keys" Oct 31 00:02:26.140645 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:26.140624 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01559921-kq4sn" Oct 31 00:02:26.188584 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:26.188544 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:02:26.194777 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:26.194751 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:02:26.194929 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:26.194871 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4fdzs\"" Oct 31 00:02:26.220778 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:26.220747 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01559921-kq4sn"] Oct 31 00:02:26.314205 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:26.313607 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26b1d951-786a-4e89-8117-4158af8b15f1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-01559921-kq4sn\" (UID: \"26b1d951-786a-4e89-8117-4158af8b15f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01559921-kq4sn" Oct 31 00:02:26.314205 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:26.313701 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26b1d951-786a-4e89-8117-4158af8b15f1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-01559921-kq4sn\" (UID: \"26b1d951-786a-4e89-8117-4158af8b15f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01559921-kq4sn" Oct 31 00:02:26.314205 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:26.313811 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6jzzv\" (UniqueName: \"kubernetes.io/projected/26b1d951-786a-4e89-8117-4158af8b15f1-kube-api-access-6jzzv\") pod \"maas-api-revoke-sub-01559921-kq4sn\" (UID: \"26b1d951-786a-4e89-8117-4158af8b15f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01559921-kq4sn" Oct 31 00:02:26.415307 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:26.415270 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26b1d951-786a-4e89-8117-4158af8b15f1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-01559921-kq4sn\" (UID: \"26b1d951-786a-4e89-8117-4158af8b15f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01559921-kq4sn" Oct 31 00:02:26.415468 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:26.415348 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26b1d951-786a-4e89-8117-4158af8b15f1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-01559921-kq4sn\" (UID: \"26b1d951-786a-4e89-8117-4158af8b15f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01559921-kq4sn" Oct 31 00:02:26.415468 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:26.415459 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jzzv\" (UniqueName: \"kubernetes.io/projected/26b1d951-786a-4e89-8117-4158af8b15f1-kube-api-access-6jzzv\") pod \"maas-api-revoke-sub-01559921-kq4sn\" (UID: \"26b1d951-786a-4e89-8117-4158af8b15f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01559921-kq4sn" Oct 31 00:02:26.416065 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:26.416040 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26b1d951-786a-4e89-8117-4158af8b15f1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-01559921-kq4sn\" (UID: \"26b1d951-786a-4e89-8117-4158af8b15f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01559921-kq4sn" Oct 31 00:02:26.417967 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:26.417942 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26b1d951-786a-4e89-8117-4158af8b15f1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-01559921-kq4sn\" (UID: \"26b1d951-786a-4e89-8117-4158af8b15f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01559921-kq4sn" Oct 31 00:02:26.478261 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:26.478226 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6jzzv\" (UniqueName: \"kubernetes.io/projected/26b1d951-786a-4e89-8117-4158af8b15f1-kube-api-access-6jzzv\") pod \"maas-api-revoke-sub-01559921-kq4sn\" (UID: \"26b1d951-786a-4e89-8117-4158af8b15f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01559921-kq4sn" Oct 31 00:02:26.760090 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:26.760055 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01559921-kq4sn" Oct 31 00:02:27.090221 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:27.090143 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f4e75ff-6c3e-4d35-a73c-4b11c2ec2b7d" path="/var/lib/kubelet/pods/6f4e75ff-6c3e-4d35-a73c-4b11c2ec2b7d/volumes" Oct 31 00:02:27.160134 ip-10-0-141-180 kubenswrapper[4927]: W1031 00:02:27.160093 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26b1d951_786a_4e89_8117_4158af8b15f1.slice/crio-b30683571faf95a16bea548eec73c8a72a1f50e3649c9862321270eab0f08ffb WatchSource:0}: Error finding container b30683571faf95a16bea548eec73c8a72a1f50e3649c9862321270eab0f08ffb: Status 404 returned error can't find the container with id b30683571faf95a16bea548eec73c8a72a1f50e3649c9862321270eab0f08ffb Oct 31 00:02:27.161518 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:27.161490 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01559921-kq4sn"] Oct 31 00:02:27.788270 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:27.788179 4927 scope.go:122] "RemoveContainer" containerID="21816518bcb866b47e586e81fb59288492cc740e563e318142f0cacd50859734" Oct 31 00:02:27.803588 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:27.803564 4927 scope.go:122] "RemoveContainer" containerID="b7cc1b138ee189acede2bb5ab38d61da87fb37e3b1461cec887402c08b103cab" Oct 31 00:02:27.815279 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:27.815255 4927 scope.go:122] "RemoveContainer" containerID="6e986a43dd884073b95d832fd7f709546742cb06280a1c11e76bcf8f0789e02f" Oct 31 00:02:27.825171 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:27.825149 4927 scope.go:122] "RemoveContainer" containerID="4e9c7b64f8c0d9d6856d9c2c56b270022e186a136fc71417eedb574253b51145" Oct 31 00:02:27.999810 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:27.999768 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01559921-kq4sn" event={"ID":"26b1d951-786a-4e89-8117-4158af8b15f1","Type":"ContainerStarted","Data":"6d9d2bbef7fd0732a1f1728da6ff740e00762e63e0c5da1eec8a43e5c582d796"} Oct 31 00:02:28.000009 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:27.999831 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01559921-kq4sn" event={"ID":"26b1d951-786a-4e89-8117-4158af8b15f1","Type":"ContainerStarted","Data":"b30683571faf95a16bea548eec73c8a72a1f50e3649c9862321270eab0f08ffb"} Oct 31 00:02:28.123013 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:28.122902 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01559921-kq4sn" podStartSLOduration=2.122885614 podStartE2EDuration="2.122885614s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:02:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:02:28.118546634 +0000 UTC m=+1261.154877488" watchObservedRunningTime="2026-10-31 00:02:28.122885614 +0000 UTC m=+1261.159216487" Oct 31 00:02:28.174762 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:28.174735 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-27170a5e-j56gh"] Oct 31 00:02:28.179040 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:28.179021 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27170a5e-j56gh" Oct 31 00:02:28.237400 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:28.237366 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-27170a5e-j56gh"] Oct 31 00:02:28.248850 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:28.248817 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5nqj4\" (UniqueName: \"kubernetes.io/projected/2c899441-013e-4ff6-a7a2-d8ea415215dd-kube-api-access-5nqj4\") pod \"maas-api-revoke-sub-27170a5e-j56gh\" (UID: \"2c899441-013e-4ff6-a7a2-d8ea415215dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27170a5e-j56gh" Oct 31 00:02:28.249001 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:28.248898 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c899441-013e-4ff6-a7a2-d8ea415215dd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-27170a5e-j56gh\" (UID: \"2c899441-013e-4ff6-a7a2-d8ea415215dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27170a5e-j56gh" Oct 31 00:02:28.249388 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:28.249193 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c899441-013e-4ff6-a7a2-d8ea415215dd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-27170a5e-j56gh\" (UID: \"2c899441-013e-4ff6-a7a2-d8ea415215dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27170a5e-j56gh" Oct 31 00:02:28.350137 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:28.350096 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c899441-013e-4ff6-a7a2-d8ea415215dd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-27170a5e-j56gh\" (UID: \"2c899441-013e-4ff6-a7a2-d8ea415215dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27170a5e-j56gh" Oct 31 00:02:28.350297 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:28.350183 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5nqj4\" (UniqueName: \"kubernetes.io/projected/2c899441-013e-4ff6-a7a2-d8ea415215dd-kube-api-access-5nqj4\") pod \"maas-api-revoke-sub-27170a5e-j56gh\" (UID: \"2c899441-013e-4ff6-a7a2-d8ea415215dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27170a5e-j56gh" Oct 31 00:02:28.350297 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:28.350245 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c899441-013e-4ff6-a7a2-d8ea415215dd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-27170a5e-j56gh\" (UID: \"2c899441-013e-4ff6-a7a2-d8ea415215dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27170a5e-j56gh" Oct 31 00:02:28.350876 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:28.350854 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c899441-013e-4ff6-a7a2-d8ea415215dd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-27170a5e-j56gh\" (UID: \"2c899441-013e-4ff6-a7a2-d8ea415215dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27170a5e-j56gh" Oct 31 00:02:28.352806 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:28.352771 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c899441-013e-4ff6-a7a2-d8ea415215dd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-27170a5e-j56gh\" (UID: \"2c899441-013e-4ff6-a7a2-d8ea415215dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27170a5e-j56gh" Oct 31 00:02:28.413129 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:28.413098 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5nqj4\" (UniqueName: \"kubernetes.io/projected/2c899441-013e-4ff6-a7a2-d8ea415215dd-kube-api-access-5nqj4\") pod \"maas-api-revoke-sub-27170a5e-j56gh\" (UID: \"2c899441-013e-4ff6-a7a2-d8ea415215dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27170a5e-j56gh" Oct 31 00:02:28.566379 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:28.566322 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27170a5e-j56gh" Oct 31 00:02:28.958906 ip-10-0-141-180 kubenswrapper[4927]: W1031 00:02:28.958877 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c899441_013e_4ff6_a7a2_d8ea415215dd.slice/crio-8b19d9c73d81c042c92958c59f7ad2cabe76a7df4bec53904cc603581065e605 WatchSource:0}: Error finding container 8b19d9c73d81c042c92958c59f7ad2cabe76a7df4bec53904cc603581065e605: Status 404 returned error can't find the container with id 8b19d9c73d81c042c92958c59f7ad2cabe76a7df4bec53904cc603581065e605 Oct 31 00:02:28.961248 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:28.961225 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-27170a5e-j56gh"] Oct 31 00:02:29.009536 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:29.009503 4927 generic.go:356] "Generic (PLEG): container finished" podID="26b1d951-786a-4e89-8117-4158af8b15f1" containerID="6d9d2bbef7fd0732a1f1728da6ff740e00762e63e0c5da1eec8a43e5c582d796" exitCode=0 Oct 31 00:02:29.009634 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:29.009590 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01559921-kq4sn" event={"ID":"26b1d951-786a-4e89-8117-4158af8b15f1","Type":"ContainerDied","Data":"6d9d2bbef7fd0732a1f1728da6ff740e00762e63e0c5da1eec8a43e5c582d796"} Oct 31 00:02:29.011149 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:29.011121 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27170a5e-j56gh" event={"ID":"2c899441-013e-4ff6-a7a2-d8ea415215dd","Type":"ContainerStarted","Data":"8b19d9c73d81c042c92958c59f7ad2cabe76a7df4bec53904cc603581065e605"} Oct 31 00:02:29.055483 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:29.055444 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c802f331-n5s86"] Oct 31 00:02:29.060588 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:29.060557 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c802f331-n5s86"] Oct 31 00:02:29.087647 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:29.087616 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d617fcff-5199-415f-9d97-2185aae73433" path="/var/lib/kubelet/pods/d617fcff-5199-415f-9d97-2185aae73433/volumes" Oct 31 00:02:29.939074 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:29.939039 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-khcqd"] Oct 31 00:02:29.943664 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:29.943642 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-khcqd" Oct 31 00:02:29.972180 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:29.972150 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b317f84-83ac-49aa-921e-0242889f2b4f-istiod-ca-cert\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-khcqd\" (UID: \"2b317f84-83ac-49aa-921e-0242889f2b4f\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-khcqd" Oct 31 00:02:29.972275 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:29.972186 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b317f84-83ac-49aa-921e-0242889f2b4f-workload-socket\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-khcqd\" (UID: \"2b317f84-83ac-49aa-921e-0242889f2b4f\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-khcqd" Oct 31 00:02:29.972275 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:29.972212 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b317f84-83ac-49aa-921e-0242889f2b4f-istio-data\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-khcqd\" (UID: \"2b317f84-83ac-49aa-921e-0242889f2b4f\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-khcqd" Oct 31 00:02:29.972400 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:29.972378 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b317f84-83ac-49aa-921e-0242889f2b4f-istio-token\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-khcqd\" (UID: \"2b317f84-83ac-49aa-921e-0242889f2b4f\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-khcqd" Oct 31 00:02:29.972448 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:29.972415 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b317f84-83ac-49aa-921e-0242889f2b4f-istio-envoy\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-khcqd\" (UID: \"2b317f84-83ac-49aa-921e-0242889f2b4f\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-khcqd" Oct 31 00:02:29.972551 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:29.972536 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b317f84-83ac-49aa-921e-0242889f2b4f-credential-socket\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-khcqd\" (UID: \"2b317f84-83ac-49aa-921e-0242889f2b4f\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-khcqd" Oct 31 00:02:29.972592 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:29.972569 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7xcn\" (UniqueName: \"kubernetes.io/projected/2b317f84-83ac-49aa-921e-0242889f2b4f-kube-api-access-f7xcn\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-khcqd\" (UID: \"2b317f84-83ac-49aa-921e-0242889f2b4f\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-khcqd" Oct 31 00:02:29.972650 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:29.972638 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b317f84-83ac-49aa-921e-0242889f2b4f-istio-podinfo\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-khcqd\" (UID: \"2b317f84-83ac-49aa-921e-0242889f2b4f\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-khcqd" Oct 31 00:02:29.972713 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:29.972702 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b317f84-83ac-49aa-921e-0242889f2b4f-workload-certs\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-khcqd\" (UID: \"2b317f84-83ac-49aa-921e-0242889f2b4f\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-khcqd" Oct 31 00:02:30.028136 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:30.028041 4927 generic.go:356] "Generic (PLEG): container finished" podID="2c899441-013e-4ff6-a7a2-d8ea415215dd" containerID="41ee9ee3779e3afe1a36fb2ec8d1a1df6e11a177c7fdde9fd8d2cc7012d11b8e" exitCode=0 Oct 31 00:02:30.028298 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:30.028155 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27170a5e-j56gh" event={"ID":"2c899441-013e-4ff6-a7a2-d8ea415215dd","Type":"ContainerDied","Data":"41ee9ee3779e3afe1a36fb2ec8d1a1df6e11a177c7fdde9fd8d2cc7012d11b8e"} Oct 31 00:02:30.055238 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:30.055197 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-khcqd"] Oct 31 00:02:30.075237 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:30.074326 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b317f84-83ac-49aa-921e-0242889f2b4f-istio-podinfo\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-khcqd\" (UID: \"2b317f84-83ac-49aa-921e-0242889f2b4f\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-khcqd" Oct 31 00:02:30.075237 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:30.074413 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b317f84-83ac-49aa-921e-0242889f2b4f-workload-certs\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-khcqd\" (UID: \"2b317f84-83ac-49aa-921e-0242889f2b4f\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-khcqd" Oct 31 00:02:30.075237 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:30.074475 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b317f84-83ac-49aa-921e-0242889f2b4f-istiod-ca-cert\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-khcqd\" (UID: \"2b317f84-83ac-49aa-921e-0242889f2b4f\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-khcqd" Oct 31 00:02:30.075237 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:30.074503 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b317f84-83ac-49aa-921e-0242889f2b4f-workload-socket\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-khcqd\" (UID: \"2b317f84-83ac-49aa-921e-0242889f2b4f\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-khcqd" Oct 31 00:02:30.075237 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:30.074530 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b317f84-83ac-49aa-921e-0242889f2b4f-istio-data\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-khcqd\" (UID: \"2b317f84-83ac-49aa-921e-0242889f2b4f\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-khcqd" Oct 31 00:02:30.075237 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:30.074578 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b317f84-83ac-49aa-921e-0242889f2b4f-istio-token\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-khcqd\" (UID: \"2b317f84-83ac-49aa-921e-0242889f2b4f\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-khcqd" Oct 31 00:02:30.075237 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:30.074600 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b317f84-83ac-49aa-921e-0242889f2b4f-istio-envoy\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-khcqd\" (UID: \"2b317f84-83ac-49aa-921e-0242889f2b4f\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-khcqd" Oct 31 00:02:30.075237 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:30.074683 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b317f84-83ac-49aa-921e-0242889f2b4f-credential-socket\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-khcqd\" (UID: \"2b317f84-83ac-49aa-921e-0242889f2b4f\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-khcqd" Oct 31 00:02:30.075237 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:30.074707 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7xcn\" (UniqueName: \"kubernetes.io/projected/2b317f84-83ac-49aa-921e-0242889f2b4f-kube-api-access-f7xcn\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-khcqd\" (UID: \"2b317f84-83ac-49aa-921e-0242889f2b4f\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-khcqd" Oct 31 00:02:30.075237 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:30.075173 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b317f84-83ac-49aa-921e-0242889f2b4f-workload-certs\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-khcqd\" (UID: \"2b317f84-83ac-49aa-921e-0242889f2b4f\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-khcqd" Oct 31 00:02:30.076315 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:30.075528 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b317f84-83ac-49aa-921e-0242889f2b4f-istio-data\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-khcqd\" (UID: \"2b317f84-83ac-49aa-921e-0242889f2b4f\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-khcqd" Oct 31 00:02:30.076315 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:30.076077 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b317f84-83ac-49aa-921e-0242889f2b4f-credential-socket\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-khcqd\" (UID: \"2b317f84-83ac-49aa-921e-0242889f2b4f\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-khcqd" Oct 31 00:02:30.076315 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:30.076145 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b317f84-83ac-49aa-921e-0242889f2b4f-workload-socket\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-khcqd\" (UID: \"2b317f84-83ac-49aa-921e-0242889f2b4f\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-khcqd" Oct 31 00:02:30.076513 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:30.076485 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b317f84-83ac-49aa-921e-0242889f2b4f-istiod-ca-cert\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-khcqd\" (UID: \"2b317f84-83ac-49aa-921e-0242889f2b4f\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-khcqd" Oct 31 00:02:30.077771 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:30.077745 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b317f84-83ac-49aa-921e-0242889f2b4f-istio-podinfo\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-khcqd\" (UID: \"2b317f84-83ac-49aa-921e-0242889f2b4f\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-khcqd" Oct 31 00:02:30.078621 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:30.078593 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b317f84-83ac-49aa-921e-0242889f2b4f-istio-envoy\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-khcqd\" (UID: \"2b317f84-83ac-49aa-921e-0242889f2b4f\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-khcqd" Oct 31 00:02:30.138529 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:30.138456 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b317f84-83ac-49aa-921e-0242889f2b4f-istio-token\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-khcqd\" (UID: \"2b317f84-83ac-49aa-921e-0242889f2b4f\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-khcqd" Oct 31 00:02:30.139658 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:30.139631 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7xcn\" (UniqueName: \"kubernetes.io/projected/2b317f84-83ac-49aa-921e-0242889f2b4f-kube-api-access-f7xcn\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-khcqd\" (UID: \"2b317f84-83ac-49aa-921e-0242889f2b4f\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-khcqd" Oct 31 00:02:30.255451 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:30.255419 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01559921-kq4sn" Oct 31 00:02:30.258502 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:30.258479 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-khcqd" Oct 31 00:02:30.278564 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:30.278504 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/26b1d951-786a-4e89-8117-4158af8b15f1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26b1d951-786a-4e89-8117-4158af8b15f1-maas-api-xks-ca\") pod \"26b1d951-786a-4e89-8117-4158af8b15f1\" (UID: \"26b1d951-786a-4e89-8117-4158af8b15f1\") " Oct 31 00:02:30.278765 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:30.278742 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/26b1d951-786a-4e89-8117-4158af8b15f1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26b1d951-786a-4e89-8117-4158af8b15f1-maas-api-openshift-ca\") pod \"26b1d951-786a-4e89-8117-4158af8b15f1\" (UID: \"26b1d951-786a-4e89-8117-4158af8b15f1\") " Oct 31 00:02:30.278899 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:30.278881 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26b1d951-786a-4e89-8117-4158af8b15f1-kube-api-access-6jzzv\" (UniqueName: \"kubernetes.io/projected/26b1d951-786a-4e89-8117-4158af8b15f1-kube-api-access-6jzzv\") pod \"26b1d951-786a-4e89-8117-4158af8b15f1\" (UID: \"26b1d951-786a-4e89-8117-4158af8b15f1\") " Oct 31 00:02:30.279115 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:30.279089 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26b1d951-786a-4e89-8117-4158af8b15f1-maas-api-openshift-ca" pod "26b1d951-786a-4e89-8117-4158af8b15f1" (UID: "26b1d951-786a-4e89-8117-4158af8b15f1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:02:30.279895 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:30.279867 4927 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26b1d951-786a-4e89-8117-4158af8b15f1-maas-api-openshift-ca\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:02:30.281177 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:30.281154 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26b1d951-786a-4e89-8117-4158af8b15f1-maas-api-xks-ca" pod "26b1d951-786a-4e89-8117-4158af8b15f1" (UID: "26b1d951-786a-4e89-8117-4158af8b15f1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:02:30.281493 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:30.281473 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26b1d951-786a-4e89-8117-4158af8b15f1-kube-api-access-6jzzv" pod "26b1d951-786a-4e89-8117-4158af8b15f1" (UID: "26b1d951-786a-4e89-8117-4158af8b15f1"). InnerVolumeSpecName "kube-api-access-6jzzv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:02:30.381283 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:30.381249 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6jzzv\" (UniqueName: \"kubernetes.io/projected/26b1d951-786a-4e89-8117-4158af8b15f1-kube-api-access-6jzzv\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:02:30.381283 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:30.381284 4927 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26b1d951-786a-4e89-8117-4158af8b15f1-maas-api-xks-ca\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:02:30.702224 ip-10-0-141-180 kubenswrapper[4927]: W1031 00:02:30.702190 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b317f84_83ac_49aa_921e_0242889f2b4f.slice/crio-186e18ef3a62a47297bb18293b16591f3c30e78e12fea1793ac95b1e7883e0da WatchSource:0}: Error finding container 186e18ef3a62a47297bb18293b16591f3c30e78e12fea1793ac95b1e7883e0da: Status 404 returned error can't find the container with id 186e18ef3a62a47297bb18293b16591f3c30e78e12fea1793ac95b1e7883e0da Oct 31 00:02:30.703927 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:30.703904 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-khcqd"] Oct 31 00:02:30.705201 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:30.705168 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:02:30.705256 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:30.705234 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:02:30.705288 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:30.705260 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:02:30.795257 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:30.795224 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:02:30.821157 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:30.821104 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8bt76"] Oct 31 00:02:30.822771 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:30.822726 4927 cpu_manager.go:414] "containerMap: removing container" podUID="26b1d951-786a-4e89-8117-4158af8b15f1" containerName="revoke-keys" Oct 31 00:02:30.822931 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:30.822915 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26b1d951-786a-4e89-8117-4158af8b15f1" containerName="revoke-keys" Oct 31 00:02:30.823563 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:30.823542 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="26b1d951-786a-4e89-8117-4158af8b15f1" containerName="revoke-keys" Oct 31 00:02:30.865367 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:30.865137 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8bt76" Oct 31 00:02:30.887295 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:30.887257 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8789e67e-5656-4479-9b4f-fdc746fcfd15-workload-certs\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8bt76\" (UID: \"8789e67e-5656-4479-9b4f-fdc746fcfd15\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8bt76" Oct 31 00:02:30.888817 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:30.887535 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8789e67e-5656-4479-9b4f-fdc746fcfd15-istio-data\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8bt76\" (UID: \"8789e67e-5656-4479-9b4f-fdc746fcfd15\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8bt76" Oct 31 00:02:30.888817 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:30.887584 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8789e67e-5656-4479-9b4f-fdc746fcfd15-workload-socket\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8bt76\" (UID: \"8789e67e-5656-4479-9b4f-fdc746fcfd15\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8bt76" Oct 31 00:02:30.888817 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:30.887670 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8789e67e-5656-4479-9b4f-fdc746fcfd15-credential-socket\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8bt76\" (UID: \"8789e67e-5656-4479-9b4f-fdc746fcfd15\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8bt76" Oct 31 00:02:30.888817 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:30.887732 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8789e67e-5656-4479-9b4f-fdc746fcfd15-istio-envoy\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8bt76\" (UID: \"8789e67e-5656-4479-9b4f-fdc746fcfd15\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8bt76" Oct 31 00:02:30.888817 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:30.887832 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8789e67e-5656-4479-9b4f-fdc746fcfd15-istio-token\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8bt76\" (UID: \"8789e67e-5656-4479-9b4f-fdc746fcfd15\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8bt76" Oct 31 00:02:30.888817 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:30.887867 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8789e67e-5656-4479-9b4f-fdc746fcfd15-istiod-ca-cert\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8bt76\" (UID: \"8789e67e-5656-4479-9b4f-fdc746fcfd15\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8bt76" Oct 31 00:02:30.888817 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:30.887913 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-92jrp\" (UniqueName: \"kubernetes.io/projected/8789e67e-5656-4479-9b4f-fdc746fcfd15-kube-api-access-92jrp\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8bt76\" (UID: \"8789e67e-5656-4479-9b4f-fdc746fcfd15\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8bt76" Oct 31 00:02:30.888817 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:30.888009 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8789e67e-5656-4479-9b4f-fdc746fcfd15-istio-podinfo\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8bt76\" (UID: \"8789e67e-5656-4479-9b4f-fdc746fcfd15\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8bt76" Oct 31 00:02:30.891187 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:30.889820 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-944gh"] Oct 31 00:02:30.891187 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:30.890043 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-944gh" podUID="91f11663-5773-418a-bde2-532ead3c628c" containerName="istio-proxy" containerID="cri-o://0d9a7b1bae0ff7417b190cdc75722d41470432042fdc361ecee591c45a550339" gracePeriod=30 Oct 31 00:02:30.976982 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:30.976904 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-wftsq"] Oct 31 00:02:31.001325 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:30.991641 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-wftsq" podUID="222f0528-7bb2-4803-91ed-cda98b5541dc" containerName="istio-proxy" containerID="cri-o://ce6d91159de4fcc3a26120c66b762524ee35b794fc7ce06ba5449b64309722f9" gracePeriod=30 Oct 31 00:02:31.001325 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:30.993405 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8789e67e-5656-4479-9b4f-fdc746fcfd15-credential-socket\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8bt76\" (UID: \"8789e67e-5656-4479-9b4f-fdc746fcfd15\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8bt76" Oct 31 00:02:31.001325 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:30.993513 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8789e67e-5656-4479-9b4f-fdc746fcfd15-istio-envoy\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8bt76\" (UID: \"8789e67e-5656-4479-9b4f-fdc746fcfd15\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8bt76" Oct 31 00:02:31.001325 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:30.993607 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8789e67e-5656-4479-9b4f-fdc746fcfd15-istio-token\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8bt76\" (UID: \"8789e67e-5656-4479-9b4f-fdc746fcfd15\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8bt76" Oct 31 00:02:31.001325 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:30.993645 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8789e67e-5656-4479-9b4f-fdc746fcfd15-istiod-ca-cert\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8bt76\" (UID: \"8789e67e-5656-4479-9b4f-fdc746fcfd15\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8bt76" Oct 31 00:02:31.001325 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:30.993694 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92jrp\" (UniqueName: \"kubernetes.io/projected/8789e67e-5656-4479-9b4f-fdc746fcfd15-kube-api-access-92jrp\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8bt76\" (UID: \"8789e67e-5656-4479-9b4f-fdc746fcfd15\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8bt76" Oct 31 00:02:31.001325 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:30.993781 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8789e67e-5656-4479-9b4f-fdc746fcfd15-istio-podinfo\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8bt76\" (UID: \"8789e67e-5656-4479-9b4f-fdc746fcfd15\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8bt76" Oct 31 00:02:31.001325 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:30.993864 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8789e67e-5656-4479-9b4f-fdc746fcfd15-workload-certs\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8bt76\" (UID: \"8789e67e-5656-4479-9b4f-fdc746fcfd15\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8bt76" Oct 31 00:02:31.001325 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:30.993921 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8789e67e-5656-4479-9b4f-fdc746fcfd15-istio-data\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8bt76\" (UID: \"8789e67e-5656-4479-9b4f-fdc746fcfd15\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8bt76" Oct 31 00:02:31.001325 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:30.993960 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8789e67e-5656-4479-9b4f-fdc746fcfd15-workload-socket\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8bt76\" (UID: \"8789e67e-5656-4479-9b4f-fdc746fcfd15\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8bt76" Oct 31 00:02:31.001325 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:30.995854 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8789e67e-5656-4479-9b4f-fdc746fcfd15-workload-socket\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8bt76\" (UID: \"8789e67e-5656-4479-9b4f-fdc746fcfd15\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8bt76" Oct 31 00:02:31.001325 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:30.996139 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8789e67e-5656-4479-9b4f-fdc746fcfd15-credential-socket\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8bt76\" (UID: \"8789e67e-5656-4479-9b4f-fdc746fcfd15\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8bt76" Oct 31 00:02:31.001325 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:30.998977 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8789e67e-5656-4479-9b4f-fdc746fcfd15-istiod-ca-cert\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8bt76\" (UID: \"8789e67e-5656-4479-9b4f-fdc746fcfd15\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8bt76" Oct 31 00:02:31.001325 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:30.999252 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8789e67e-5656-4479-9b4f-fdc746fcfd15-workload-certs\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8bt76\" (UID: \"8789e67e-5656-4479-9b4f-fdc746fcfd15\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8bt76" Oct 31 00:02:31.001325 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:31.000057 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8789e67e-5656-4479-9b4f-fdc746fcfd15-istio-data\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8bt76\" (UID: \"8789e67e-5656-4479-9b4f-fdc746fcfd15\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8bt76" Oct 31 00:02:31.084423 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:31.066386 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-xsgv7"] Oct 31 00:02:31.084423 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:31.066629 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-xsgv7" podUID="647695af-bfbe-41f4-b395-4d1ff1c0286f" containerName="istio-proxy" containerID="cri-o://e8af9c72a5830de22e6214c1545562b68ddf01f2d153b236bc34644acce466d6" gracePeriod=30 Oct 31 00:02:31.084423 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:31.070799 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-khcqd" event={"ID":"2b317f84-83ac-49aa-921e-0242889f2b4f","Type":"ContainerStarted","Data":"f2edac246d0fdf2378197ddfb3fb9eab46f7ea47415499c914edaee90412f22b"} Oct 31 00:02:31.084423 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:31.070871 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-khcqd" event={"ID":"2b317f84-83ac-49aa-921e-0242889f2b4f","Type":"ContainerStarted","Data":"186e18ef3a62a47297bb18293b16591f3c30e78e12fea1793ac95b1e7883e0da"} Oct 31 00:02:31.096512 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:31.096349 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-92jrp\" (UniqueName: \"kubernetes.io/projected/8789e67e-5656-4479-9b4f-fdc746fcfd15-kube-api-access-92jrp\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8bt76\" (UID: \"8789e67e-5656-4479-9b4f-fdc746fcfd15\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8bt76" Oct 31 00:02:31.096512 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:31.096430 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-xkbnw"] Oct 31 00:02:31.096512 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:31.096446 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8789e67e-5656-4479-9b4f-fdc746fcfd15-istio-token\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8bt76\" (UID: \"8789e67e-5656-4479-9b4f-fdc746fcfd15\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8bt76" Oct 31 00:02:31.097167 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:31.096986 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8789e67e-5656-4479-9b4f-fdc746fcfd15-istio-envoy\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8bt76\" (UID: \"8789e67e-5656-4479-9b4f-fdc746fcfd15\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8bt76" Oct 31 00:02:31.128884 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:31.126103 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8789e67e-5656-4479-9b4f-fdc746fcfd15-istio-podinfo\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8bt76\" (UID: \"8789e67e-5656-4479-9b4f-fdc746fcfd15\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8bt76" Oct 31 00:02:31.143591 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:31.143562 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-xkbnw" Oct 31 00:02:31.156569 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:31.156522 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01559921-kq4sn" Oct 31 00:02:31.204366 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:31.202227 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8bt76"] Oct 31 00:02:31.204366 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:31.202296 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01559921-kq4sn" event={"ID":"26b1d951-786a-4e89-8117-4158af8b15f1","Type":"ContainerDied","Data":"b30683571faf95a16bea548eec73c8a72a1f50e3649c9862321270eab0f08ffb"} Oct 31 00:02:31.204366 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:31.202351 4927 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b30683571faf95a16bea548eec73c8a72a1f50e3649c9862321270eab0f08ffb" Oct 31 00:02:31.204366 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:31.202369 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-xkbnw"] Oct 31 00:02:31.259707 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:31.259606 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-khcqd" Oct 31 00:02:31.261722 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:31.261694 4927 patch_prober.go:29] interesting pod/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-khcqd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" start-of-body= Oct 31 00:02:31.261970 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:31.261943 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-khcqd" podUID="2b317f84-83ac-49aa-921e-0242889f2b4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" Oct 31 00:02:31.297881 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:31.297839 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8bt76" Oct 31 00:02:31.318605 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:31.318570 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/469ac12b-4c77-403a-a190-af4037422211-credential-socket\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-xkbnw\" (UID: \"469ac12b-4c77-403a-a190-af4037422211\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-xkbnw" Oct 31 00:02:31.318862 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:31.318837 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/469ac12b-4c77-403a-a190-af4037422211-workload-certs\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-xkbnw\" (UID: \"469ac12b-4c77-403a-a190-af4037422211\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-xkbnw" Oct 31 00:02:31.319057 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:31.319042 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/469ac12b-4c77-403a-a190-af4037422211-workload-socket\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-xkbnw\" (UID: \"469ac12b-4c77-403a-a190-af4037422211\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-xkbnw" Oct 31 00:02:31.319345 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:31.319311 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/469ac12b-4c77-403a-a190-af4037422211-istio-envoy\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-xkbnw\" (UID: \"469ac12b-4c77-403a-a190-af4037422211\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-xkbnw" Oct 31 00:02:31.319468 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:31.319450 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/469ac12b-4c77-403a-a190-af4037422211-istio-podinfo\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-xkbnw\" (UID: \"469ac12b-4c77-403a-a190-af4037422211\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-xkbnw" Oct 31 00:02:31.319580 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:31.319566 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/469ac12b-4c77-403a-a190-af4037422211-istiod-ca-cert\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-xkbnw\" (UID: \"469ac12b-4c77-403a-a190-af4037422211\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-xkbnw" Oct 31 00:02:31.319702 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:31.319688 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/469ac12b-4c77-403a-a190-af4037422211-istio-token\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-xkbnw\" (UID: \"469ac12b-4c77-403a-a190-af4037422211\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-xkbnw" Oct 31 00:02:31.319903 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:31.319884 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/469ac12b-4c77-403a-a190-af4037422211-istio-data\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-xkbnw\" (UID: \"469ac12b-4c77-403a-a190-af4037422211\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-xkbnw" Oct 31 00:02:31.320015 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:31.320000 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rx6l7\" (UniqueName: \"kubernetes.io/projected/469ac12b-4c77-403a-a190-af4037422211-kube-api-access-rx6l7\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-xkbnw\" (UID: \"469ac12b-4c77-403a-a190-af4037422211\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-xkbnw" Oct 31 00:02:31.408536 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:31.406904 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27170a5e-j56gh" Oct 31 00:02:31.426413 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:31.424487 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2c899441-013e-4ff6-a7a2-d8ea415215dd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c899441-013e-4ff6-a7a2-d8ea415215dd-maas-api-xks-ca\") pod \"2c899441-013e-4ff6-a7a2-d8ea415215dd\" (UID: \"2c899441-013e-4ff6-a7a2-d8ea415215dd\") " Oct 31 00:02:31.426413 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:31.424730 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c899441-013e-4ff6-a7a2-d8ea415215dd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c899441-013e-4ff6-a7a2-d8ea415215dd-maas-api-openshift-ca\") pod \"2c899441-013e-4ff6-a7a2-d8ea415215dd\" (UID: \"2c899441-013e-4ff6-a7a2-d8ea415215dd\") " Oct 31 00:02:31.426413 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:31.424832 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c899441-013e-4ff6-a7a2-d8ea415215dd-kube-api-access-5nqj4\" (UniqueName: \"kubernetes.io/projected/2c899441-013e-4ff6-a7a2-d8ea415215dd-kube-api-access-5nqj4\") pod \"2c899441-013e-4ff6-a7a2-d8ea415215dd\" (UID: \"2c899441-013e-4ff6-a7a2-d8ea415215dd\") " Oct 31 00:02:31.426413 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:31.425031 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/469ac12b-4c77-403a-a190-af4037422211-istio-data\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-xkbnw\" (UID: \"469ac12b-4c77-403a-a190-af4037422211\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-xkbnw" Oct 31 00:02:31.426413 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:31.425068 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rx6l7\" (UniqueName: \"kubernetes.io/projected/469ac12b-4c77-403a-a190-af4037422211-kube-api-access-rx6l7\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-xkbnw\" (UID: \"469ac12b-4c77-403a-a190-af4037422211\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-xkbnw" Oct 31 00:02:31.426413 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:31.425241 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/469ac12b-4c77-403a-a190-af4037422211-credential-socket\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-xkbnw\" (UID: \"469ac12b-4c77-403a-a190-af4037422211\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-xkbnw" Oct 31 00:02:31.426413 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:31.425298 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/469ac12b-4c77-403a-a190-af4037422211-workload-certs\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-xkbnw\" (UID: \"469ac12b-4c77-403a-a190-af4037422211\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-xkbnw" Oct 31 00:02:31.426413 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:31.425400 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/469ac12b-4c77-403a-a190-af4037422211-workload-socket\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-xkbnw\" (UID: \"469ac12b-4c77-403a-a190-af4037422211\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-xkbnw" Oct 31 00:02:31.426413 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:31.425537 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/469ac12b-4c77-403a-a190-af4037422211-istio-envoy\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-xkbnw\" (UID: \"469ac12b-4c77-403a-a190-af4037422211\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-xkbnw" Oct 31 00:02:31.426413 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:31.425566 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/469ac12b-4c77-403a-a190-af4037422211-istio-podinfo\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-xkbnw\" (UID: \"469ac12b-4c77-403a-a190-af4037422211\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-xkbnw" Oct 31 00:02:31.426413 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:31.425602 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/469ac12b-4c77-403a-a190-af4037422211-istiod-ca-cert\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-xkbnw\" (UID: \"469ac12b-4c77-403a-a190-af4037422211\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-xkbnw" Oct 31 00:02:31.426413 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:31.425649 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/469ac12b-4c77-403a-a190-af4037422211-istio-token\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-xkbnw\" (UID: \"469ac12b-4c77-403a-a190-af4037422211\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-xkbnw" Oct 31 00:02:31.427805 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:31.427735 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/469ac12b-4c77-403a-a190-af4037422211-credential-socket\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-xkbnw\" (UID: \"469ac12b-4c77-403a-a190-af4037422211\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-xkbnw" Oct 31 00:02:31.428422 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:31.428371 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c899441-013e-4ff6-a7a2-d8ea415215dd-maas-api-openshift-ca" pod "2c899441-013e-4ff6-a7a2-d8ea415215dd" (UID: "2c899441-013e-4ff6-a7a2-d8ea415215dd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:02:31.429669 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:31.429266 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/469ac12b-4c77-403a-a190-af4037422211-istio-data\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-xkbnw\" (UID: \"469ac12b-4c77-403a-a190-af4037422211\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-xkbnw" Oct 31 00:02:31.430453 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:31.430431 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/469ac12b-4c77-403a-a190-af4037422211-workload-certs\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-xkbnw\" (UID: \"469ac12b-4c77-403a-a190-af4037422211\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-xkbnw" Oct 31 00:02:31.431219 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:31.431194 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/469ac12b-4c77-403a-a190-af4037422211-istiod-ca-cert\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-xkbnw\" (UID: \"469ac12b-4c77-403a-a190-af4037422211\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-xkbnw" Oct 31 00:02:31.431937 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:31.431325 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/469ac12b-4c77-403a-a190-af4037422211-workload-socket\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-xkbnw\" (UID: \"469ac12b-4c77-403a-a190-af4037422211\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-xkbnw" Oct 31 00:02:31.431937 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:31.431673 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c899441-013e-4ff6-a7a2-d8ea415215dd-kube-api-access-5nqj4" pod "2c899441-013e-4ff6-a7a2-d8ea415215dd" (UID: "2c899441-013e-4ff6-a7a2-d8ea415215dd"). InnerVolumeSpecName "kube-api-access-5nqj4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:02:31.433656 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:31.433611 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/469ac12b-4c77-403a-a190-af4037422211-istio-podinfo\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-xkbnw\" (UID: \"469ac12b-4c77-403a-a190-af4037422211\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-xkbnw" Oct 31 00:02:31.434173 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:31.434144 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/469ac12b-4c77-403a-a190-af4037422211-istio-envoy\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-xkbnw\" (UID: \"469ac12b-4c77-403a-a190-af4037422211\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-xkbnw" Oct 31 00:02:31.438844 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:31.437464 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c899441-013e-4ff6-a7a2-d8ea415215dd-maas-api-xks-ca" pod "2c899441-013e-4ff6-a7a2-d8ea415215dd" (UID: "2c899441-013e-4ff6-a7a2-d8ea415215dd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:02:31.488182 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:31.488140 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/469ac12b-4c77-403a-a190-af4037422211-istio-token\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-xkbnw\" (UID: \"469ac12b-4c77-403a-a190-af4037422211\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-xkbnw" Oct 31 00:02:31.491370 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:31.491342 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rx6l7\" (UniqueName: \"kubernetes.io/projected/469ac12b-4c77-403a-a190-af4037422211-kube-api-access-rx6l7\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-xkbnw\" (UID: \"469ac12b-4c77-403a-a190-af4037422211\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-xkbnw" Oct 31 00:02:31.527809 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:31.527719 4927 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c899441-013e-4ff6-a7a2-d8ea415215dd-maas-api-xks-ca\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:02:31.527809 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:31.527758 4927 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c899441-013e-4ff6-a7a2-d8ea415215dd-maas-api-openshift-ca\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:02:31.527809 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:31.527774 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5nqj4\" (UniqueName: \"kubernetes.io/projected/2c899441-013e-4ff6-a7a2-d8ea415215dd-kube-api-access-5nqj4\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:02:31.529697 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:31.529668 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-xkbnw" Oct 31 00:02:31.746636 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:31.746588 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8bt76"] Oct 31 00:02:31.746636 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:31.746615 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:02:31.746809 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:31.746696 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:02:31.746809 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:31.746744 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:02:31.750642 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:31.750560 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-khcqd" podStartSLOduration=2.7505427559999998 podStartE2EDuration="2.750542756s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:02:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:02:31.742025865 +0000 UTC m=+1264.778356689" watchObservedRunningTime="2026-10-31 00:02:31.750542756 +0000 UTC m=+1264.786873581" Oct 31 00:02:31.968271 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:31.968225 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:02:31.968782 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:31.968317 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:02:31.968782 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:31.968380 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:02:31.969636 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:31.969571 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-xkbnw"] Oct 31 00:02:32.178658 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:32.178626 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27170a5e-j56gh" Oct 31 00:02:32.180309 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:32.178689 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27170a5e-j56gh" event={"ID":"2c899441-013e-4ff6-a7a2-d8ea415215dd","Type":"ContainerDied","Data":"8b19d9c73d81c042c92958c59f7ad2cabe76a7df4bec53904cc603581065e605"} Oct 31 00:02:32.180309 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:32.180274 4927 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8b19d9c73d81c042c92958c59f7ad2cabe76a7df4bec53904cc603581065e605" Oct 31 00:02:32.182884 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:32.182859 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8bt76" event={"ID":"8789e67e-5656-4479-9b4f-fdc746fcfd15","Type":"ContainerStarted","Data":"648dcd6d004af2c4231082241023009a70d09509111860829d6171775e879774"} Oct 31 00:02:32.183015 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:32.182903 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8bt76" event={"ID":"8789e67e-5656-4479-9b4f-fdc746fcfd15","Type":"ContainerStarted","Data":"e0b61239a82c4d05a730737da3bd3c89c9f9c1d5b1f08da54ca7ff370ca87687"} Oct 31 00:02:32.185459 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:32.185431 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-xkbnw" event={"ID":"469ac12b-4c77-403a-a190-af4037422211","Type":"ContainerStarted","Data":"4177f70e567ff945d5d30decef1ae28787d68a5859d631971494bd201d921f06"} Oct 31 00:02:32.185568 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:32.185490 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-xkbnw" event={"ID":"469ac12b-4c77-403a-a190-af4037422211","Type":"ContainerStarted","Data":"77d2a04204ae0f8f966370e9bc266cbee6d66c42043bb78e13b701d4c499b7fb"} Oct 31 00:02:32.259451 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:32.258977 4927 patch_prober.go:29] interesting pod/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-khcqd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" start-of-body= Oct 31 00:02:32.259451 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:32.259033 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-khcqd" podUID="2b317f84-83ac-49aa-921e-0242889f2b4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" Oct 31 00:02:32.298518 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:32.298484 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8bt76" Oct 31 00:02:32.299909 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:32.299887 4927 patch_prober.go:29] interesting pod/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8bt76 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body= Oct 31 00:02:32.299988 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:32.299941 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8bt76" podUID="8789e67e-5656-4479-9b4f-fdc746fcfd15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" Oct 31 00:02:32.363463 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:32.361078 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8bt76" podStartSLOduration=2.361058717 podStartE2EDuration="2.361058717s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:02:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:02:32.33115168 +0000 UTC m=+1265.367482503" watchObservedRunningTime="2026-10-31 00:02:32.361058717 +0000 UTC m=+1265.397389552" Oct 31 00:02:32.544146 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:32.544103 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-xkbnw" Oct 31 00:02:32.546384 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:32.546267 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-xkbnw" podStartSLOduration=2.546247427 podStartE2EDuration="2.546247427s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:02:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:02:32.525552224 +0000 UTC m=+1265.561883075" watchObservedRunningTime="2026-10-31 00:02:32.546247427 +0000 UTC m=+1265.582578253" Oct 31 00:02:32.557630 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:32.555547 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-270eef34-4b2nh"] Oct 31 00:02:32.557630 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:32.556573 4927 cpu_manager.go:414] "containerMap: removing container" podUID="2c899441-013e-4ff6-a7a2-d8ea415215dd" containerName="revoke-keys" Oct 31 00:02:32.557630 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:32.556589 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c899441-013e-4ff6-a7a2-d8ea415215dd" containerName="revoke-keys" Oct 31 00:02:32.557630 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:32.556784 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c899441-013e-4ff6-a7a2-d8ea415215dd" containerName="revoke-keys" Oct 31 00:02:32.560729 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:32.560708 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-270eef34-4b2nh" Oct 31 00:02:32.574360 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:32.573871 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32855a90-baa7-4b9a-b1cb-c97ab9ca8e7d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-270eef34-4b2nh\" (UID: \"32855a90-baa7-4b9a-b1cb-c97ab9ca8e7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-270eef34-4b2nh" Oct 31 00:02:32.574360 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:32.574116 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nl2s7\" (UniqueName: \"kubernetes.io/projected/32855a90-baa7-4b9a-b1cb-c97ab9ca8e7d-kube-api-access-nl2s7\") pod \"maas-api-revoke-sub-270eef34-4b2nh\" (UID: \"32855a90-baa7-4b9a-b1cb-c97ab9ca8e7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-270eef34-4b2nh" Oct 31 00:02:32.574360 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:32.574171 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32855a90-baa7-4b9a-b1cb-c97ab9ca8e7d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-270eef34-4b2nh\" (UID: \"32855a90-baa7-4b9a-b1cb-c97ab9ca8e7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-270eef34-4b2nh" Oct 31 00:02:32.639407 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:32.629676 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4fdzs\"" Oct 31 00:02:32.649406 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:32.647579 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:02:32.650166 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:32.650138 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-270eef34-4b2nh"] Oct 31 00:02:32.676120 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:32.676087 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nl2s7\" (UniqueName: \"kubernetes.io/projected/32855a90-baa7-4b9a-b1cb-c97ab9ca8e7d-kube-api-access-nl2s7\") pod \"maas-api-revoke-sub-270eef34-4b2nh\" (UID: \"32855a90-baa7-4b9a-b1cb-c97ab9ca8e7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-270eef34-4b2nh" Oct 31 00:02:32.676284 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:32.676146 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32855a90-baa7-4b9a-b1cb-c97ab9ca8e7d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-270eef34-4b2nh\" (UID: \"32855a90-baa7-4b9a-b1cb-c97ab9ca8e7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-270eef34-4b2nh" Oct 31 00:02:32.676284 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:32.676274 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32855a90-baa7-4b9a-b1cb-c97ab9ca8e7d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-270eef34-4b2nh\" (UID: \"32855a90-baa7-4b9a-b1cb-c97ab9ca8e7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-270eef34-4b2nh" Oct 31 00:02:32.677605 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:32.677574 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32855a90-baa7-4b9a-b1cb-c97ab9ca8e7d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-270eef34-4b2nh\" (UID: \"32855a90-baa7-4b9a-b1cb-c97ab9ca8e7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-270eef34-4b2nh" Oct 31 00:02:32.678927 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:32.678908 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32855a90-baa7-4b9a-b1cb-c97ab9ca8e7d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-270eef34-4b2nh\" (UID: \"32855a90-baa7-4b9a-b1cb-c97ab9ca8e7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-270eef34-4b2nh" Oct 31 00:02:32.759763 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:32.759726 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nl2s7\" (UniqueName: \"kubernetes.io/projected/32855a90-baa7-4b9a-b1cb-c97ab9ca8e7d-kube-api-access-nl2s7\") pod \"maas-api-revoke-sub-270eef34-4b2nh\" (UID: \"32855a90-baa7-4b9a-b1cb-c97ab9ca8e7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-270eef34-4b2nh" Oct 31 00:02:32.883210 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:32.882732 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-270eef34-4b2nh" Oct 31 00:02:33.263316 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:33.263220 4927 patch_prober.go:29] interesting pod/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-khcqd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" start-of-body= Oct 31 00:02:33.263935 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:33.263885 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-khcqd" podUID="2b317f84-83ac-49aa-921e-0242889f2b4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" Oct 31 00:02:33.416459 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:33.400080 4927 patch_prober.go:29] interesting pod/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8bt76 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body= Oct 31 00:02:33.416459 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:33.400147 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8bt76" podUID="8789e67e-5656-4479-9b4f-fdc746fcfd15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" Oct 31 00:02:33.603511 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:33.603404 4927 patch_prober.go:29] interesting pod/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-xkbnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body= Oct 31 00:02:33.603794 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:33.603756 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-xkbnw" podUID="469ac12b-4c77-403a-a190-af4037422211" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" Oct 31 00:02:34.286357 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:34.284472 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fe01fa/distinct-2-w5-fe01fa-kserve-79b99966d8-dkhzd" podUID="d5c1c9e8-ce47-4868-860d-b8413ff74ab5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.111:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 00:02:34.341121 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:34.341079 4927 patch_prober.go:29] interesting pod/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-khcqd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" start-of-body= Oct 31 00:02:34.341322 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:34.341294 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-khcqd" podUID="2b317f84-83ac-49aa-921e-0242889f2b4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" Oct 31 00:02:34.355403 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:34.347661 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b1844f1a-t6tp6"] Oct 31 00:02:34.451158 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:34.407631 4927 patch_prober.go:29] interesting pod/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8bt76 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body= Oct 31 00:02:34.451362 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:34.451189 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8bt76" podUID="8789e67e-5656-4479-9b4f-fdc746fcfd15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" Oct 31 00:02:34.468467 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:34.468425 4927 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-986d37/premium-w3-986d37-kserve-5c7c7757fb-r99j7" podUID="f4e38df7-355f-4b47-a0b5-7e0b907f3326" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:02:34.488184 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:34.488130 4927 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-fe01fa/distinct-2-w5-fe01fa-kserve-79b99966d8-dkhzd" podUID="d5c1c9e8-ce47-4868-860d-b8413ff74ab5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.111:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:02:34.542886 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:34.523802 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b1844f1a-t6tp6"] Oct 31 00:02:34.570780 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:34.556986 4927 patch_prober.go:29] interesting pod/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-xkbnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body= Oct 31 00:02:34.570780 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:34.557068 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-xkbnw" podUID="469ac12b-4c77-403a-a190-af4037422211" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" Oct 31 00:02:34.633360 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:34.630796 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-27e82b/distinct-w2-27e82b-kserve-75f475fcd9-mccsz" podUID="4ba8ced4-e93f-46ac-87c5-3d0b7fadc02d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:02:34.736223 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:34.736188 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-270eef34-4b2nh"] Oct 31 00:02:34.876435 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:34.876395 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bdd5efd5-9957g"] Oct 31 00:02:34.906587 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:34.904745 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-270eef34-4b2nh" event={"ID":"32855a90-baa7-4b9a-b1cb-c97ab9ca8e7d","Type":"ContainerStarted","Data":"c62e1e74c49679914f18fc383f043a25e59a10236350a6661ea937edef9ffe7d"} Oct 31 00:02:34.906587 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:34.904860 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdd5efd5-9957g" Oct 31 00:02:35.037356 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:35.021772 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bdd5efd5-9957g"] Oct 31 00:02:35.126816 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:35.064900 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4baf4a61-aae7-4f15-9b1a-fff0ed014c9b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bdd5efd5-9957g\" (UID: \"4baf4a61-aae7-4f15-9b1a-fff0ed014c9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdd5efd5-9957g" Oct 31 00:02:35.127145 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:35.127121 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4baf4a61-aae7-4f15-9b1a-fff0ed014c9b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bdd5efd5-9957g\" (UID: \"4baf4a61-aae7-4f15-9b1a-fff0ed014c9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdd5efd5-9957g" Oct 31 00:02:35.127603 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:35.127577 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-558wh\" (UniqueName: \"kubernetes.io/projected/4baf4a61-aae7-4f15-9b1a-fff0ed014c9b-kube-api-access-558wh\") pod \"maas-api-revoke-sub-bdd5efd5-9957g\" (UID: \"4baf4a61-aae7-4f15-9b1a-fff0ed014c9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdd5efd5-9957g" Oct 31 00:02:35.263362 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:35.256619 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4baf4a61-aae7-4f15-9b1a-fff0ed014c9b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bdd5efd5-9957g\" (UID: \"4baf4a61-aae7-4f15-9b1a-fff0ed014c9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdd5efd5-9957g" Oct 31 00:02:35.263362 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:35.256735 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4baf4a61-aae7-4f15-9b1a-fff0ed014c9b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bdd5efd5-9957g\" (UID: \"4baf4a61-aae7-4f15-9b1a-fff0ed014c9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdd5efd5-9957g" Oct 31 00:02:35.263362 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:35.256907 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-558wh\" (UniqueName: \"kubernetes.io/projected/4baf4a61-aae7-4f15-9b1a-fff0ed014c9b-kube-api-access-558wh\") pod \"maas-api-revoke-sub-bdd5efd5-9957g\" (UID: \"4baf4a61-aae7-4f15-9b1a-fff0ed014c9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdd5efd5-9957g" Oct 31 00:02:35.263362 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:35.258195 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4baf4a61-aae7-4f15-9b1a-fff0ed014c9b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bdd5efd5-9957g\" (UID: \"4baf4a61-aae7-4f15-9b1a-fff0ed014c9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdd5efd5-9957g" Oct 31 00:02:35.263362 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:35.258716 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af1791d5-2960-42bd-95a8-6f6b0ef78e93" path="/var/lib/kubelet/pods/af1791d5-2960-42bd-95a8-6f6b0ef78e93/volumes" Oct 31 00:02:35.316385 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:35.315801 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4baf4a61-aae7-4f15-9b1a-fff0ed014c9b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bdd5efd5-9957g\" (UID: \"4baf4a61-aae7-4f15-9b1a-fff0ed014c9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdd5efd5-9957g" Oct 31 00:02:35.342355 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:35.336448 4927 patch_prober.go:29] interesting pod/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-khcqd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" start-of-body= Oct 31 00:02:35.342355 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:35.336501 4927 patch_prober.go:29] interesting pod/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8bt76 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body= Oct 31 00:02:35.342355 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:35.336509 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-khcqd" podUID="2b317f84-83ac-49aa-921e-0242889f2b4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" Oct 31 00:02:35.342355 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:35.336547 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8bt76" podUID="8789e67e-5656-4479-9b4f-fdc746fcfd15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" Oct 31 00:02:35.391149 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:35.391067 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-558wh\" (UniqueName: \"kubernetes.io/projected/4baf4a61-aae7-4f15-9b1a-fff0ed014c9b-kube-api-access-558wh\") pod \"maas-api-revoke-sub-bdd5efd5-9957g\" (UID: \"4baf4a61-aae7-4f15-9b1a-fff0ed014c9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdd5efd5-9957g" Oct 31 00:02:35.434555 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:35.434524 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdd5efd5-9957g" Oct 31 00:02:35.574086 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:35.574049 4927 patch_prober.go:29] interesting pod/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-xkbnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body= Oct 31 00:02:35.689464 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:35.689416 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-xkbnw" podUID="469ac12b-4c77-403a-a190-af4037422211" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" Oct 31 00:02:36.044357 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:36.041097 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:02:36.328416 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:36.305784 4927 patch_prober.go:29] interesting pod/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8bt76 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body= Oct 31 00:02:36.328416 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:36.305848 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8bt76" podUID="8789e67e-5656-4479-9b4f-fdc746fcfd15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" Oct 31 00:02:36.328416 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:36.307088 4927 patch_prober.go:29] interesting pod/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-khcqd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" start-of-body= Oct 31 00:02:36.328416 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:36.307133 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-khcqd" podUID="2b317f84-83ac-49aa-921e-0242889f2b4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" Oct 31 00:02:36.566356 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:36.538748 4927 patch_prober.go:29] interesting pod/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-xkbnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body= Oct 31 00:02:36.567562 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:36.567523 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-xkbnw" podUID="469ac12b-4c77-403a-a190-af4037422211" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" Oct 31 00:02:36.567757 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:36.545781 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-270eef34-4b2nh" event={"ID":"32855a90-baa7-4b9a-b1cb-c97ab9ca8e7d","Type":"ContainerStarted","Data":"c42659543526c5fb508a31f7433fdb5de3fa9b2afdee03a97d42a92b63faceac"} Oct 31 00:02:37.279506 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:37.279467 4927 patch_prober.go:29] interesting pod/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-khcqd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" start-of-body= Oct 31 00:02:37.279742 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:37.279715 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-khcqd" podUID="2b317f84-83ac-49aa-921e-0242889f2b4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" Oct 31 00:02:37.437359 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:37.437057 4927 patch_prober.go:29] interesting pod/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8bt76 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body= Oct 31 00:02:37.437359 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:37.437126 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8bt76" podUID="8789e67e-5656-4479-9b4f-fdc746fcfd15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" Oct 31 00:02:37.465714 ip-10-0-141-180 kubenswrapper[4927]: W1031 00:02:37.462684 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4baf4a61_aae7_4f15_9b1a_fff0ed014c9b.slice/crio-2af03f9c2e4ff03a0ec534fadfefba8f0bc60905f81025ecc713319968808327 WatchSource:0}: Error finding container 2af03f9c2e4ff03a0ec534fadfefba8f0bc60905f81025ecc713319968808327: Status 404 returned error can't find the container with id 2af03f9c2e4ff03a0ec534fadfefba8f0bc60905f81025ecc713319968808327 Oct 31 00:02:37.750208 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:37.702502 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-270eef34-4b2nh" podStartSLOduration=5.702473218 podStartE2EDuration="5.702473218s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:02:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:02:36.79170356 +0000 UTC m=+1269.828034402" watchObservedRunningTime="2026-10-31 00:02:37.702473218 +0000 UTC m=+1270.738804041" Oct 31 00:02:37.778384 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:37.757559 4927 patch_prober.go:29] interesting pod/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-xkbnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body= Oct 31 00:02:37.812368 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:37.802109 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bdd5efd5-9957g"] Oct 31 00:02:37.839363 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:37.806978 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-xkbnw" podUID="469ac12b-4c77-403a-a190-af4037422211" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" Oct 31 00:02:37.893941 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:37.893822 4927 generic.go:356] "Generic (PLEG): container finished" podID="222f0528-7bb2-4803-91ed-cda98b5541dc" containerID="ce6d91159de4fcc3a26120c66b762524ee35b794fc7ce06ba5449b64309722f9" exitCode=0 Oct 31 00:02:37.893941 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:37.893913 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-wftsq" event={"ID":"222f0528-7bb2-4803-91ed-cda98b5541dc","Type":"ContainerDied","Data":"ce6d91159de4fcc3a26120c66b762524ee35b794fc7ce06ba5449b64309722f9"} Oct 31 00:02:37.898297 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:37.898267 4927 generic.go:356] "Generic (PLEG): container finished" podID="91f11663-5773-418a-bde2-532ead3c628c" containerID="0d9a7b1bae0ff7417b190cdc75722d41470432042fdc361ecee591c45a550339" exitCode=0 Oct 31 00:02:37.898695 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:37.898674 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-944gh" event={"ID":"91f11663-5773-418a-bde2-532ead3c628c","Type":"ContainerDied","Data":"0d9a7b1bae0ff7417b190cdc75722d41470432042fdc361ecee591c45a550339"} Oct 31 00:02:37.900514 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:37.900489 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdd5efd5-9957g" event={"ID":"4baf4a61-aae7-4f15-9b1a-fff0ed014c9b","Type":"ContainerStarted","Data":"2af03f9c2e4ff03a0ec534fadfefba8f0bc60905f81025ecc713319968808327"} Oct 31 00:02:37.902691 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:37.902668 4927 generic.go:356] "Generic (PLEG): container finished" podID="647695af-bfbe-41f4-b395-4d1ff1c0286f" containerID="e8af9c72a5830de22e6214c1545562b68ddf01f2d153b236bc34644acce466d6" exitCode=0 Oct 31 00:02:37.902919 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:37.902902 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-xsgv7" event={"ID":"647695af-bfbe-41f4-b395-4d1ff1c0286f","Type":"ContainerDied","Data":"e8af9c72a5830de22e6214c1545562b68ddf01f2d153b236bc34644acce466d6"} Oct 31 00:02:37.904954 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:37.904932 4927 generic.go:356] "Generic (PLEG): container finished" podID="32855a90-baa7-4b9a-b1cb-c97ab9ca8e7d" containerID="c42659543526c5fb508a31f7433fdb5de3fa9b2afdee03a97d42a92b63faceac" exitCode=0 Oct 31 00:02:37.905099 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:37.905084 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-270eef34-4b2nh" event={"ID":"32855a90-baa7-4b9a-b1cb-c97ab9ca8e7d","Type":"ContainerDied","Data":"c42659543526c5fb508a31f7433fdb5de3fa9b2afdee03a97d42a92b63faceac"} Oct 31 00:02:38.107536 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:38.107431 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-xsgv7" Oct 31 00:02:38.206004 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:38.205972 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/647695af-bfbe-41f4-b395-4d1ff1c0286f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/647695af-bfbe-41f4-b395-4d1ff1c0286f-workload-socket\") pod \"647695af-bfbe-41f4-b395-4d1ff1c0286f\" (UID: \"647695af-bfbe-41f4-b395-4d1ff1c0286f\") " Oct 31 00:02:38.206143 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:38.206045 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/647695af-bfbe-41f4-b395-4d1ff1c0286f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/647695af-bfbe-41f4-b395-4d1ff1c0286f-credential-socket\") pod \"647695af-bfbe-41f4-b395-4d1ff1c0286f\" (UID: \"647695af-bfbe-41f4-b395-4d1ff1c0286f\") " Oct 31 00:02:38.206143 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:38.206098 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/647695af-bfbe-41f4-b395-4d1ff1c0286f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/647695af-bfbe-41f4-b395-4d1ff1c0286f-workload-certs\") pod \"647695af-bfbe-41f4-b395-4d1ff1c0286f\" (UID: \"647695af-bfbe-41f4-b395-4d1ff1c0286f\") " Oct 31 00:02:38.206282 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:38.206212 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/647695af-bfbe-41f4-b395-4d1ff1c0286f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/647695af-bfbe-41f4-b395-4d1ff1c0286f-istio-envoy\") pod \"647695af-bfbe-41f4-b395-4d1ff1c0286f\" (UID: \"647695af-bfbe-41f4-b395-4d1ff1c0286f\") " Oct 31 00:02:38.206282 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:38.206241 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/647695af-bfbe-41f4-b395-4d1ff1c0286f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/647695af-bfbe-41f4-b395-4d1ff1c0286f-istio-podinfo\") pod \"647695af-bfbe-41f4-b395-4d1ff1c0286f\" (UID: \"647695af-bfbe-41f4-b395-4d1ff1c0286f\") " Oct 31 00:02:38.206427 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:38.206372 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/647695af-bfbe-41f4-b395-4d1ff1c0286f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/647695af-bfbe-41f4-b395-4d1ff1c0286f-istiod-ca-cert\") pod \"647695af-bfbe-41f4-b395-4d1ff1c0286f\" (UID: \"647695af-bfbe-41f4-b395-4d1ff1c0286f\") " Oct 31 00:02:38.206427 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:38.206401 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/647695af-bfbe-41f4-b395-4d1ff1c0286f-istio-token\" (UniqueName: \"kubernetes.io/projected/647695af-bfbe-41f4-b395-4d1ff1c0286f-istio-token\") pod \"647695af-bfbe-41f4-b395-4d1ff1c0286f\" (UID: \"647695af-bfbe-41f4-b395-4d1ff1c0286f\") " Oct 31 00:02:38.206537 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:38.206528 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/647695af-bfbe-41f4-b395-4d1ff1c0286f-kube-api-access-9k6jb\" (UniqueName: \"kubernetes.io/projected/647695af-bfbe-41f4-b395-4d1ff1c0286f-kube-api-access-9k6jb\") pod \"647695af-bfbe-41f4-b395-4d1ff1c0286f\" (UID: \"647695af-bfbe-41f4-b395-4d1ff1c0286f\") " Oct 31 00:02:38.206714 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:38.206697 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/647695af-bfbe-41f4-b395-4d1ff1c0286f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/647695af-bfbe-41f4-b395-4d1ff1c0286f-istio-data\") pod \"647695af-bfbe-41f4-b395-4d1ff1c0286f\" (UID: \"647695af-bfbe-41f4-b395-4d1ff1c0286f\") " Oct 31 00:02:38.206794 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:38.206743 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/647695af-bfbe-41f4-b395-4d1ff1c0286f-credential-socket" pod "647695af-bfbe-41f4-b395-4d1ff1c0286f" (UID: "647695af-bfbe-41f4-b395-4d1ff1c0286f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:02:38.207629 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:38.207599 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/647695af-bfbe-41f4-b395-4d1ff1c0286f-istiod-ca-cert" pod "647695af-bfbe-41f4-b395-4d1ff1c0286f" (UID: "647695af-bfbe-41f4-b395-4d1ff1c0286f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:02:38.213383 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:38.210925 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/647695af-bfbe-41f4-b395-4d1ff1c0286f-workload-certs" pod "647695af-bfbe-41f4-b395-4d1ff1c0286f" (UID: "647695af-bfbe-41f4-b395-4d1ff1c0286f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:02:38.249356 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:38.232766 4927 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/647695af-bfbe-41f4-b395-4d1ff1c0286f-credential-socket\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:02:38.249356 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:38.232804 4927 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/647695af-bfbe-41f4-b395-4d1ff1c0286f-workload-certs\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:02:38.249356 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:38.232822 4927 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/647695af-bfbe-41f4-b395-4d1ff1c0286f-istiod-ca-cert\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:02:38.249356 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:38.233044 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/647695af-bfbe-41f4-b395-4d1ff1c0286f-istio-data" pod "647695af-bfbe-41f4-b395-4d1ff1c0286f" (UID: "647695af-bfbe-41f4-b395-4d1ff1c0286f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:02:38.269445 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:38.260719 4927 patch_prober.go:29] interesting pod/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-khcqd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" start-of-body= Oct 31 00:02:38.269445 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:38.260781 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-khcqd" podUID="2b317f84-83ac-49aa-921e-0242889f2b4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" Oct 31 00:02:38.269773 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:38.269739 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/647695af-bfbe-41f4-b395-4d1ff1c0286f-workload-socket" pod "647695af-bfbe-41f4-b395-4d1ff1c0286f" (UID: "647695af-bfbe-41f4-b395-4d1ff1c0286f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:02:38.301261 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:38.300660 4927 patch_prober.go:29] interesting pod/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8bt76 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body= Oct 31 00:02:38.301261 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:38.300717 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8bt76" podUID="8789e67e-5656-4479-9b4f-fdc746fcfd15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" Oct 31 00:02:38.360401 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:38.347651 4927 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/647695af-bfbe-41f4-b395-4d1ff1c0286f-workload-socket\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:02:38.372567 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:38.360684 4927 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/647695af-bfbe-41f4-b395-4d1ff1c0286f-istio-data\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:02:38.500559 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:38.500495 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/647695af-bfbe-41f4-b395-4d1ff1c0286f-istio-podinfo" pod "647695af-bfbe-41f4-b395-4d1ff1c0286f" (UID: "647695af-bfbe-41f4-b395-4d1ff1c0286f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:02:38.501667 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:38.501635 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/647695af-bfbe-41f4-b395-4d1ff1c0286f-istio-token" pod "647695af-bfbe-41f4-b395-4d1ff1c0286f" (UID: "647695af-bfbe-41f4-b395-4d1ff1c0286f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:02:38.506600 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:38.506566 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/647695af-bfbe-41f4-b395-4d1ff1c0286f-istio-envoy" pod "647695af-bfbe-41f4-b395-4d1ff1c0286f" (UID: "647695af-bfbe-41f4-b395-4d1ff1c0286f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:02:38.506788 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:38.506659 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/647695af-bfbe-41f4-b395-4d1ff1c0286f-kube-api-access-9k6jb" pod "647695af-bfbe-41f4-b395-4d1ff1c0286f" (UID: "647695af-bfbe-41f4-b395-4d1ff1c0286f"). InnerVolumeSpecName "kube-api-access-9k6jb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:02:38.542235 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:38.542199 4927 patch_prober.go:29] interesting pod/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-xkbnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body= Oct 31 00:02:38.559298 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:38.555594 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-wftsq" Oct 31 00:02:38.559504 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:38.559473 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-xkbnw" podUID="469ac12b-4c77-403a-a190-af4037422211" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" Oct 31 00:02:38.575270 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:38.575239 4927 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/647695af-bfbe-41f4-b395-4d1ff1c0286f-istio-envoy\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:02:38.575485 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:38.575468 4927 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/647695af-bfbe-41f4-b395-4d1ff1c0286f-istio-podinfo\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:02:38.575804 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:38.575785 4927 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/647695af-bfbe-41f4-b395-4d1ff1c0286f-istio-token\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:02:38.576849 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:38.576829 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9k6jb\" (UniqueName: \"kubernetes.io/projected/647695af-bfbe-41f4-b395-4d1ff1c0286f-kube-api-access-9k6jb\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:02:38.592075 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:38.592036 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-944gh" Oct 31 00:02:38.680355 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:38.678489 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/222f0528-7bb2-4803-91ed-cda98b5541dc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/222f0528-7bb2-4803-91ed-cda98b5541dc-istiod-ca-cert\") pod \"222f0528-7bb2-4803-91ed-cda98b5541dc\" (UID: \"222f0528-7bb2-4803-91ed-cda98b5541dc\") " Oct 31 00:02:38.680355 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:38.678587 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/91f11663-5773-418a-bde2-532ead3c628c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91f11663-5773-418a-bde2-532ead3c628c-istio-podinfo\") pod \"91f11663-5773-418a-bde2-532ead3c628c\" (UID: \"91f11663-5773-418a-bde2-532ead3c628c\") " Oct 31 00:02:38.680355 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:38.678637 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/222f0528-7bb2-4803-91ed-cda98b5541dc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/222f0528-7bb2-4803-91ed-cda98b5541dc-credential-socket\") pod \"222f0528-7bb2-4803-91ed-cda98b5541dc\" (UID: \"222f0528-7bb2-4803-91ed-cda98b5541dc\") " Oct 31 00:02:38.680355 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:38.678702 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91f11663-5773-418a-bde2-532ead3c628c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91f11663-5773-418a-bde2-532ead3c628c-istio-data\") pod \"91f11663-5773-418a-bde2-532ead3c628c\" (UID: \"91f11663-5773-418a-bde2-532ead3c628c\") " Oct 31 00:02:38.680355 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:38.678848 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91f11663-5773-418a-bde2-532ead3c628c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91f11663-5773-418a-bde2-532ead3c628c-workload-certs\") pod \"91f11663-5773-418a-bde2-532ead3c628c\" (UID: \"91f11663-5773-418a-bde2-532ead3c628c\") " Oct 31 00:02:38.680355 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:38.678887 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/222f0528-7bb2-4803-91ed-cda98b5541dc-istiod-ca-cert" pod "222f0528-7bb2-4803-91ed-cda98b5541dc" (UID: "222f0528-7bb2-4803-91ed-cda98b5541dc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:02:38.680355 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:38.678890 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/222f0528-7bb2-4803-91ed-cda98b5541dc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/222f0528-7bb2-4803-91ed-cda98b5541dc-istio-data\") pod \"222f0528-7bb2-4803-91ed-cda98b5541dc\" (UID: \"222f0528-7bb2-4803-91ed-cda98b5541dc\") " Oct 31 00:02:38.680355 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:38.678962 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/222f0528-7bb2-4803-91ed-cda98b5541dc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/222f0528-7bb2-4803-91ed-cda98b5541dc-workload-certs\") pod \"222f0528-7bb2-4803-91ed-cda98b5541dc\" (UID: \"222f0528-7bb2-4803-91ed-cda98b5541dc\") " Oct 31 00:02:38.680355 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:38.679036 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/222f0528-7bb2-4803-91ed-cda98b5541dc-kube-api-access-9mk79\" (UniqueName: \"kubernetes.io/projected/222f0528-7bb2-4803-91ed-cda98b5541dc-kube-api-access-9mk79\") pod \"222f0528-7bb2-4803-91ed-cda98b5541dc\" (UID: \"222f0528-7bb2-4803-91ed-cda98b5541dc\") " Oct 31 00:02:38.680355 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:38.679111 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/222f0528-7bb2-4803-91ed-cda98b5541dc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/222f0528-7bb2-4803-91ed-cda98b5541dc-istio-podinfo\") pod \"222f0528-7bb2-4803-91ed-cda98b5541dc\" (UID: \"222f0528-7bb2-4803-91ed-cda98b5541dc\") " Oct 31 00:02:38.680355 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:38.679172 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91f11663-5773-418a-bde2-532ead3c628c-istio-token\" (UniqueName: \"kubernetes.io/projected/91f11663-5773-418a-bde2-532ead3c628c-istio-token\") pod \"91f11663-5773-418a-bde2-532ead3c628c\" (UID: \"91f11663-5773-418a-bde2-532ead3c628c\") " Oct 31 00:02:38.680355 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:38.679183 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/222f0528-7bb2-4803-91ed-cda98b5541dc-credential-socket" pod "222f0528-7bb2-4803-91ed-cda98b5541dc" (UID: "222f0528-7bb2-4803-91ed-cda98b5541dc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:02:38.680355 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:38.679246 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91f11663-5773-418a-bde2-532ead3c628c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91f11663-5773-418a-bde2-532ead3c628c-workload-socket\") pod \"91f11663-5773-418a-bde2-532ead3c628c\" (UID: \"91f11663-5773-418a-bde2-532ead3c628c\") " Oct 31 00:02:38.680355 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:38.679290 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/222f0528-7bb2-4803-91ed-cda98b5541dc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/222f0528-7bb2-4803-91ed-cda98b5541dc-workload-socket\") pod \"222f0528-7bb2-4803-91ed-cda98b5541dc\" (UID: \"222f0528-7bb2-4803-91ed-cda98b5541dc\") " Oct 31 00:02:38.680355 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:38.679439 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/222f0528-7bb2-4803-91ed-cda98b5541dc-istio-token\" (UniqueName: \"kubernetes.io/projected/222f0528-7bb2-4803-91ed-cda98b5541dc-istio-token\") pod \"222f0528-7bb2-4803-91ed-cda98b5541dc\" (UID: \"222f0528-7bb2-4803-91ed-cda98b5541dc\") " Oct 31 00:02:38.680355 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:38.679471 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91f11663-5773-418a-bde2-532ead3c628c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91f11663-5773-418a-bde2-532ead3c628c-istio-envoy\") pod \"91f11663-5773-418a-bde2-532ead3c628c\" (UID: \"91f11663-5773-418a-bde2-532ead3c628c\") " Oct 31 00:02:38.680355 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:38.679535 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91f11663-5773-418a-bde2-532ead3c628c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91f11663-5773-418a-bde2-532ead3c628c-credential-socket\") pod \"91f11663-5773-418a-bde2-532ead3c628c\" (UID: \"91f11663-5773-418a-bde2-532ead3c628c\") " Oct 31 00:02:38.680355 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:38.679592 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/91f11663-5773-418a-bde2-532ead3c628c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91f11663-5773-418a-bde2-532ead3c628c-istiod-ca-cert\") pod \"91f11663-5773-418a-bde2-532ead3c628c\" (UID: \"91f11663-5773-418a-bde2-532ead3c628c\") " Oct 31 00:02:38.680355 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:38.679668 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91f11663-5773-418a-bde2-532ead3c628c-kube-api-access-k5fb6\" (UniqueName: \"kubernetes.io/projected/91f11663-5773-418a-bde2-532ead3c628c-kube-api-access-k5fb6\") pod \"91f11663-5773-418a-bde2-532ead3c628c\" (UID: \"91f11663-5773-418a-bde2-532ead3c628c\") " Oct 31 00:02:38.680355 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:38.679709 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/222f0528-7bb2-4803-91ed-cda98b5541dc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/222f0528-7bb2-4803-91ed-cda98b5541dc-istio-envoy\") pod \"222f0528-7bb2-4803-91ed-cda98b5541dc\" (UID: \"222f0528-7bb2-4803-91ed-cda98b5541dc\") " Oct 31 00:02:38.681602 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:38.680617 4927 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/222f0528-7bb2-4803-91ed-cda98b5541dc-istiod-ca-cert\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:02:38.681602 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:38.680660 4927 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/222f0528-7bb2-4803-91ed-cda98b5541dc-credential-socket\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:02:38.688376 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:38.682292 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91f11663-5773-418a-bde2-532ead3c628c-istio-data" pod "91f11663-5773-418a-bde2-532ead3c628c" (UID: "91f11663-5773-418a-bde2-532ead3c628c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:02:38.688376 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:38.682777 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91f11663-5773-418a-bde2-532ead3c628c-credential-socket" pod "91f11663-5773-418a-bde2-532ead3c628c" (UID: "91f11663-5773-418a-bde2-532ead3c628c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:02:38.688376 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:38.683106 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91f11663-5773-418a-bde2-532ead3c628c-istiod-ca-cert" pod "91f11663-5773-418a-bde2-532ead3c628c" (UID: "91f11663-5773-418a-bde2-532ead3c628c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:02:38.688376 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:38.683694 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/222f0528-7bb2-4803-91ed-cda98b5541dc-workload-socket" pod "222f0528-7bb2-4803-91ed-cda98b5541dc" (UID: "222f0528-7bb2-4803-91ed-cda98b5541dc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:02:38.688376 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:38.683825 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/222f0528-7bb2-4803-91ed-cda98b5541dc-istio-podinfo" pod "222f0528-7bb2-4803-91ed-cda98b5541dc" (UID: "222f0528-7bb2-4803-91ed-cda98b5541dc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:02:38.688376 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:38.684405 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91f11663-5773-418a-bde2-532ead3c628c-workload-socket" pod "91f11663-5773-418a-bde2-532ead3c628c" (UID: "91f11663-5773-418a-bde2-532ead3c628c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:02:38.688376 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:38.684711 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/222f0528-7bb2-4803-91ed-cda98b5541dc-istio-data" pod "222f0528-7bb2-4803-91ed-cda98b5541dc" (UID: "222f0528-7bb2-4803-91ed-cda98b5541dc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:02:38.688376 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:38.685086 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/222f0528-7bb2-4803-91ed-cda98b5541dc-workload-certs" pod "222f0528-7bb2-4803-91ed-cda98b5541dc" (UID: "222f0528-7bb2-4803-91ed-cda98b5541dc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:02:38.689024 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:38.688982 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91f11663-5773-418a-bde2-532ead3c628c-workload-certs" pod "91f11663-5773-418a-bde2-532ead3c628c" (UID: "91f11663-5773-418a-bde2-532ead3c628c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:02:38.692591 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:38.689475 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/222f0528-7bb2-4803-91ed-cda98b5541dc-istio-envoy" pod "222f0528-7bb2-4803-91ed-cda98b5541dc" (UID: "222f0528-7bb2-4803-91ed-cda98b5541dc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:02:38.692591 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:38.692173 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/91f11663-5773-418a-bde2-532ead3c628c-istio-podinfo" pod "91f11663-5773-418a-bde2-532ead3c628c" (UID: "91f11663-5773-418a-bde2-532ead3c628c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:02:38.693050 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:38.692954 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/222f0528-7bb2-4803-91ed-cda98b5541dc-kube-api-access-9mk79" pod "222f0528-7bb2-4803-91ed-cda98b5541dc" (UID: "222f0528-7bb2-4803-91ed-cda98b5541dc"). InnerVolumeSpecName "kube-api-access-9mk79". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:02:38.693155 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:38.693068 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91f11663-5773-418a-bde2-532ead3c628c-kube-api-access-k5fb6" pod "91f11663-5773-418a-bde2-532ead3c628c" (UID: "91f11663-5773-418a-bde2-532ead3c628c"). InnerVolumeSpecName "kube-api-access-k5fb6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:02:38.693489 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:38.693323 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/222f0528-7bb2-4803-91ed-cda98b5541dc-istio-token" pod "222f0528-7bb2-4803-91ed-cda98b5541dc" (UID: "222f0528-7bb2-4803-91ed-cda98b5541dc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:02:38.693489 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:38.693438 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91f11663-5773-418a-bde2-532ead3c628c-istio-envoy" pod "91f11663-5773-418a-bde2-532ead3c628c" (UID: "91f11663-5773-418a-bde2-532ead3c628c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:02:38.693648 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:38.693537 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91f11663-5773-418a-bde2-532ead3c628c-istio-token" pod "91f11663-5773-418a-bde2-532ead3c628c" (UID: "91f11663-5773-418a-bde2-532ead3c628c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:02:38.782359 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:38.782254 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9mk79\" (UniqueName: \"kubernetes.io/projected/222f0528-7bb2-4803-91ed-cda98b5541dc-kube-api-access-9mk79\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:02:38.782359 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:38.782286 4927 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/222f0528-7bb2-4803-91ed-cda98b5541dc-istio-podinfo\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:02:38.782359 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:38.782301 4927 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91f11663-5773-418a-bde2-532ead3c628c-istio-token\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:02:38.782359 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:38.782313 4927 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91f11663-5773-418a-bde2-532ead3c628c-workload-socket\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:02:38.782359 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:38.782325 4927 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/222f0528-7bb2-4803-91ed-cda98b5541dc-workload-socket\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:02:38.782359 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:38.782357 4927 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/222f0528-7bb2-4803-91ed-cda98b5541dc-istio-token\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:02:38.782692 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:38.782370 4927 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91f11663-5773-418a-bde2-532ead3c628c-istio-envoy\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:02:38.782692 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:38.782382 4927 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91f11663-5773-418a-bde2-532ead3c628c-credential-socket\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:02:38.782692 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:38.782394 4927 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91f11663-5773-418a-bde2-532ead3c628c-istiod-ca-cert\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:02:38.782692 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:38.782407 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k5fb6\" (UniqueName: \"kubernetes.io/projected/91f11663-5773-418a-bde2-532ead3c628c-kube-api-access-k5fb6\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:02:38.782692 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:38.782419 4927 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/222f0528-7bb2-4803-91ed-cda98b5541dc-istio-envoy\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:02:38.782692 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:38.782432 4927 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91f11663-5773-418a-bde2-532ead3c628c-istio-podinfo\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:02:38.782692 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:38.782445 4927 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91f11663-5773-418a-bde2-532ead3c628c-istio-data\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:02:38.782692 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:38.782458 4927 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91f11663-5773-418a-bde2-532ead3c628c-workload-certs\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:02:38.782692 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:38.782470 4927 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/222f0528-7bb2-4803-91ed-cda98b5541dc-istio-data\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:02:38.782692 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:38.782484 4927 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/222f0528-7bb2-4803-91ed-cda98b5541dc-workload-certs\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:02:38.965726 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:38.965685 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdd5efd5-9957g" event={"ID":"4baf4a61-aae7-4f15-9b1a-fff0ed014c9b","Type":"ContainerStarted","Data":"9c5d4fd5b410d5b7dc9814577810a2efcb388653d3b474f54e2819f9ed340aae"} Oct 31 00:02:38.968231 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:38.968182 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-xsgv7" event={"ID":"647695af-bfbe-41f4-b395-4d1ff1c0286f","Type":"ContainerDied","Data":"48c868f2a578b03210c7d2ddf4fc25bd077dcd1ae368fa648d2fb2ba1c403a04"} Oct 31 00:02:38.968413 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:38.968250 4927 scope.go:122] "RemoveContainer" containerID="e8af9c72a5830de22e6214c1545562b68ddf01f2d153b236bc34644acce466d6" Oct 31 00:02:38.968487 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:38.968413 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-xsgv7" Oct 31 00:02:38.970319 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:38.970290 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-wftsq" event={"ID":"222f0528-7bb2-4803-91ed-cda98b5541dc","Type":"ContainerDied","Data":"1a4bea5f2e143edd36f624a9eed6f3a3e162e0291946363b17632d0366d9cf5e"} Oct 31 00:02:38.970454 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:38.970302 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-wftsq" Oct 31 00:02:38.972458 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:38.972380 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-944gh" event={"ID":"91f11663-5773-418a-bde2-532ead3c628c","Type":"ContainerDied","Data":"7ad3ee7cfa501b72fc8bbceadfd70465a874089f15bb06774958c31f2a2577e8"} Oct 31 00:02:38.973497 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:38.972593 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-944gh" Oct 31 00:02:39.001630 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:39.001266 4927 scope.go:122] "RemoveContainer" containerID="ce6d91159de4fcc3a26120c66b762524ee35b794fc7ce06ba5449b64309722f9" Oct 31 00:02:39.032657 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:39.032482 4927 scope.go:122] "RemoveContainer" containerID="0d9a7b1bae0ff7417b190cdc75722d41470432042fdc361ecee591c45a550339" Oct 31 00:02:39.116866 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:39.115324 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdd5efd5-9957g" podStartSLOduration=5.115301156 podStartE2EDuration="5.115301156s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:02:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:02:39.088449844 +0000 UTC m=+1272.124780706" watchObservedRunningTime="2026-10-31 00:02:39.115301156 +0000 UTC m=+1272.151631978" Oct 31 00:02:39.326355 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:39.304674 4927 patch_prober.go:29] interesting pod/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8bt76 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body= Oct 31 00:02:39.326355 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:39.304736 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8bt76" podUID="8789e67e-5656-4479-9b4f-fdc746fcfd15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" Oct 31 00:02:39.543403 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:39.543297 4927 patch_prober.go:29] interesting pod/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-xkbnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body= Oct 31 00:02:39.543924 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:39.543891 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-xkbnw" podUID="469ac12b-4c77-403a-a190-af4037422211" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" Oct 31 00:02:39.615356 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:39.606276 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-944gh"] Oct 31 00:02:39.633356 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:39.621229 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-270eef34-4b2nh" Oct 31 00:02:39.633356 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:39.625990 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-944gh"] Oct 31 00:02:39.725356 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:39.724850 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-wftsq"] Oct 31 00:02:39.733354 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:39.730435 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-wftsq"] Oct 31 00:02:39.768354 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:39.746257 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/32855a90-baa7-4b9a-b1cb-c97ab9ca8e7d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32855a90-baa7-4b9a-b1cb-c97ab9ca8e7d-maas-api-xks-ca\") pod \"32855a90-baa7-4b9a-b1cb-c97ab9ca8e7d\" (UID: \"32855a90-baa7-4b9a-b1cb-c97ab9ca8e7d\") " Oct 31 00:02:39.768354 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:39.746801 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/32855a90-baa7-4b9a-b1cb-c97ab9ca8e7d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32855a90-baa7-4b9a-b1cb-c97ab9ca8e7d-maas-api-openshift-ca\") pod \"32855a90-baa7-4b9a-b1cb-c97ab9ca8e7d\" (UID: \"32855a90-baa7-4b9a-b1cb-c97ab9ca8e7d\") " Oct 31 00:02:39.768354 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:39.747448 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32855a90-baa7-4b9a-b1cb-c97ab9ca8e7d-maas-api-openshift-ca" pod "32855a90-baa7-4b9a-b1cb-c97ab9ca8e7d" (UID: "32855a90-baa7-4b9a-b1cb-c97ab9ca8e7d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:02:39.768354 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:39.747764 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32855a90-baa7-4b9a-b1cb-c97ab9ca8e7d-kube-api-access-nl2s7\" (UniqueName: \"kubernetes.io/projected/32855a90-baa7-4b9a-b1cb-c97ab9ca8e7d-kube-api-access-nl2s7\") pod \"32855a90-baa7-4b9a-b1cb-c97ab9ca8e7d\" (UID: \"32855a90-baa7-4b9a-b1cb-c97ab9ca8e7d\") " Oct 31 00:02:39.768354 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:39.748828 4927 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32855a90-baa7-4b9a-b1cb-c97ab9ca8e7d-maas-api-openshift-ca\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:02:39.768354 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:39.752933 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32855a90-baa7-4b9a-b1cb-c97ab9ca8e7d-maas-api-xks-ca" pod "32855a90-baa7-4b9a-b1cb-c97ab9ca8e7d" (UID: "32855a90-baa7-4b9a-b1cb-c97ab9ca8e7d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:02:39.768354 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:39.753815 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32855a90-baa7-4b9a-b1cb-c97ab9ca8e7d-kube-api-access-nl2s7" pod "32855a90-baa7-4b9a-b1cb-c97ab9ca8e7d" (UID: "32855a90-baa7-4b9a-b1cb-c97ab9ca8e7d"). InnerVolumeSpecName "kube-api-access-nl2s7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:02:39.857406 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:39.856983 4927 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32855a90-baa7-4b9a-b1cb-c97ab9ca8e7d-maas-api-xks-ca\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:02:39.857406 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:39.857020 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nl2s7\" (UniqueName: \"kubernetes.io/projected/32855a90-baa7-4b9a-b1cb-c97ab9ca8e7d-kube-api-access-nl2s7\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:02:39.880361 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:39.869708 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-xsgv7"] Oct 31 00:02:39.900509 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:39.900427 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-xsgv7"] Oct 31 00:02:40.110488 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:40.066868 4927 generic.go:356] "Generic (PLEG): container finished" podID="4baf4a61-aae7-4f15-9b1a-fff0ed014c9b" containerID="9c5d4fd5b410d5b7dc9814577810a2efcb388653d3b474f54e2819f9ed340aae" exitCode=0 Oct 31 00:02:40.110488 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:40.067108 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdd5efd5-9957g" event={"ID":"4baf4a61-aae7-4f15-9b1a-fff0ed014c9b","Type":"ContainerDied","Data":"9c5d4fd5b410d5b7dc9814577810a2efcb388653d3b474f54e2819f9ed340aae"} Oct 31 00:02:40.110488 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:40.071227 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-270eef34-4b2nh" event={"ID":"32855a90-baa7-4b9a-b1cb-c97ab9ca8e7d","Type":"ContainerDied","Data":"c62e1e74c49679914f18fc383f043a25e59a10236350a6661ea937edef9ffe7d"} Oct 31 00:02:40.110488 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:40.071287 4927 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c62e1e74c49679914f18fc383f043a25e59a10236350a6661ea937edef9ffe7d" Oct 31 00:02:40.110488 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:40.071421 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-270eef34-4b2nh" Oct 31 00:02:40.507348 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:40.356299 4927 patch_prober.go:29] interesting pod/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8bt76 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body= Oct 31 00:02:40.507348 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:40.356378 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8bt76" podUID="8789e67e-5656-4479-9b4f-fdc746fcfd15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" Oct 31 00:02:40.507348 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:40.473000 4927 patch_prober.go:29] interesting pod/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-khcqd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:02:40.507348 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:40.473086 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-khcqd" podUID="2b317f84-83ac-49aa-921e-0242889f2b4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:02:40.554358 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:40.552213 4927 patch_prober.go:29] interesting pod/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-xkbnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body= Oct 31 00:02:40.554358 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:40.552282 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-xkbnw" podUID="469ac12b-4c77-403a-a190-af4037422211" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" Oct 31 00:02:40.880910 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:40.880818 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-35a187/distinct-2-w6-35a187-kserve-7ff4b6dc55-944g6" podUID="429ee092-42d8-475e-b0c6-b5a9930766fc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.113:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:02:41.047357 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:41.037497 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-46b586/simulator-w0-46b586-kserve-67588b8dbd-z8xmn" podUID="9a25e252-223d-41f2-abf7-632067c77e0b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:02:41.247287 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:41.247184 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:02:41.278112 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:41.276484 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fe01fa/distinct-2-w5-fe01fa-kserve-79b99966d8-dkhzd" podUID="d5c1c9e8-ce47-4868-860d-b8413ff74ab5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.111:8000/ready\": context deadline exceeded" Oct 31 00:02:41.278112 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:41.277481 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5c3374/premium-w1-5c3374-kserve-6c56d4dcd-llwcl" podUID="dc813f02-e077-4251-8c1e-fea0986b0f36" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:02:41.304737 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:41.304620 4927 patch_prober.go:29] interesting pod/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8bt76 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body= Oct 31 00:02:41.304737 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:41.304677 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8bt76" podUID="8789e67e-5656-4479-9b4f-fdc746fcfd15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" Oct 31 00:02:41.346773 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:41.346729 4927 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-27e82b/distinct-w2-27e82b-kserve-75f475fcd9-mccsz" podUID="4ba8ced4-e93f-46ac-87c5-3d0b7fadc02d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/health\": context deadline exceeded" Oct 31 00:02:41.410359 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:41.408890 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="222f0528-7bb2-4803-91ed-cda98b5541dc" path="/var/lib/kubelet/pods/222f0528-7bb2-4803-91ed-cda98b5541dc/volumes" Oct 31 00:02:41.410359 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:41.410058 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="647695af-bfbe-41f4-b395-4d1ff1c0286f" path="/var/lib/kubelet/pods/647695af-bfbe-41f4-b395-4d1ff1c0286f/volumes" Oct 31 00:02:41.417363 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:41.416878 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="91f11663-5773-418a-bde2-532ead3c628c" path="/var/lib/kubelet/pods/91f11663-5773-418a-bde2-532ead3c628c/volumes" Oct 31 00:02:41.533419 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:41.524296 4927 patch_prober.go:29] interesting pod/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-khcqd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:02:41.533419 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:41.524391 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-khcqd" podUID="2b317f84-83ac-49aa-921e-0242889f2b4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:02:41.534693 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:41.534560 4927 patch_prober.go:29] interesting pod/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-xkbnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body= Oct 31 00:02:41.534693 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:41.534621 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-xkbnw" podUID="469ac12b-4c77-403a-a190-af4037422211" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" Oct 31 00:02:41.603357 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:02:41.577209 4927 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32855a90_baa7_4b9a_b1cb_c97ab9ca8e7d.slice/crio-c62e1e74c49679914f18fc383f043a25e59a10236350a6661ea937edef9ffe7d\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32855a90_baa7_4b9a_b1cb_c97ab9ca8e7d.slice\": RecentStats: unable to find data in memory cache]" Oct 31 00:02:41.637159 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:41.635697 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-27e82b/distinct-w2-27e82b-kserve-75f475fcd9-mccsz" podUID="4ba8ced4-e93f-46ac-87c5-3d0b7fadc02d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:02:41.655086 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:41.604090 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-986d37/premium-w3-986d37-kserve-5c7c7757fb-r99j7" podUID="f4e38df7-355f-4b47-a0b5-7e0b907f3326" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:02:41.904810 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:41.904683 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-35a187/distinct-2-w6-35a187-kserve-7ff4b6dc55-944g6" podUID="429ee092-42d8-475e-b0c6-b5a9930766fc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.113:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:02:41.905486 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:41.905420 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-35a187/distinct-2-w6-35a187-kserve-7ff4b6dc55-944g6" Oct 31 00:02:42.049620 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:42.049541 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-46b586/simulator-w0-46b586-kserve-67588b8dbd-z8xmn" podUID="9a25e252-223d-41f2-abf7-632067c77e0b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:02:42.057355 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:42.055170 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-46b586/simulator-w0-46b586-kserve-67588b8dbd-z8xmn" Oct 31 00:02:42.459910 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:42.431452 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5c3374/premium-w1-5c3374-kserve-6c56d4dcd-llwcl" podUID="dc813f02-e077-4251-8c1e-fea0986b0f36" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:02:42.459910 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:42.445730 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fe01fa/distinct-2-w5-fe01fa-kserve-79b99966d8-dkhzd" podUID="d5c1c9e8-ce47-4868-860d-b8413ff74ab5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.111:8000/ready\": context deadline exceeded" Oct 31 00:02:42.459910 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:42.455648 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5c3374/premium-w1-5c3374-kserve-6c56d4dcd-llwcl" Oct 31 00:02:42.459910 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:42.455711 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-fe01fa/distinct-2-w5-fe01fa-kserve-79b99966d8-dkhzd" Oct 31 00:02:42.540503 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:42.540465 4927 patch_prober.go:29] interesting pod/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-khcqd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:02:42.541536 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:42.540759 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-khcqd" podUID="2b317f84-83ac-49aa-921e-0242889f2b4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:02:42.710620 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:42.679420 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-27e82b/distinct-w2-27e82b-kserve-75f475fcd9-mccsz" podUID="4ba8ced4-e93f-46ac-87c5-3d0b7fadc02d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:02:42.736602 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:42.736555 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-27e82b/distinct-w2-27e82b-kserve-75f475fcd9-mccsz" Oct 31 00:02:42.909356 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:42.905430 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-35a187/distinct-2-w6-35a187-kserve-7ff4b6dc55-944g6" podUID="429ee092-42d8-475e-b0c6-b5a9930766fc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.113:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:02:43.046989 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:43.046849 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-986d37/premium-w3-986d37-kserve-5c7c7757fb-r99j7" podUID="f4e38df7-355f-4b47-a0b5-7e0b907f3326" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:02:43.047905 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:43.047880 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-986d37/premium-w3-986d37-kserve-5c7c7757fb-r99j7" Oct 31 00:02:43.255357 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:43.252634 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-46b586/simulator-w0-46b586-kserve-67588b8dbd-z8xmn" podUID="9a25e252-223d-41f2-abf7-632067c77e0b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded" Oct 31 00:02:43.448387 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:43.442193 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5c3374/premium-w1-5c3374-kserve-6c56d4dcd-llwcl" podUID="dc813f02-e077-4251-8c1e-fea0986b0f36" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded" Oct 31 00:02:43.450895 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:43.448895 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fe01fa/distinct-2-w5-fe01fa-kserve-79b99966d8-dkhzd" podUID="d5c1c9e8-ce47-4868-860d-b8413ff74ab5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.111:8000/ready\": context deadline exceeded" Oct 31 00:02:43.555353 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:43.548440 4927 patch_prober.go:29] interesting pod/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8bt76 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:02:43.555353 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:43.548503 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8bt76" podUID="8789e67e-5656-4479-9b4f-fdc746fcfd15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:02:43.636507 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:43.612492 4927 patch_prober.go:29] interesting pod/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-khcqd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:02:43.636507 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:43.612559 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-khcqd" podUID="2b317f84-83ac-49aa-921e-0242889f2b4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:02:43.681105 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:43.679917 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-27e82b/distinct-w2-27e82b-kserve-75f475fcd9-mccsz" podUID="4ba8ced4-e93f-46ac-87c5-3d0b7fadc02d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/ready\": context deadline exceeded" Oct 31 00:02:43.761561 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:43.761464 4927 patch_prober.go:29] interesting pod/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-xkbnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:02:43.762170 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:43.762132 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-xkbnw" podUID="469ac12b-4c77-403a-a190-af4037422211" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:02:44.022909 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:44.017819 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-35a187/distinct-2-w6-35a187-kserve-7ff4b6dc55-944g6" podUID="429ee092-42d8-475e-b0c6-b5a9930766fc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.113:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:02:44.073362 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:44.060502 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-986d37/premium-w3-986d37-kserve-5c7c7757fb-r99j7" podUID="f4e38df7-355f-4b47-a0b5-7e0b907f3326" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:02:44.284554 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:44.256998 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-46b586/simulator-w0-46b586-kserve-67588b8dbd-z8xmn" podUID="9a25e252-223d-41f2-abf7-632067c77e0b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded" Oct 31 00:02:44.378425 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:44.368991 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdd5efd5-9957g" Oct 31 00:02:44.528027 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:44.527976 4927 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-35a187/distinct-2-w6-35a187-kserve-7ff4b6dc55-944g6" podUID="429ee092-42d8-475e-b0c6-b5a9930766fc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.113:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:02:44.562781 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:44.550398 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5c3374/premium-w1-5c3374-kserve-6c56d4dcd-llwcl" podUID="dc813f02-e077-4251-8c1e-fea0986b0f36" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded" Oct 31 00:02:44.568858 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:44.568830 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4baf4a61-aae7-4f15-9b1a-fff0ed014c9b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4baf4a61-aae7-4f15-9b1a-fff0ed014c9b-maas-api-openshift-ca\") pod \"4baf4a61-aae7-4f15-9b1a-fff0ed014c9b\" (UID: \"4baf4a61-aae7-4f15-9b1a-fff0ed014c9b\") " Oct 31 00:02:44.569632 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:44.569590 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4baf4a61-aae7-4f15-9b1a-fff0ed014c9b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4baf4a61-aae7-4f15-9b1a-fff0ed014c9b-maas-api-xks-ca\") pod \"4baf4a61-aae7-4f15-9b1a-fff0ed014c9b\" (UID: \"4baf4a61-aae7-4f15-9b1a-fff0ed014c9b\") " Oct 31 00:02:44.863174 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:44.863076 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4baf4a61-aae7-4f15-9b1a-fff0ed014c9b-kube-api-access-558wh\" (UniqueName: \"kubernetes.io/projected/4baf4a61-aae7-4f15-9b1a-fff0ed014c9b-kube-api-access-558wh\") pod \"4baf4a61-aae7-4f15-9b1a-fff0ed014c9b\" (UID: \"4baf4a61-aae7-4f15-9b1a-fff0ed014c9b\") " Oct 31 00:02:44.863679 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:44.596453 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4baf4a61-aae7-4f15-9b1a-fff0ed014c9b-maas-api-openshift-ca" pod "4baf4a61-aae7-4f15-9b1a-fff0ed014c9b" (UID: "4baf4a61-aae7-4f15-9b1a-fff0ed014c9b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:02:44.863771 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:44.599913 4927 patch_prober.go:29] interesting pod/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8bt76 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:02:44.863771 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:44.863727 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8bt76" podUID="8789e67e-5656-4479-9b4f-fdc746fcfd15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:02:44.877378 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:44.873175 4927 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4baf4a61-aae7-4f15-9b1a-fff0ed014c9b-maas-api-openshift-ca\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:02:44.877378 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:44.662453 4927 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-fe01fa/distinct-2-w5-fe01fa-kserve-79b99966d8-dkhzd" podUID="d5c1c9e8-ce47-4868-860d-b8413ff74ab5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.111:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 00:02:44.877378 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:44.499709 4927 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-986d37/premium-w3-986d37-kserve-5c7c7757fb-r99j7" podUID="f4e38df7-355f-4b47-a0b5-7e0b907f3326" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:02:44.881873 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:44.599867 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fe01fa/distinct-2-w5-fe01fa-kserve-79b99966d8-dkhzd" podUID="d5c1c9e8-ce47-4868-860d-b8413ff74ab5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.111:8000/ready\": context deadline exceeded" Oct 31 00:02:44.905413 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:44.621237 4927 patch_prober.go:29] interesting pod/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-khcqd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:02:44.945472 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:44.945419 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-khcqd" podUID="2b317f84-83ac-49aa-921e-0242889f2b4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:02:44.946249 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:44.794172 4927 patch_prober.go:29] interesting pod/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-xkbnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:02:45.536003 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:45.535888 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-xkbnw" podUID="469ac12b-4c77-403a-a190-af4037422211" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:02:45.536695 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:44.708859 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-27e82b/distinct-w2-27e82b-kserve-75f475fcd9-mccsz" podUID="4ba8ced4-e93f-46ac-87c5-3d0b7fadc02d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:02:45.537234 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:45.206555 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4baf4a61-aae7-4f15-9b1a-fff0ed014c9b-kube-api-access-558wh" pod "4baf4a61-aae7-4f15-9b1a-fff0ed014c9b" (UID: "4baf4a61-aae7-4f15-9b1a-fff0ed014c9b"). InnerVolumeSpecName "kube-api-access-558wh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:02:45.537396 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:45.061470 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-98d91d7d-b7th9"] Oct 31 00:02:45.562309 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:45.299765 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-46b586/simulator-w0-46b586-kserve-67588b8dbd-z8xmn" podUID="9a25e252-223d-41f2-abf7-632067c77e0b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:02:45.604945 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:45.604912 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4baf4a61-aae7-4f15-9b1a-fff0ed014c9b-kube-api-access-558wh\" (UniqueName: \"kubernetes.io/projected/4baf4a61-aae7-4f15-9b1a-fff0ed014c9b-kube-api-access-558wh\") pod \"4baf4a61-aae7-4f15-9b1a-fff0ed014c9b\" (UID: \"4baf4a61-aae7-4f15-9b1a-fff0ed014c9b\") " Oct 31 00:02:45.611355 ip-10-0-141-180 kubenswrapper[4927]: W1031 00:02:45.607278 4927 empty_dir.go:505] Warning: Unmount skipped because path does not exist: /var/lib/kubelet/pods/4baf4a61-aae7-4f15-9b1a-fff0ed014c9b/volumes/kubernetes.io~projected/kube-api-access-558wh Oct 31 00:02:45.636526 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:45.636469 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4baf4a61-aae7-4f15-9b1a-fff0ed014c9b-kube-api-access-558wh" pod "4baf4a61-aae7-4f15-9b1a-fff0ed014c9b" (UID: "4baf4a61-aae7-4f15-9b1a-fff0ed014c9b"). InnerVolumeSpecName "kube-api-access-558wh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:02:45.674149 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:45.023368 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-35a187/distinct-2-w6-35a187-kserve-7ff4b6dc55-944g6" podUID="429ee092-42d8-475e-b0c6-b5a9930766fc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.113:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:02:45.674588 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:45.061423 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-986d37/premium-w3-986d37-kserve-5c7c7757fb-r99j7" podUID="f4e38df7-355f-4b47-a0b5-7e0b907f3326" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded" Oct 31 00:02:45.675253 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:45.675218 4927 cpu_manager.go:414] "containerMap: removing container" podUID="4baf4a61-aae7-4f15-9b1a-fff0ed014c9b" containerName="revoke-keys" Oct 31 00:02:45.675368 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:45.675254 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4baf4a61-aae7-4f15-9b1a-fff0ed014c9b" containerName="revoke-keys" Oct 31 00:02:45.675368 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:45.675300 4927 cpu_manager.go:414] "containerMap: removing container" podUID="91f11663-5773-418a-bde2-532ead3c628c" containerName="istio-proxy" Oct 31 00:02:45.675368 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:45.675309 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91f11663-5773-418a-bde2-532ead3c628c" containerName="istio-proxy" Oct 31 00:02:45.755474 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:45.755430 4927 patch_prober.go:29] interesting pod/console-operator-7cd56b8dbc-pqt2c container/console-operator namespace/openshift-console-operator: Liveness probe status=failure output="Get \"https://10.132.0.9:8443/healthz\": context deadline exceeded" start-of-body= Oct 31 00:02:45.755809 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:45.755741 4927 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-console-operator/console-operator-7cd56b8dbc-pqt2c" podUID="6382dfb5-a363-47a2-bc79-0b96f7bec0c3" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.9:8443/healthz\": context deadline exceeded" Oct 31 00:02:45.800410 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:45.790510 4927 cpu_manager.go:414] "containerMap: removing container" podUID="222f0528-7bb2-4803-91ed-cda98b5541dc" containerName="istio-proxy" Oct 31 00:02:45.800410 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:45.790551 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="222f0528-7bb2-4803-91ed-cda98b5541dc" containerName="istio-proxy" Oct 31 00:02:45.800410 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:45.790601 4927 cpu_manager.go:414] "containerMap: removing container" podUID="32855a90-baa7-4b9a-b1cb-c97ab9ca8e7d" containerName="revoke-keys" Oct 31 00:02:45.800410 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:45.790624 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32855a90-baa7-4b9a-b1cb-c97ab9ca8e7d" containerName="revoke-keys" Oct 31 00:02:45.800410 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:45.790663 4927 cpu_manager.go:414] "containerMap: removing container" podUID="647695af-bfbe-41f4-b395-4d1ff1c0286f" containerName="istio-proxy" Oct 31 00:02:45.800410 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:45.790671 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="647695af-bfbe-41f4-b395-4d1ff1c0286f" containerName="istio-proxy" Oct 31 00:02:45.800410 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:45.791479 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="91f11663-5773-418a-bde2-532ead3c628c" containerName="istio-proxy" Oct 31 00:02:45.800410 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:45.791506 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="222f0528-7bb2-4803-91ed-cda98b5541dc" containerName="istio-proxy" Oct 31 00:02:45.800410 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:45.791530 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="32855a90-baa7-4b9a-b1cb-c97ab9ca8e7d" containerName="revoke-keys" Oct 31 00:02:45.800410 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:45.791551 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="647695af-bfbe-41f4-b395-4d1ff1c0286f" containerName="istio-proxy" Oct 31 00:02:45.800410 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:45.791588 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="4baf4a61-aae7-4f15-9b1a-fff0ed014c9b" containerName="revoke-keys" Oct 31 00:02:45.876688 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:45.876633 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5c3374/premium-w1-5c3374-kserve-6c56d4dcd-llwcl" podUID="dc813f02-e077-4251-8c1e-fea0986b0f36" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:02:45.995313 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:45.979259 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-558wh\" (UniqueName: \"kubernetes.io/projected/4baf4a61-aae7-4f15-9b1a-fff0ed014c9b-kube-api-access-558wh\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:02:46.039353 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:46.019629 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-khcqd" Oct 31 00:02:46.079701 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:46.039143 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98d91d7d-b7th9" Oct 31 00:02:46.083067 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:46.040773 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4baf4a61-aae7-4f15-9b1a-fff0ed014c9b-maas-api-xks-ca" pod "4baf4a61-aae7-4f15-9b1a-fff0ed014c9b" (UID: "4baf4a61-aae7-4f15-9b1a-fff0ed014c9b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:02:46.151659 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:46.143932 4927 patch_prober.go:29] interesting pod/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8bt76 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 00:02:46.313752 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:46.313698 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8bt76" podUID="8789e67e-5656-4479-9b4f-fdc746fcfd15" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:02:46.316091 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:46.153089 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fe01fa/distinct-2-w5-fe01fa-kserve-79b99966d8-dkhzd" podUID="d5c1c9e8-ce47-4868-860d-b8413ff74ab5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.111:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:02:46.317791 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:46.289031 4927 patch_prober.go:29] interesting pod/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-khcqd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:02:46.318016 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:46.317978 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-khcqd" podUID="2b317f84-83ac-49aa-921e-0242889f2b4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:02:46.499356 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:46.246528 4927 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4baf4a61-aae7-4f15-9b1a-fff0ed014c9b-maas-api-xks-ca\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:02:46.566578 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:46.566511 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-27e82b/distinct-w2-27e82b-kserve-75f475fcd9-mccsz" podUID="4ba8ced4-e93f-46ac-87c5-3d0b7fadc02d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 00:02:46.602056 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:46.602019 4927 patch_prober.go:29] interesting pod/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-xkbnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:02:46.602325 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:46.602296 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-xkbnw" podUID="469ac12b-4c77-403a-a190-af4037422211" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:02:46.764515 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:46.764424 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:02:46.805352 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:46.804747 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-46b586/simulator-w0-46b586-kserve-67588b8dbd-z8xmn" podUID="9a25e252-223d-41f2-abf7-632067c77e0b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded" Oct 31 00:02:46.849750 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:46.833104 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39cb8c11-9c39-48ef-b94e-822f423e464b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-98d91d7d-b7th9\" (UID: \"39cb8c11-9c39-48ef-b94e-822f423e464b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98d91d7d-b7th9" Oct 31 00:02:46.849750 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:46.833293 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pqv5s\" (UniqueName: \"kubernetes.io/projected/39cb8c11-9c39-48ef-b94e-822f423e464b-kube-api-access-pqv5s\") pod \"maas-api-revoke-sub-98d91d7d-b7th9\" (UID: \"39cb8c11-9c39-48ef-b94e-822f423e464b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98d91d7d-b7th9" Oct 31 00:02:46.849750 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:46.833362 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39cb8c11-9c39-48ef-b94e-822f423e464b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-98d91d7d-b7th9\" (UID: \"39cb8c11-9c39-48ef-b94e-822f423e464b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98d91d7d-b7th9" Oct 31 00:02:46.952475 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:46.952426 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdd5efd5-9957g" Oct 31 00:02:46.998279 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:46.911320 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdd5efd5-9957g" event={"ID":"4baf4a61-aae7-4f15-9b1a-fff0ed014c9b","Type":"ContainerDied","Data":"2af03f9c2e4ff03a0ec534fadfefba8f0bc60905f81025ecc713319968808327"} Oct 31 00:02:46.998458 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:46.998315 4927 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2af03f9c2e4ff03a0ec534fadfefba8f0bc60905f81025ecc713319968808327" Oct 31 00:02:46.998458 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:46.998408 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-986d37/premium-w3-986d37-kserve-5c7c7757fb-r99j7" Oct 31 00:02:46.998458 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:46.998432 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-98d91d7d-b7th9"] Oct 31 00:02:46.998645 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:46.998482 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-35a187/distinct-2-w6-35a187-kserve-7ff4b6dc55-944g6" Oct 31 00:02:46.998645 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:46.998499 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3e0479e-2vfsd"] Oct 31 00:02:46.998645 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:46.998520 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3e0479e-2vfsd"] Oct 31 00:02:46.998645 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:46.998544 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8bt76" Oct 31 00:02:46.998645 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:46.998572 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-xkbnw" Oct 31 00:02:46.998645 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:46.998616 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5c3374/premium-w1-5c3374-kserve-6c56d4dcd-llwcl" Oct 31 00:02:46.998911 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:46.998650 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-46b586/simulator-w0-46b586-kserve-67588b8dbd-z8xmn" Oct 31 00:02:46.998911 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:46.998684 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-fe01fa/distinct-2-w5-fe01fa-kserve-79b99966d8-dkhzd" Oct 31 00:02:46.998911 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:46.998715 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-27e82b/distinct-w2-27e82b-kserve-75f475fcd9-mccsz" Oct 31 00:02:46.998911 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:46.977383 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39cb8c11-9c39-48ef-b94e-822f423e464b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-98d91d7d-b7th9\" (UID: \"39cb8c11-9c39-48ef-b94e-822f423e464b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98d91d7d-b7th9" Oct 31 00:02:46.999098 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:46.998987 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqv5s\" (UniqueName: \"kubernetes.io/projected/39cb8c11-9c39-48ef-b94e-822f423e464b-kube-api-access-pqv5s\") pod \"maas-api-revoke-sub-98d91d7d-b7th9\" (UID: \"39cb8c11-9c39-48ef-b94e-822f423e464b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98d91d7d-b7th9" Oct 31 00:02:46.999098 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:46.999059 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39cb8c11-9c39-48ef-b94e-822f423e464b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-98d91d7d-b7th9\" (UID: \"39cb8c11-9c39-48ef-b94e-822f423e464b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98d91d7d-b7th9" Oct 31 00:02:47.001513 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:46.978254 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39cb8c11-9c39-48ef-b94e-822f423e464b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-98d91d7d-b7th9\" (UID: \"39cb8c11-9c39-48ef-b94e-822f423e464b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98d91d7d-b7th9" Oct 31 00:02:47.008361 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:47.003094 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39cb8c11-9c39-48ef-b94e-822f423e464b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-98d91d7d-b7th9\" (UID: \"39cb8c11-9c39-48ef-b94e-822f423e464b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98d91d7d-b7th9" Oct 31 00:02:47.068103 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:47.068041 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqv5s\" (UniqueName: \"kubernetes.io/projected/39cb8c11-9c39-48ef-b94e-822f423e464b-kube-api-access-pqv5s\") pod \"maas-api-revoke-sub-98d91d7d-b7th9\" (UID: \"39cb8c11-9c39-48ef-b94e-822f423e464b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98d91d7d-b7th9" Oct 31 00:02:47.104248 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:47.104213 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1354a346-145b-495d-84ea-c9d92967d477" path="/var/lib/kubelet/pods/1354a346-145b-495d-84ea-c9d92967d477/volumes" Oct 31 00:02:47.272844 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:47.272698 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-khcqd" Oct 31 00:02:47.352408 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:47.339296 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98d91d7d-b7th9" Oct 31 00:02:47.356352 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:47.355843 4927 patch_prober.go:29] interesting pod/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8bt76 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:02:47.356352 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:47.355905 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8bt76" podUID="8789e67e-5656-4479-9b4f-fdc746fcfd15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:02:47.548314 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:47.524109 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8bt76" Oct 31 00:02:47.669386 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:47.664469 4927 patch_prober.go:29] interesting pod/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-xkbnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:02:47.669386 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:47.664530 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-xkbnw" podUID="469ac12b-4c77-403a-a190-af4037422211" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:02:47.885127 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:47.884994 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-khcqd" Oct 31 00:02:47.885127 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:47.885074 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8bt76" Oct 31 00:02:48.060458 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:48.060429 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-98d91d7d-b7th9"] Oct 31 00:02:48.107041 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:48.107015 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-xkbnw" Oct 31 00:02:48.111379 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:48.108403 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-xkbnw" Oct 31 00:02:48.148387 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:48.147847 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fd61788b-nmjjt"] Oct 31 00:02:48.155815 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:48.155089 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd61788b-nmjjt" Oct 31 00:02:48.190108 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:48.190072 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da3ace09-2b4e-4ccd-9fb3-31473672a48f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fd61788b-nmjjt\" (UID: \"da3ace09-2b4e-4ccd-9fb3-31473672a48f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd61788b-nmjjt" Oct 31 00:02:48.190283 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:48.190260 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da3ace09-2b4e-4ccd-9fb3-31473672a48f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fd61788b-nmjjt\" (UID: \"da3ace09-2b4e-4ccd-9fb3-31473672a48f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd61788b-nmjjt" Oct 31 00:02:48.190703 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:48.190675 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2v6d2\" (UniqueName: \"kubernetes.io/projected/da3ace09-2b4e-4ccd-9fb3-31473672a48f-kube-api-access-2v6d2\") pod \"maas-api-revoke-sub-fd61788b-nmjjt\" (UID: \"da3ace09-2b4e-4ccd-9fb3-31473672a48f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd61788b-nmjjt" Oct 31 00:02:48.205670 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:48.205637 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fd61788b-nmjjt"] Oct 31 00:02:48.300325 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:48.300286 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da3ace09-2b4e-4ccd-9fb3-31473672a48f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fd61788b-nmjjt\" (UID: \"da3ace09-2b4e-4ccd-9fb3-31473672a48f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd61788b-nmjjt" Oct 31 00:02:48.300532 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:48.300412 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2v6d2\" (UniqueName: \"kubernetes.io/projected/da3ace09-2b4e-4ccd-9fb3-31473672a48f-kube-api-access-2v6d2\") pod \"maas-api-revoke-sub-fd61788b-nmjjt\" (UID: \"da3ace09-2b4e-4ccd-9fb3-31473672a48f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd61788b-nmjjt" Oct 31 00:02:48.300640 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:48.300620 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da3ace09-2b4e-4ccd-9fb3-31473672a48f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fd61788b-nmjjt\" (UID: \"da3ace09-2b4e-4ccd-9fb3-31473672a48f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd61788b-nmjjt" Oct 31 00:02:48.301028 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:48.301001 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da3ace09-2b4e-4ccd-9fb3-31473672a48f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fd61788b-nmjjt\" (UID: \"da3ace09-2b4e-4ccd-9fb3-31473672a48f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd61788b-nmjjt" Oct 31 00:02:48.303220 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:48.303198 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da3ace09-2b4e-4ccd-9fb3-31473672a48f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fd61788b-nmjjt\" (UID: \"da3ace09-2b4e-4ccd-9fb3-31473672a48f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd61788b-nmjjt" Oct 31 00:02:48.362634 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:48.362606 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2v6d2\" (UniqueName: \"kubernetes.io/projected/da3ace09-2b4e-4ccd-9fb3-31473672a48f-kube-api-access-2v6d2\") pod \"maas-api-revoke-sub-fd61788b-nmjjt\" (UID: \"da3ace09-2b4e-4ccd-9fb3-31473672a48f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd61788b-nmjjt" Oct 31 00:02:48.472412 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:48.472378 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd61788b-nmjjt" Oct 31 00:02:48.477240 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:48.477212 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:02:48.873517 ip-10-0-141-180 kubenswrapper[4927]: W1031 00:02:48.873473 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda3ace09_2b4e_4ccd_9fb3_31473672a48f.slice/crio-57b57ec82af8fb3defd2d1dc5d6850f043b52efeb43895d4939e44fb75bbe73b WatchSource:0}: Error finding container 57b57ec82af8fb3defd2d1dc5d6850f043b52efeb43895d4939e44fb75bbe73b: Status 404 returned error can't find the container with id 57b57ec82af8fb3defd2d1dc5d6850f043b52efeb43895d4939e44fb75bbe73b Oct 31 00:02:48.874368 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:48.874326 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fd61788b-nmjjt"] Oct 31 00:02:48.898031 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:48.894752 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd61788b-nmjjt" event={"ID":"da3ace09-2b4e-4ccd-9fb3-31473672a48f","Type":"ContainerStarted","Data":"57b57ec82af8fb3defd2d1dc5d6850f043b52efeb43895d4939e44fb75bbe73b"} Oct 31 00:02:48.900081 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:48.900046 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98d91d7d-b7th9" event={"ID":"39cb8c11-9c39-48ef-b94e-822f423e464b","Type":"ContainerStarted","Data":"5c6bdaede9cc3fb802e3151b43e4208578d9ed6ff9dd8acc7918de4c5c3c6afe"} Oct 31 00:02:48.900232 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:48.900111 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98d91d7d-b7th9" event={"ID":"39cb8c11-9c39-48ef-b94e-822f423e464b","Type":"ContainerStarted","Data":"8f4ba2946ed5283c57650369f52b17c94296e1cfdf412c7e0063c319d6032b1c"} Oct 31 00:02:49.023141 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:49.023086 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98d91d7d-b7th9" podStartSLOduration=5.02307081 podStartE2EDuration="5.02307081s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:02:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:02:49.0192122 +0000 UTC m=+1282.055543022" watchObservedRunningTime="2026-10-31 00:02:49.02307081 +0000 UTC m=+1282.059401687" Oct 31 00:02:49.682051 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:02:49.682010 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sqv69" containerName="extractor" Oct 31 00:02:49.912760 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:49.912727 4927 generic.go:356] "Generic (PLEG): container finished" podID="da3ace09-2b4e-4ccd-9fb3-31473672a48f" containerID="bb9ada750c53abad00d3f18683cbd0419ee40dd9fb79709bd43a8742cb49854b" exitCode=0 Oct 31 00:02:49.912946 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:49.912811 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd61788b-nmjjt" event={"ID":"da3ace09-2b4e-4ccd-9fb3-31473672a48f","Type":"ContainerDied","Data":"bb9ada750c53abad00d3f18683cbd0419ee40dd9fb79709bd43a8742cb49854b"} Oct 31 00:02:49.914776 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:49.914750 4927 generic.go:356] "Generic (PLEG): container finished" podID="39cb8c11-9c39-48ef-b94e-822f423e464b" containerID="5c6bdaede9cc3fb802e3151b43e4208578d9ed6ff9dd8acc7918de4c5c3c6afe" exitCode=0 Oct 31 00:02:49.914924 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:49.914828 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98d91d7d-b7th9" event={"ID":"39cb8c11-9c39-48ef-b94e-822f423e464b","Type":"ContainerDied","Data":"5c6bdaede9cc3fb802e3151b43e4208578d9ed6ff9dd8acc7918de4c5c3c6afe"} Oct 31 00:02:50.063937 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:50.063854 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5012722b-8hqgd"] Oct 31 00:02:50.068509 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:50.068476 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5012722b-8hqgd"] Oct 31 00:02:51.093027 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:51.092968 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5bda55b5-e785-43af-874c-de9d032489a5" path="/var/lib/kubelet/pods/5bda55b5-e785-43af-874c-de9d032489a5/volumes" Oct 31 00:02:51.175907 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:51.175622 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98d91d7d-b7th9" Oct 31 00:02:51.178458 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:51.178411 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd61788b-nmjjt" Oct 31 00:02:51.254313 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:51.254274 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/da3ace09-2b4e-4ccd-9fb3-31473672a48f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da3ace09-2b4e-4ccd-9fb3-31473672a48f-maas-api-xks-ca\") pod \"da3ace09-2b4e-4ccd-9fb3-31473672a48f\" (UID: \"da3ace09-2b4e-4ccd-9fb3-31473672a48f\") " Oct 31 00:02:51.254523 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:51.254398 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da3ace09-2b4e-4ccd-9fb3-31473672a48f-kube-api-access-2v6d2\" (UniqueName: \"kubernetes.io/projected/da3ace09-2b4e-4ccd-9fb3-31473672a48f-kube-api-access-2v6d2\") pod \"da3ace09-2b4e-4ccd-9fb3-31473672a48f\" (UID: \"da3ace09-2b4e-4ccd-9fb3-31473672a48f\") " Oct 31 00:02:51.254523 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:51.254476 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da3ace09-2b4e-4ccd-9fb3-31473672a48f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da3ace09-2b4e-4ccd-9fb3-31473672a48f-maas-api-openshift-ca\") pod \"da3ace09-2b4e-4ccd-9fb3-31473672a48f\" (UID: \"da3ace09-2b4e-4ccd-9fb3-31473672a48f\") " Oct 31 00:02:51.254649 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:51.254628 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39cb8c11-9c39-48ef-b94e-822f423e464b-kube-api-access-pqv5s\" (UniqueName: \"kubernetes.io/projected/39cb8c11-9c39-48ef-b94e-822f423e464b-kube-api-access-pqv5s\") pod \"39cb8c11-9c39-48ef-b94e-822f423e464b\" (UID: \"39cb8c11-9c39-48ef-b94e-822f423e464b\") " Oct 31 00:02:51.254722 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:51.254701 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/39cb8c11-9c39-48ef-b94e-822f423e464b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39cb8c11-9c39-48ef-b94e-822f423e464b-maas-api-openshift-ca\") pod \"39cb8c11-9c39-48ef-b94e-822f423e464b\" (UID: \"39cb8c11-9c39-48ef-b94e-822f423e464b\") " Oct 31 00:02:51.254773 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:51.254747 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/39cb8c11-9c39-48ef-b94e-822f423e464b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39cb8c11-9c39-48ef-b94e-822f423e464b-maas-api-xks-ca\") pod \"39cb8c11-9c39-48ef-b94e-822f423e464b\" (UID: \"39cb8c11-9c39-48ef-b94e-822f423e464b\") " Oct 31 00:02:51.254915 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:51.254886 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da3ace09-2b4e-4ccd-9fb3-31473672a48f-maas-api-openshift-ca" pod "da3ace09-2b4e-4ccd-9fb3-31473672a48f" (UID: "da3ace09-2b4e-4ccd-9fb3-31473672a48f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:02:51.255592 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:51.255274 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39cb8c11-9c39-48ef-b94e-822f423e464b-maas-api-openshift-ca" pod "39cb8c11-9c39-48ef-b94e-822f423e464b" (UID: "39cb8c11-9c39-48ef-b94e-822f423e464b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:02:51.255592 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:51.255543 4927 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39cb8c11-9c39-48ef-b94e-822f423e464b-maas-api-openshift-ca\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:02:51.255592 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:51.255565 4927 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da3ace09-2b4e-4ccd-9fb3-31473672a48f-maas-api-openshift-ca\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:02:51.256702 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:51.256675 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da3ace09-2b4e-4ccd-9fb3-31473672a48f-maas-api-xks-ca" pod "da3ace09-2b4e-4ccd-9fb3-31473672a48f" (UID: "da3ace09-2b4e-4ccd-9fb3-31473672a48f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:02:51.256879 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:51.256847 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da3ace09-2b4e-4ccd-9fb3-31473672a48f-kube-api-access-2v6d2" pod "da3ace09-2b4e-4ccd-9fb3-31473672a48f" (UID: "da3ace09-2b4e-4ccd-9fb3-31473672a48f"). InnerVolumeSpecName "kube-api-access-2v6d2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:02:51.257088 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:51.257061 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39cb8c11-9c39-48ef-b94e-822f423e464b-kube-api-access-pqv5s" pod "39cb8c11-9c39-48ef-b94e-822f423e464b" (UID: "39cb8c11-9c39-48ef-b94e-822f423e464b"). InnerVolumeSpecName "kube-api-access-pqv5s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:02:51.257202 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:51.257179 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/39cb8c11-9c39-48ef-b94e-822f423e464b-maas-api-xks-ca" pod "39cb8c11-9c39-48ef-b94e-822f423e464b" (UID: "39cb8c11-9c39-48ef-b94e-822f423e464b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:02:51.305076 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:51.305017 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:02:51.396138 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:51.395266 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pqv5s\" (UniqueName: \"kubernetes.io/projected/39cb8c11-9c39-48ef-b94e-822f423e464b-kube-api-access-pqv5s\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:02:51.396138 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:51.395312 4927 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39cb8c11-9c39-48ef-b94e-822f423e464b-maas-api-xks-ca\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:02:51.396138 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:51.395344 4927 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da3ace09-2b4e-4ccd-9fb3-31473672a48f-maas-api-xks-ca\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:02:51.396138 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:51.395361 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2v6d2\" (UniqueName: \"kubernetes.io/projected/da3ace09-2b4e-4ccd-9fb3-31473672a48f-kube-api-access-2v6d2\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:02:51.940038 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:51.939999 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd61788b-nmjjt" event={"ID":"da3ace09-2b4e-4ccd-9fb3-31473672a48f","Type":"ContainerDied","Data":"57b57ec82af8fb3defd2d1dc5d6850f043b52efeb43895d4939e44fb75bbe73b"} Oct 31 00:02:51.940206 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:51.940064 4927 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="57b57ec82af8fb3defd2d1dc5d6850f043b52efeb43895d4939e44fb75bbe73b" Oct 31 00:02:51.940206 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:51.940007 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd61788b-nmjjt" Oct 31 00:02:51.941800 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:51.941774 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98d91d7d-b7th9" Oct 31 00:02:51.941943 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:51.941812 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98d91d7d-b7th9" event={"ID":"39cb8c11-9c39-48ef-b94e-822f423e464b","Type":"ContainerDied","Data":"8f4ba2946ed5283c57650369f52b17c94296e1cfdf412c7e0063c319d6032b1c"} Oct 31 00:02:51.941943 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:51.941859 4927 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8f4ba2946ed5283c57650369f52b17c94296e1cfdf412c7e0063c319d6032b1c" Oct 31 00:02:53.352542 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:53.352511 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d93b6188-cbrjh"] Oct 31 00:02:53.353190 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:53.353175 4927 cpu_manager.go:414] "containerMap: removing container" podUID="da3ace09-2b4e-4ccd-9fb3-31473672a48f" containerName="revoke-keys" Oct 31 00:02:53.353233 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:53.353190 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da3ace09-2b4e-4ccd-9fb3-31473672a48f" containerName="revoke-keys" Oct 31 00:02:53.353233 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:53.353208 4927 cpu_manager.go:414] "containerMap: removing container" podUID="39cb8c11-9c39-48ef-b94e-822f423e464b" containerName="revoke-keys" Oct 31 00:02:53.353233 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:53.353214 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39cb8c11-9c39-48ef-b94e-822f423e464b" containerName="revoke-keys" Oct 31 00:02:53.353358 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:53.353320 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="39cb8c11-9c39-48ef-b94e-822f423e464b" containerName="revoke-keys" Oct 31 00:02:53.353358 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:53.353356 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="da3ace09-2b4e-4ccd-9fb3-31473672a48f" containerName="revoke-keys" Oct 31 00:02:53.356616 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:53.356599 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d93b6188-cbrjh" Oct 31 00:02:53.414538 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:53.414505 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d93b6188-cbrjh"] Oct 31 00:02:53.422608 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:53.422576 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ccaf3489-e47d-4dfb-8d46-aa559aeeeb5b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d93b6188-cbrjh\" (UID: \"ccaf3489-e47d-4dfb-8d46-aa559aeeeb5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d93b6188-cbrjh" Oct 31 00:02:53.422770 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:53.422629 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58mbm\" (UniqueName: \"kubernetes.io/projected/ccaf3489-e47d-4dfb-8d46-aa559aeeeb5b-kube-api-access-58mbm\") pod \"maas-api-revoke-sub-d93b6188-cbrjh\" (UID: \"ccaf3489-e47d-4dfb-8d46-aa559aeeeb5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d93b6188-cbrjh" Oct 31 00:02:53.422865 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:53.422838 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ccaf3489-e47d-4dfb-8d46-aa559aeeeb5b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d93b6188-cbrjh\" (UID: \"ccaf3489-e47d-4dfb-8d46-aa559aeeeb5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d93b6188-cbrjh" Oct 31 00:02:53.459214 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:53.459182 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:02:53.459214 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:53.459190 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4fdzs\"" Oct 31 00:02:53.524498 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:53.524463 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ccaf3489-e47d-4dfb-8d46-aa559aeeeb5b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d93b6188-cbrjh\" (UID: \"ccaf3489-e47d-4dfb-8d46-aa559aeeeb5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d93b6188-cbrjh" Oct 31 00:02:53.524670 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:53.524523 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58mbm\" (UniqueName: \"kubernetes.io/projected/ccaf3489-e47d-4dfb-8d46-aa559aeeeb5b-kube-api-access-58mbm\") pod \"maas-api-revoke-sub-d93b6188-cbrjh\" (UID: \"ccaf3489-e47d-4dfb-8d46-aa559aeeeb5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d93b6188-cbrjh" Oct 31 00:02:53.524670 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:53.524645 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ccaf3489-e47d-4dfb-8d46-aa559aeeeb5b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d93b6188-cbrjh\" (UID: \"ccaf3489-e47d-4dfb-8d46-aa559aeeeb5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d93b6188-cbrjh" Oct 31 00:02:53.525187 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:53.525167 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ccaf3489-e47d-4dfb-8d46-aa559aeeeb5b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d93b6188-cbrjh\" (UID: \"ccaf3489-e47d-4dfb-8d46-aa559aeeeb5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d93b6188-cbrjh" Oct 31 00:02:53.527040 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:53.527018 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ccaf3489-e47d-4dfb-8d46-aa559aeeeb5b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d93b6188-cbrjh\" (UID: \"ccaf3489-e47d-4dfb-8d46-aa559aeeeb5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d93b6188-cbrjh" Oct 31 00:02:53.588172 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:53.588138 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-58mbm\" (UniqueName: \"kubernetes.io/projected/ccaf3489-e47d-4dfb-8d46-aa559aeeeb5b-kube-api-access-58mbm\") pod \"maas-api-revoke-sub-d93b6188-cbrjh\" (UID: \"ccaf3489-e47d-4dfb-8d46-aa559aeeeb5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d93b6188-cbrjh" Oct 31 00:02:53.671923 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:53.671888 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d93b6188-cbrjh" Oct 31 00:02:54.063053 ip-10-0-141-180 kubenswrapper[4927]: W1031 00:02:54.063012 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podccaf3489_e47d_4dfb_8d46_aa559aeeeb5b.slice/crio-6298113858fa158f75673a089330d7976780d8fca789541d940574fcd2394ac5 WatchSource:0}: Error finding container 6298113858fa158f75673a089330d7976780d8fca789541d940574fcd2394ac5: Status 404 returned error can't find the container with id 6298113858fa158f75673a089330d7976780d8fca789541d940574fcd2394ac5 Oct 31 00:02:54.063856 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:54.063831 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d93b6188-cbrjh"] Oct 31 00:02:54.972364 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:54.972303 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d93b6188-cbrjh" event={"ID":"ccaf3489-e47d-4dfb-8d46-aa559aeeeb5b","Type":"ContainerStarted","Data":"86ba12939bb302695c3e154c84a46972603c8532c1c3ee31f5693c9eb7cbdebb"} Oct 31 00:02:54.972851 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:54.972386 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d93b6188-cbrjh" event={"ID":"ccaf3489-e47d-4dfb-8d46-aa559aeeeb5b","Type":"ContainerStarted","Data":"6298113858fa158f75673a089330d7976780d8fca789541d940574fcd2394ac5"} Oct 31 00:02:55.007056 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:02:55.007028 4927 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podccaf3489_e47d_4dfb_8d46_aa559aeeeb5b.slice/crio-86ba12939bb302695c3e154c84a46972603c8532c1c3ee31f5693c9eb7cbdebb.scope\": RecentStats: unable to find data in memory cache]" Oct 31 00:02:55.088149 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:55.088116 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-499c7567-66x7m"] Oct 31 00:02:55.092561 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:55.092540 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-499c7567-66x7m" Oct 31 00:02:55.145302 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:55.145268 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5hvx\" (UniqueName: \"kubernetes.io/projected/4178174c-aa4a-4bcf-8147-6f9d5987d9ca-kube-api-access-r5hvx\") pod \"maas-api-revoke-sub-499c7567-66x7m\" (UID: \"4178174c-aa4a-4bcf-8147-6f9d5987d9ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-499c7567-66x7m" Oct 31 00:02:55.145614 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:55.145588 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4178174c-aa4a-4bcf-8147-6f9d5987d9ca-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-499c7567-66x7m\" (UID: \"4178174c-aa4a-4bcf-8147-6f9d5987d9ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-499c7567-66x7m" Oct 31 00:02:55.145776 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:55.145628 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4178174c-aa4a-4bcf-8147-6f9d5987d9ca-maas-api-xks-ca\") pod \"maas-api-revoke-sub-499c7567-66x7m\" (UID: \"4178174c-aa4a-4bcf-8147-6f9d5987d9ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-499c7567-66x7m" Oct 31 00:02:55.149345 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:55.149278 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d93b6188-cbrjh" podStartSLOduration=2.149261014 podStartE2EDuration="2.149261014s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:02:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:02:55.132633903 +0000 UTC m=+1288.168964724" watchObservedRunningTime="2026-10-31 00:02:55.149261014 +0000 UTC m=+1288.185591839" Oct 31 00:02:55.151968 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:55.151946 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-499c7567-66x7m"] Oct 31 00:02:55.247220 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:55.247127 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4178174c-aa4a-4bcf-8147-6f9d5987d9ca-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-499c7567-66x7m\" (UID: \"4178174c-aa4a-4bcf-8147-6f9d5987d9ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-499c7567-66x7m" Oct 31 00:02:55.247220 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:55.247180 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4178174c-aa4a-4bcf-8147-6f9d5987d9ca-maas-api-xks-ca\") pod \"maas-api-revoke-sub-499c7567-66x7m\" (UID: \"4178174c-aa4a-4bcf-8147-6f9d5987d9ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-499c7567-66x7m" Oct 31 00:02:55.247455 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:55.247416 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5hvx\" (UniqueName: \"kubernetes.io/projected/4178174c-aa4a-4bcf-8147-6f9d5987d9ca-kube-api-access-r5hvx\") pod \"maas-api-revoke-sub-499c7567-66x7m\" (UID: \"4178174c-aa4a-4bcf-8147-6f9d5987d9ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-499c7567-66x7m" Oct 31 00:02:55.247915 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:55.247888 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4178174c-aa4a-4bcf-8147-6f9d5987d9ca-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-499c7567-66x7m\" (UID: \"4178174c-aa4a-4bcf-8147-6f9d5987d9ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-499c7567-66x7m" Oct 31 00:02:55.249648 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:55.249625 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4178174c-aa4a-4bcf-8147-6f9d5987d9ca-maas-api-xks-ca\") pod \"maas-api-revoke-sub-499c7567-66x7m\" (UID: \"4178174c-aa4a-4bcf-8147-6f9d5987d9ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-499c7567-66x7m" Oct 31 00:02:55.310979 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:55.310936 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5hvx\" (UniqueName: \"kubernetes.io/projected/4178174c-aa4a-4bcf-8147-6f9d5987d9ca-kube-api-access-r5hvx\") pod \"maas-api-revoke-sub-499c7567-66x7m\" (UID: \"4178174c-aa4a-4bcf-8147-6f9d5987d9ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-499c7567-66x7m" Oct 31 00:02:55.407388 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:55.407313 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-499c7567-66x7m" Oct 31 00:02:55.826907 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:55.826867 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-499c7567-66x7m"] Oct 31 00:02:55.981980 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:55.981939 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-499c7567-66x7m" event={"ID":"4178174c-aa4a-4bcf-8147-6f9d5987d9ca","Type":"ContainerStarted","Data":"8fe8633d3bdcbff9d0b8807d87e9937f2ac9957d24209ed754562a3fbcc930dc"} Oct 31 00:02:55.983617 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:55.983591 4927 generic.go:356] "Generic (PLEG): container finished" podID="ccaf3489-e47d-4dfb-8d46-aa559aeeeb5b" containerID="86ba12939bb302695c3e154c84a46972603c8532c1c3ee31f5693c9eb7cbdebb" exitCode=0 Oct 31 00:02:55.983747 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:55.983672 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d93b6188-cbrjh" event={"ID":"ccaf3489-e47d-4dfb-8d46-aa559aeeeb5b","Type":"ContainerDied","Data":"86ba12939bb302695c3e154c84a46972603c8532c1c3ee31f5693c9eb7cbdebb"} Oct 31 00:02:56.004728 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:56.004694 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:02:56.995380 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:56.995279 4927 generic.go:356] "Generic (PLEG): container finished" podID="4178174c-aa4a-4bcf-8147-6f9d5987d9ca" containerID="72521bbea75240eedc7a0d0a5fd707ca08fb713c932ac30a5a205af782f1902d" exitCode=0 Oct 31 00:02:56.995380 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:56.995368 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-499c7567-66x7m" event={"ID":"4178174c-aa4a-4bcf-8147-6f9d5987d9ca","Type":"ContainerDied","Data":"72521bbea75240eedc7a0d0a5fd707ca08fb713c932ac30a5a205af782f1902d"} Oct 31 00:02:57.114376 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:57.109506 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-18e00928-dv47f"] Oct 31 00:02:57.118007 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:57.117975 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-18e00928-dv47f"] Oct 31 00:02:57.158296 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:57.158266 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d93b6188-cbrjh" Oct 31 00:02:57.177488 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:57.177461 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ccaf3489-e47d-4dfb-8d46-aa559aeeeb5b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ccaf3489-e47d-4dfb-8d46-aa559aeeeb5b-maas-api-xks-ca\") pod \"ccaf3489-e47d-4dfb-8d46-aa559aeeeb5b\" (UID: \"ccaf3489-e47d-4dfb-8d46-aa559aeeeb5b\") " Oct 31 00:02:57.177664 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:57.177604 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ccaf3489-e47d-4dfb-8d46-aa559aeeeb5b-kube-api-access-58mbm\" (UniqueName: \"kubernetes.io/projected/ccaf3489-e47d-4dfb-8d46-aa559aeeeb5b-kube-api-access-58mbm\") pod \"ccaf3489-e47d-4dfb-8d46-aa559aeeeb5b\" (UID: \"ccaf3489-e47d-4dfb-8d46-aa559aeeeb5b\") " Oct 31 00:02:57.177724 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:57.177711 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ccaf3489-e47d-4dfb-8d46-aa559aeeeb5b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ccaf3489-e47d-4dfb-8d46-aa559aeeeb5b-maas-api-openshift-ca\") pod \"ccaf3489-e47d-4dfb-8d46-aa559aeeeb5b\" (UID: \"ccaf3489-e47d-4dfb-8d46-aa559aeeeb5b\") " Oct 31 00:02:57.178107 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:57.178063 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ccaf3489-e47d-4dfb-8d46-aa559aeeeb5b-maas-api-openshift-ca" pod "ccaf3489-e47d-4dfb-8d46-aa559aeeeb5b" (UID: "ccaf3489-e47d-4dfb-8d46-aa559aeeeb5b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:02:57.178412 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:57.178386 4927 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ccaf3489-e47d-4dfb-8d46-aa559aeeeb5b-maas-api-openshift-ca\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:02:57.179723 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:57.179693 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ccaf3489-e47d-4dfb-8d46-aa559aeeeb5b-maas-api-xks-ca" pod "ccaf3489-e47d-4dfb-8d46-aa559aeeeb5b" (UID: "ccaf3489-e47d-4dfb-8d46-aa559aeeeb5b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:02:57.179821 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:57.179768 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ccaf3489-e47d-4dfb-8d46-aa559aeeeb5b-kube-api-access-58mbm" pod "ccaf3489-e47d-4dfb-8d46-aa559aeeeb5b" (UID: "ccaf3489-e47d-4dfb-8d46-aa559aeeeb5b"). InnerVolumeSpecName "kube-api-access-58mbm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:02:57.279466 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:57.279365 4927 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ccaf3489-e47d-4dfb-8d46-aa559aeeeb5b-maas-api-xks-ca\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:02:57.279466 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:57.279399 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-58mbm\" (UniqueName: \"kubernetes.io/projected/ccaf3489-e47d-4dfb-8d46-aa559aeeeb5b-kube-api-access-58mbm\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:02:58.017766 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:58.017732 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d93b6188-cbrjh" Oct 31 00:02:58.018221 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:58.017728 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d93b6188-cbrjh" event={"ID":"ccaf3489-e47d-4dfb-8d46-aa559aeeeb5b","Type":"ContainerDied","Data":"6298113858fa158f75673a089330d7976780d8fca789541d940574fcd2394ac5"} Oct 31 00:02:58.018221 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:58.017913 4927 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6298113858fa158f75673a089330d7976780d8fca789541d940574fcd2394ac5" Oct 31 00:02:58.207585 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:58.207550 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-499c7567-66x7m" Oct 31 00:02:58.293821 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:58.293736 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4178174c-aa4a-4bcf-8147-6f9d5987d9ca-kube-api-access-r5hvx\" (UniqueName: \"kubernetes.io/projected/4178174c-aa4a-4bcf-8147-6f9d5987d9ca-kube-api-access-r5hvx\") pod \"4178174c-aa4a-4bcf-8147-6f9d5987d9ca\" (UID: \"4178174c-aa4a-4bcf-8147-6f9d5987d9ca\") " Oct 31 00:02:58.293981 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:58.293941 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4178174c-aa4a-4bcf-8147-6f9d5987d9ca-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4178174c-aa4a-4bcf-8147-6f9d5987d9ca-maas-api-openshift-ca\") pod \"4178174c-aa4a-4bcf-8147-6f9d5987d9ca\" (UID: \"4178174c-aa4a-4bcf-8147-6f9d5987d9ca\") " Oct 31 00:02:58.294159 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:58.294140 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4178174c-aa4a-4bcf-8147-6f9d5987d9ca-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4178174c-aa4a-4bcf-8147-6f9d5987d9ca-maas-api-xks-ca\") pod \"4178174c-aa4a-4bcf-8147-6f9d5987d9ca\" (UID: \"4178174c-aa4a-4bcf-8147-6f9d5987d9ca\") " Oct 31 00:02:58.294373 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:58.294310 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4178174c-aa4a-4bcf-8147-6f9d5987d9ca-maas-api-openshift-ca" pod "4178174c-aa4a-4bcf-8147-6f9d5987d9ca" (UID: "4178174c-aa4a-4bcf-8147-6f9d5987d9ca"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:02:58.294764 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:58.294739 4927 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4178174c-aa4a-4bcf-8147-6f9d5987d9ca-maas-api-openshift-ca\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:02:58.296015 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:58.295992 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4178174c-aa4a-4bcf-8147-6f9d5987d9ca-kube-api-access-r5hvx" pod "4178174c-aa4a-4bcf-8147-6f9d5987d9ca" (UID: "4178174c-aa4a-4bcf-8147-6f9d5987d9ca"). InnerVolumeSpecName "kube-api-access-r5hvx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:02:58.296259 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:58.296231 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4178174c-aa4a-4bcf-8147-6f9d5987d9ca-maas-api-xks-ca" pod "4178174c-aa4a-4bcf-8147-6f9d5987d9ca" (UID: "4178174c-aa4a-4bcf-8147-6f9d5987d9ca"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:02:58.396257 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:58.396219 4927 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4178174c-aa4a-4bcf-8147-6f9d5987d9ca-maas-api-xks-ca\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:02:58.396257 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:58.396253 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r5hvx\" (UniqueName: \"kubernetes.io/projected/4178174c-aa4a-4bcf-8147-6f9d5987d9ca-kube-api-access-r5hvx\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:02:59.027729 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:59.027699 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-499c7567-66x7m" Oct 31 00:02:59.028138 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:59.027726 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-499c7567-66x7m" event={"ID":"4178174c-aa4a-4bcf-8147-6f9d5987d9ca","Type":"ContainerDied","Data":"8fe8633d3bdcbff9d0b8807d87e9937f2ac9957d24209ed754562a3fbcc930dc"} Oct 31 00:02:59.028138 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:59.027783 4927 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8fe8633d3bdcbff9d0b8807d87e9937f2ac9957d24209ed754562a3fbcc930dc" Oct 31 00:02:59.087565 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:59.087530 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f5417b9-0f02-4589-9953-aff04e83f410" path="/var/lib/kubelet/pods/5f5417b9-0f02-4589-9953-aff04e83f410/volumes" Oct 31 00:02:59.512465 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:59.512425 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-7w9dq"] Oct 31 00:02:59.513446 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:59.513423 4927 cpu_manager.go:414] "containerMap: removing container" podUID="ccaf3489-e47d-4dfb-8d46-aa559aeeeb5b" containerName="revoke-keys" Oct 31 00:02:59.513446 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:59.513445 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ccaf3489-e47d-4dfb-8d46-aa559aeeeb5b" containerName="revoke-keys" Oct 31 00:02:59.513595 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:59.513507 4927 cpu_manager.go:414] "containerMap: removing container" podUID="4178174c-aa4a-4bcf-8147-6f9d5987d9ca" containerName="revoke-keys" Oct 31 00:02:59.513595 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:59.513516 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4178174c-aa4a-4bcf-8147-6f9d5987d9ca" containerName="revoke-keys" Oct 31 00:02:59.513714 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:59.513670 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="ccaf3489-e47d-4dfb-8d46-aa559aeeeb5b" containerName="revoke-keys" Oct 31 00:02:59.513714 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:59.513698 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="4178174c-aa4a-4bcf-8147-6f9d5987d9ca" containerName="revoke-keys" Oct 31 00:02:59.517483 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:59.517464 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-7w9dq" Oct 31 00:02:59.613873 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:59.613835 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d93f8ab4-0aff-4eba-843d-2c07ab53d3bc-workload-socket\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-7w9dq\" (UID: \"d93f8ab4-0aff-4eba-843d-2c07ab53d3bc\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-7w9dq" Oct 31 00:02:59.613873 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:59.613877 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d93f8ab4-0aff-4eba-843d-2c07ab53d3bc-istio-envoy\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-7w9dq\" (UID: \"d93f8ab4-0aff-4eba-843d-2c07ab53d3bc\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-7w9dq" Oct 31 00:02:59.614135 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:59.614019 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d93f8ab4-0aff-4eba-843d-2c07ab53d3bc-istio-data\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-7w9dq\" (UID: \"d93f8ab4-0aff-4eba-843d-2c07ab53d3bc\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-7w9dq" Oct 31 00:02:59.614135 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:59.614078 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d93f8ab4-0aff-4eba-843d-2c07ab53d3bc-workload-certs\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-7w9dq\" (UID: \"d93f8ab4-0aff-4eba-843d-2c07ab53d3bc\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-7w9dq" Oct 31 00:02:59.614135 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:59.614100 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d93f8ab4-0aff-4eba-843d-2c07ab53d3bc-istiod-ca-cert\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-7w9dq\" (UID: \"d93f8ab4-0aff-4eba-843d-2c07ab53d3bc\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-7w9dq" Oct 31 00:02:59.614265 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:59.614153 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d93f8ab4-0aff-4eba-843d-2c07ab53d3bc-istio-podinfo\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-7w9dq\" (UID: \"d93f8ab4-0aff-4eba-843d-2c07ab53d3bc\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-7w9dq" Oct 31 00:02:59.614325 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:59.614310 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tpmjq\" (UniqueName: \"kubernetes.io/projected/d93f8ab4-0aff-4eba-843d-2c07ab53d3bc-kube-api-access-tpmjq\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-7w9dq\" (UID: \"d93f8ab4-0aff-4eba-843d-2c07ab53d3bc\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-7w9dq" Oct 31 00:02:59.614406 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:59.614374 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d93f8ab4-0aff-4eba-843d-2c07ab53d3bc-credential-socket\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-7w9dq\" (UID: \"d93f8ab4-0aff-4eba-843d-2c07ab53d3bc\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-7w9dq" Oct 31 00:02:59.614482 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:59.614468 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d93f8ab4-0aff-4eba-843d-2c07ab53d3bc-istio-token\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-7w9dq\" (UID: \"d93f8ab4-0aff-4eba-843d-2c07ab53d3bc\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-7w9dq" Oct 31 00:02:59.628101 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:59.628066 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-7w9dq"] Oct 31 00:02:59.716061 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:59.716029 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d93f8ab4-0aff-4eba-843d-2c07ab53d3bc-istio-token\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-7w9dq\" (UID: \"d93f8ab4-0aff-4eba-843d-2c07ab53d3bc\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-7w9dq" Oct 31 00:02:59.716267 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:59.716105 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d93f8ab4-0aff-4eba-843d-2c07ab53d3bc-workload-socket\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-7w9dq\" (UID: \"d93f8ab4-0aff-4eba-843d-2c07ab53d3bc\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-7w9dq" Oct 31 00:02:59.716267 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:59.716128 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d93f8ab4-0aff-4eba-843d-2c07ab53d3bc-istio-envoy\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-7w9dq\" (UID: \"d93f8ab4-0aff-4eba-843d-2c07ab53d3bc\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-7w9dq" Oct 31 00:02:59.716267 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:59.716199 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d93f8ab4-0aff-4eba-843d-2c07ab53d3bc-istio-data\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-7w9dq\" (UID: \"d93f8ab4-0aff-4eba-843d-2c07ab53d3bc\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-7w9dq" Oct 31 00:02:59.716267 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:59.716260 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d93f8ab4-0aff-4eba-843d-2c07ab53d3bc-workload-certs\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-7w9dq\" (UID: \"d93f8ab4-0aff-4eba-843d-2c07ab53d3bc\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-7w9dq" Oct 31 00:02:59.716550 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:59.716282 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d93f8ab4-0aff-4eba-843d-2c07ab53d3bc-istiod-ca-cert\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-7w9dq\" (UID: \"d93f8ab4-0aff-4eba-843d-2c07ab53d3bc\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-7w9dq" Oct 31 00:02:59.716550 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:59.716428 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d93f8ab4-0aff-4eba-843d-2c07ab53d3bc-istio-podinfo\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-7w9dq\" (UID: \"d93f8ab4-0aff-4eba-843d-2c07ab53d3bc\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-7w9dq" Oct 31 00:02:59.716550 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:59.716509 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tpmjq\" (UniqueName: \"kubernetes.io/projected/d93f8ab4-0aff-4eba-843d-2c07ab53d3bc-kube-api-access-tpmjq\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-7w9dq\" (UID: \"d93f8ab4-0aff-4eba-843d-2c07ab53d3bc\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-7w9dq" Oct 31 00:02:59.716693 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:59.716567 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d93f8ab4-0aff-4eba-843d-2c07ab53d3bc-credential-socket\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-7w9dq\" (UID: \"d93f8ab4-0aff-4eba-843d-2c07ab53d3bc\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-7w9dq" Oct 31 00:02:59.716693 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:59.716610 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d93f8ab4-0aff-4eba-843d-2c07ab53d3bc-workload-socket\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-7w9dq\" (UID: \"d93f8ab4-0aff-4eba-843d-2c07ab53d3bc\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-7w9dq" Oct 31 00:02:59.716813 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:59.716717 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d93f8ab4-0aff-4eba-843d-2c07ab53d3bc-istio-data\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-7w9dq\" (UID: \"d93f8ab4-0aff-4eba-843d-2c07ab53d3bc\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-7w9dq" Oct 31 00:02:59.716813 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:59.716747 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d93f8ab4-0aff-4eba-843d-2c07ab53d3bc-workload-certs\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-7w9dq\" (UID: \"d93f8ab4-0aff-4eba-843d-2c07ab53d3bc\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-7w9dq" Oct 31 00:02:59.716956 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:59.716936 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d93f8ab4-0aff-4eba-843d-2c07ab53d3bc-istiod-ca-cert\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-7w9dq\" (UID: \"d93f8ab4-0aff-4eba-843d-2c07ab53d3bc\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-7w9dq" Oct 31 00:02:59.717021 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:59.716934 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d93f8ab4-0aff-4eba-843d-2c07ab53d3bc-credential-socket\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-7w9dq\" (UID: \"d93f8ab4-0aff-4eba-843d-2c07ab53d3bc\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-7w9dq" Oct 31 00:02:59.718630 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:59.718611 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d93f8ab4-0aff-4eba-843d-2c07ab53d3bc-istio-envoy\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-7w9dq\" (UID: \"d93f8ab4-0aff-4eba-843d-2c07ab53d3bc\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-7w9dq" Oct 31 00:02:59.719212 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:59.719191 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d93f8ab4-0aff-4eba-843d-2c07ab53d3bc-istio-podinfo\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-7w9dq\" (UID: \"d93f8ab4-0aff-4eba-843d-2c07ab53d3bc\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-7w9dq" Oct 31 00:02:59.781175 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:59.781086 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d93f8ab4-0aff-4eba-843d-2c07ab53d3bc-istio-token\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-7w9dq\" (UID: \"d93f8ab4-0aff-4eba-843d-2c07ab53d3bc\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-7w9dq" Oct 31 00:02:59.782588 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:59.782565 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tpmjq\" (UniqueName: \"kubernetes.io/projected/d93f8ab4-0aff-4eba-843d-2c07ab53d3bc-kube-api-access-tpmjq\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-7w9dq\" (UID: \"d93f8ab4-0aff-4eba-843d-2c07ab53d3bc\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-7w9dq" Oct 31 00:02:59.831523 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:02:59.831489 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-7w9dq" Oct 31 00:03:00.163132 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:00.163098 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-vb9r5"] Oct 31 00:03:00.169403 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:00.169348 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-vb9r5" Oct 31 00:03:00.223682 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:00.223636 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/334f41bf-6d81-474d-bddc-d38c3492d586-workload-socket\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-vb9r5\" (UID: \"334f41bf-6d81-474d-bddc-d38c3492d586\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-vb9r5" Oct 31 00:03:00.223927 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:00.223700 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/334f41bf-6d81-474d-bddc-d38c3492d586-workload-certs\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-vb9r5\" (UID: \"334f41bf-6d81-474d-bddc-d38c3492d586\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-vb9r5" Oct 31 00:03:00.223927 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:00.223740 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/334f41bf-6d81-474d-bddc-d38c3492d586-istio-token\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-vb9r5\" (UID: \"334f41bf-6d81-474d-bddc-d38c3492d586\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-vb9r5" Oct 31 00:03:00.223927 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:00.223769 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x24zl\" (UniqueName: \"kubernetes.io/projected/334f41bf-6d81-474d-bddc-d38c3492d586-kube-api-access-x24zl\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-vb9r5\" (UID: \"334f41bf-6d81-474d-bddc-d38c3492d586\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-vb9r5" Oct 31 00:03:00.223927 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:00.223805 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/334f41bf-6d81-474d-bddc-d38c3492d586-istio-envoy\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-vb9r5\" (UID: \"334f41bf-6d81-474d-bddc-d38c3492d586\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-vb9r5" Oct 31 00:03:00.223927 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:00.223896 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/334f41bf-6d81-474d-bddc-d38c3492d586-credential-socket\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-vb9r5\" (UID: \"334f41bf-6d81-474d-bddc-d38c3492d586\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-vb9r5" Oct 31 00:03:00.224234 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:00.224204 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/334f41bf-6d81-474d-bddc-d38c3492d586-istio-podinfo\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-vb9r5\" (UID: \"334f41bf-6d81-474d-bddc-d38c3492d586\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-vb9r5" Oct 31 00:03:00.224300 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:00.224278 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/334f41bf-6d81-474d-bddc-d38c3492d586-istiod-ca-cert\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-vb9r5\" (UID: \"334f41bf-6d81-474d-bddc-d38c3492d586\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-vb9r5" Oct 31 00:03:00.224405 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:00.224386 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/334f41bf-6d81-474d-bddc-d38c3492d586-istio-data\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-vb9r5\" (UID: \"334f41bf-6d81-474d-bddc-d38c3492d586\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-vb9r5" Oct 31 00:03:00.230706 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:00.230681 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-vb9r5"] Oct 31 00:03:00.327669 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:00.327483 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/334f41bf-6d81-474d-bddc-d38c3492d586-workload-socket\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-vb9r5\" (UID: \"334f41bf-6d81-474d-bddc-d38c3492d586\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-vb9r5" Oct 31 00:03:00.327669 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:00.327542 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/334f41bf-6d81-474d-bddc-d38c3492d586-workload-certs\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-vb9r5\" (UID: \"334f41bf-6d81-474d-bddc-d38c3492d586\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-vb9r5" Oct 31 00:03:00.327669 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:00.327580 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/334f41bf-6d81-474d-bddc-d38c3492d586-istio-token\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-vb9r5\" (UID: \"334f41bf-6d81-474d-bddc-d38c3492d586\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-vb9r5" Oct 31 00:03:00.327669 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:00.327610 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x24zl\" (UniqueName: \"kubernetes.io/projected/334f41bf-6d81-474d-bddc-d38c3492d586-kube-api-access-x24zl\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-vb9r5\" (UID: \"334f41bf-6d81-474d-bddc-d38c3492d586\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-vb9r5" Oct 31 00:03:00.327669 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:00.327645 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/334f41bf-6d81-474d-bddc-d38c3492d586-istio-envoy\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-vb9r5\" (UID: \"334f41bf-6d81-474d-bddc-d38c3492d586\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-vb9r5" Oct 31 00:03:00.327861 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:00.327735 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/334f41bf-6d81-474d-bddc-d38c3492d586-credential-socket\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-vb9r5\" (UID: \"334f41bf-6d81-474d-bddc-d38c3492d586\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-vb9r5" Oct 31 00:03:00.327908 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:00.327893 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/334f41bf-6d81-474d-bddc-d38c3492d586-istio-podinfo\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-vb9r5\" (UID: \"334f41bf-6d81-474d-bddc-d38c3492d586\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-vb9r5" Oct 31 00:03:00.327961 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:00.327946 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/334f41bf-6d81-474d-bddc-d38c3492d586-istiod-ca-cert\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-vb9r5\" (UID: \"334f41bf-6d81-474d-bddc-d38c3492d586\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-vb9r5" Oct 31 00:03:00.328016 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:00.328002 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/334f41bf-6d81-474d-bddc-d38c3492d586-istio-data\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-vb9r5\" (UID: \"334f41bf-6d81-474d-bddc-d38c3492d586\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-vb9r5" Oct 31 00:03:00.328471 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:00.328451 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/334f41bf-6d81-474d-bddc-d38c3492d586-istio-data\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-vb9r5\" (UID: \"334f41bf-6d81-474d-bddc-d38c3492d586\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-vb9r5" Oct 31 00:03:00.328715 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:00.328699 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/334f41bf-6d81-474d-bddc-d38c3492d586-workload-socket\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-vb9r5\" (UID: \"334f41bf-6d81-474d-bddc-d38c3492d586\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-vb9r5" Oct 31 00:03:00.328891 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:00.328875 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/334f41bf-6d81-474d-bddc-d38c3492d586-workload-certs\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-vb9r5\" (UID: \"334f41bf-6d81-474d-bddc-d38c3492d586\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-vb9r5" Oct 31 00:03:00.330482 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:00.330454 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/334f41bf-6d81-474d-bddc-d38c3492d586-istiod-ca-cert\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-vb9r5\" (UID: \"334f41bf-6d81-474d-bddc-d38c3492d586\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-vb9r5" Oct 31 00:03:00.330814 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:00.330698 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/334f41bf-6d81-474d-bddc-d38c3492d586-credential-socket\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-vb9r5\" (UID: \"334f41bf-6d81-474d-bddc-d38c3492d586\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-vb9r5" Oct 31 00:03:00.331672 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:00.331648 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/334f41bf-6d81-474d-bddc-d38c3492d586-istio-envoy\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-vb9r5\" (UID: \"334f41bf-6d81-474d-bddc-d38c3492d586\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-vb9r5" Oct 31 00:03:00.331789 ip-10-0-141-180 kubenswrapper[4927]: W1031 00:03:00.331765 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd93f8ab4_0aff_4eba_843d_2c07ab53d3bc.slice/crio-64d025af31dd233083670a9318d3fd5f0712b40f8d2a86e8b03de7690a4ab324 WatchSource:0}: Error finding container 64d025af31dd233083670a9318d3fd5f0712b40f8d2a86e8b03de7690a4ab324: Status 404 returned error can't find the container with id 64d025af31dd233083670a9318d3fd5f0712b40f8d2a86e8b03de7690a4ab324 Oct 31 00:03:00.332743 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:00.332675 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/334f41bf-6d81-474d-bddc-d38c3492d586-istio-podinfo\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-vb9r5\" (UID: \"334f41bf-6d81-474d-bddc-d38c3492d586\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-vb9r5" Oct 31 00:03:00.334837 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:00.334800 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:03:00.334947 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:00.334867 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:03:00.334947 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:00.334897 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:03:00.336187 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:00.336165 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-7w9dq"] Oct 31 00:03:00.391265 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:00.391236 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/334f41bf-6d81-474d-bddc-d38c3492d586-istio-token\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-vb9r5\" (UID: \"334f41bf-6d81-474d-bddc-d38c3492d586\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-vb9r5" Oct 31 00:03:00.391664 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:00.391645 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x24zl\" (UniqueName: \"kubernetes.io/projected/334f41bf-6d81-474d-bddc-d38c3492d586-kube-api-access-x24zl\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-vb9r5\" (UID: \"334f41bf-6d81-474d-bddc-d38c3492d586\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-vb9r5" Oct 31 00:03:00.484870 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:00.484835 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-vb9r5" Oct 31 00:03:00.608609 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:00.608575 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:03:00.946363 ip-10-0-141-180 kubenswrapper[4927]: W1031 00:03:00.946300 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod334f41bf_6d81_474d_bddc_d38c3492d586.slice/crio-a8faa7e12e482d7a0f93946238c3a4bea11ea9a8989aca687e18df250bd5420c WatchSource:0}: Error finding container a8faa7e12e482d7a0f93946238c3a4bea11ea9a8989aca687e18df250bd5420c: Status 404 returned error can't find the container with id a8faa7e12e482d7a0f93946238c3a4bea11ea9a8989aca687e18df250bd5420c Oct 31 00:03:00.947707 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:00.947680 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-vb9r5"] Oct 31 00:03:00.948989 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:00.948956 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:03:00.949076 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:00.949039 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:03:00.949160 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:00.949083 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:03:01.067427 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:01.067296 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-vb9r5" event={"ID":"334f41bf-6d81-474d-bddc-d38c3492d586","Type":"ContainerStarted","Data":"a8faa7e12e482d7a0f93946238c3a4bea11ea9a8989aca687e18df250bd5420c"} Oct 31 00:03:01.069240 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:01.069218 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-7w9dq" event={"ID":"d93f8ab4-0aff-4eba-843d-2c07ab53d3bc","Type":"ContainerStarted","Data":"7107ae31cd4664933000482cf304270bdb91abc3d8eba2b198d7ac7eb4150851"} Oct 31 00:03:01.069365 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:01.069256 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-7w9dq" event={"ID":"d93f8ab4-0aff-4eba-843d-2c07ab53d3bc","Type":"ContainerStarted","Data":"64d025af31dd233083670a9318d3fd5f0712b40f8d2a86e8b03de7690a4ab324"} Oct 31 00:03:01.219412 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:01.201494 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-7w9dq" podStartSLOduration=2.20147896 podStartE2EDuration="2.20147896s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:02:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:03:01.198599009 +0000 UTC m=+1294.234929833" watchObservedRunningTime="2026-10-31 00:03:01.20147896 +0000 UTC m=+1294.237809784" Oct 31 00:03:01.856697 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:01.832784 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-7w9dq" Oct 31 00:03:01.856697 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:01.841466 4927 patch_prober.go:29] interesting pod/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-7w9dq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body= Oct 31 00:03:01.856697 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:01.841531 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-7w9dq" podUID="d93f8ab4-0aff-4eba-843d-2c07ab53d3bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" Oct 31 00:03:01.963240 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:01.963198 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dcc30ad5-tjf6s"] Oct 31 00:03:01.972284 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:01.972258 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcc30ad5-tjf6s" Oct 31 00:03:02.043358 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:02.033762 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4fdzs\"" Oct 31 00:03:02.043358 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:02.034512 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:03:02.073540 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:02.073482 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dcc30ad5-tjf6s"] Oct 31 00:03:02.078285 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:02.077570 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ada7676a-5992-46c1-8a41-870d23a74253-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dcc30ad5-tjf6s\" (UID: \"ada7676a-5992-46c1-8a41-870d23a74253\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcc30ad5-tjf6s" Oct 31 00:03:02.078285 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:02.077689 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gz49n\" (UniqueName: \"kubernetes.io/projected/ada7676a-5992-46c1-8a41-870d23a74253-kube-api-access-gz49n\") pod \"maas-api-revoke-sub-dcc30ad5-tjf6s\" (UID: \"ada7676a-5992-46c1-8a41-870d23a74253\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcc30ad5-tjf6s" Oct 31 00:03:02.078285 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:02.077802 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ada7676a-5992-46c1-8a41-870d23a74253-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dcc30ad5-tjf6s\" (UID: \"ada7676a-5992-46c1-8a41-870d23a74253\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcc30ad5-tjf6s" Oct 31 00:03:02.093211 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:02.093172 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-vb9r5" event={"ID":"334f41bf-6d81-474d-bddc-d38c3492d586","Type":"ContainerStarted","Data":"b2188dcfc96b95e4deeaf72da6a224c424e584af088e805d70914d27730aae5e"} Oct 31 00:03:02.187371 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:02.187313 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ada7676a-5992-46c1-8a41-870d23a74253-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dcc30ad5-tjf6s\" (UID: \"ada7676a-5992-46c1-8a41-870d23a74253\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcc30ad5-tjf6s" Oct 31 00:03:02.187712 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:02.187685 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gz49n\" (UniqueName: \"kubernetes.io/projected/ada7676a-5992-46c1-8a41-870d23a74253-kube-api-access-gz49n\") pod \"maas-api-revoke-sub-dcc30ad5-tjf6s\" (UID: \"ada7676a-5992-46c1-8a41-870d23a74253\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcc30ad5-tjf6s" Oct 31 00:03:02.188664 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:02.188642 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ada7676a-5992-46c1-8a41-870d23a74253-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dcc30ad5-tjf6s\" (UID: \"ada7676a-5992-46c1-8a41-870d23a74253\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcc30ad5-tjf6s" Oct 31 00:03:02.189860 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:02.189833 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ada7676a-5992-46c1-8a41-870d23a74253-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dcc30ad5-tjf6s\" (UID: \"ada7676a-5992-46c1-8a41-870d23a74253\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcc30ad5-tjf6s" Oct 31 00:03:02.194671 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:02.194650 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ada7676a-5992-46c1-8a41-870d23a74253-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dcc30ad5-tjf6s\" (UID: \"ada7676a-5992-46c1-8a41-870d23a74253\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcc30ad5-tjf6s" Oct 31 00:03:02.274356 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:02.274141 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gz49n\" (UniqueName: \"kubernetes.io/projected/ada7676a-5992-46c1-8a41-870d23a74253-kube-api-access-gz49n\") pod \"maas-api-revoke-sub-dcc30ad5-tjf6s\" (UID: \"ada7676a-5992-46c1-8a41-870d23a74253\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcc30ad5-tjf6s" Oct 31 00:03:02.275069 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:02.275014 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-vb9r5" podStartSLOduration=2.274996915 podStartE2EDuration="2.274996915s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:03:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:03:02.248832287 +0000 UTC m=+1295.285163128" watchObservedRunningTime="2026-10-31 00:03:02.274996915 +0000 UTC m=+1295.311327739" Oct 31 00:03:02.290967 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:02.289456 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcc30ad5-tjf6s" Oct 31 00:03:02.490187 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:02.489659 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-vb9r5" Oct 31 00:03:02.495422 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:02.494268 4927 patch_prober.go:29] interesting pod/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-vb9r5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body= Oct 31 00:03:02.495422 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:02.494379 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-vb9r5" podUID="334f41bf-6d81-474d-bddc-d38c3492d586" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" Oct 31 00:03:02.874006 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:02.870247 4927 patch_prober.go:29] interesting pod/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-7w9dq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body= Oct 31 00:03:02.874006 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:02.870315 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-7w9dq" podUID="d93f8ab4-0aff-4eba-843d-2c07ab53d3bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" Oct 31 00:03:02.881961 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:02.881932 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dcc30ad5-tjf6s"] Oct 31 00:03:03.115863 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:03.115778 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcc30ad5-tjf6s" event={"ID":"ada7676a-5992-46c1-8a41-870d23a74253","Type":"ContainerStarted","Data":"2de2822c82e2240a51776ea30b3c840780dc4e037f62f1107fb463253816a6ef"} Oct 31 00:03:03.486084 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:03.486044 4927 patch_prober.go:29] interesting pod/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-vb9r5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body= Oct 31 00:03:03.486648 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:03.486103 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-vb9r5" podUID="334f41bf-6d81-474d-bddc-d38c3492d586" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" Oct 31 00:03:03.836358 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:03.832760 4927 patch_prober.go:29] interesting pod/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-7w9dq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body= Oct 31 00:03:03.836358 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:03.832819 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-7w9dq" podUID="d93f8ab4-0aff-4eba-843d-2c07ab53d3bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" Oct 31 00:03:04.171725 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:04.171685 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcc30ad5-tjf6s" event={"ID":"ada7676a-5992-46c1-8a41-870d23a74253","Type":"ContainerStarted","Data":"d3ffee3c871aaf87297ff91c164cbe99ed4c4463c0dc38429cc3d8b494b64d7e"} Oct 31 00:03:04.369090 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:04.368991 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79cd756e-jfn6r"] Oct 31 00:03:04.373527 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:04.373507 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79cd756e-jfn6r" Oct 31 00:03:04.393481 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:04.393446 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b3f747b-72be-4a13-a670-e150d716f39e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79cd756e-jfn6r\" (UID: \"6b3f747b-72be-4a13-a670-e150d716f39e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79cd756e-jfn6r" Oct 31 00:03:04.393639 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:04.393557 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b3f747b-72be-4a13-a670-e150d716f39e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79cd756e-jfn6r\" (UID: \"6b3f747b-72be-4a13-a670-e150d716f39e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79cd756e-jfn6r" Oct 31 00:03:04.393698 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:04.393690 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c6ffz\" (UniqueName: \"kubernetes.io/projected/6b3f747b-72be-4a13-a670-e150d716f39e-kube-api-access-c6ffz\") pod \"maas-api-revoke-sub-79cd756e-jfn6r\" (UID: \"6b3f747b-72be-4a13-a670-e150d716f39e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79cd756e-jfn6r" Oct 31 00:03:04.446867 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:04.446828 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79cd756e-jfn6r"] Oct 31 00:03:04.487720 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:04.486031 4927 patch_prober.go:29] interesting pod/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-vb9r5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body= Oct 31 00:03:04.487720 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:04.486095 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-vb9r5" podUID="334f41bf-6d81-474d-bddc-d38c3492d586" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" Oct 31 00:03:04.495056 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:04.495021 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b3f747b-72be-4a13-a670-e150d716f39e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79cd756e-jfn6r\" (UID: \"6b3f747b-72be-4a13-a670-e150d716f39e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79cd756e-jfn6r" Oct 31 00:03:04.495237 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:04.495066 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b3f747b-72be-4a13-a670-e150d716f39e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79cd756e-jfn6r\" (UID: \"6b3f747b-72be-4a13-a670-e150d716f39e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79cd756e-jfn6r" Oct 31 00:03:04.495237 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:04.495199 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6ffz\" (UniqueName: \"kubernetes.io/projected/6b3f747b-72be-4a13-a670-e150d716f39e-kube-api-access-c6ffz\") pod \"maas-api-revoke-sub-79cd756e-jfn6r\" (UID: \"6b3f747b-72be-4a13-a670-e150d716f39e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79cd756e-jfn6r" Oct 31 00:03:04.496116 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:04.496087 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b3f747b-72be-4a13-a670-e150d716f39e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79cd756e-jfn6r\" (UID: \"6b3f747b-72be-4a13-a670-e150d716f39e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79cd756e-jfn6r" Oct 31 00:03:04.498603 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:04.498575 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b3f747b-72be-4a13-a670-e150d716f39e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79cd756e-jfn6r\" (UID: \"6b3f747b-72be-4a13-a670-e150d716f39e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79cd756e-jfn6r" Oct 31 00:03:04.575296 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:04.575264 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6ffz\" (UniqueName: \"kubernetes.io/projected/6b3f747b-72be-4a13-a670-e150d716f39e-kube-api-access-c6ffz\") pod \"maas-api-revoke-sub-79cd756e-jfn6r\" (UID: \"6b3f747b-72be-4a13-a670-e150d716f39e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79cd756e-jfn6r" Oct 31 00:03:04.688888 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:04.688534 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79cd756e-jfn6r" Oct 31 00:03:04.833160 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:04.832773 4927 patch_prober.go:29] interesting pod/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-7w9dq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body= Oct 31 00:03:04.833160 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:04.832835 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-7w9dq" podUID="d93f8ab4-0aff-4eba-843d-2c07ab53d3bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" Oct 31 00:03:05.180771 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:05.180738 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79cd756e-jfn6r"] Oct 31 00:03:05.193903 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:05.193863 4927 generic.go:356] "Generic (PLEG): container finished" podID="ada7676a-5992-46c1-8a41-870d23a74253" containerID="d3ffee3c871aaf87297ff91c164cbe99ed4c4463c0dc38429cc3d8b494b64d7e" exitCode=0 Oct 31 00:03:05.194280 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:05.194255 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcc30ad5-tjf6s" event={"ID":"ada7676a-5992-46c1-8a41-870d23a74253","Type":"ContainerDied","Data":"d3ffee3c871aaf87297ff91c164cbe99ed4c4463c0dc38429cc3d8b494b64d7e"} Oct 31 00:03:05.284195 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:05.284152 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:03:05.578863 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:05.526911 4927 patch_prober.go:29] interesting pod/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-vb9r5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body= Oct 31 00:03:05.578863 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:05.526975 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-vb9r5" podUID="334f41bf-6d81-474d-bddc-d38c3492d586" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" Oct 31 00:03:05.769719 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:05.769682 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcc30ad5-tjf6s" Oct 31 00:03:05.853400 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:05.846444 4927 patch_prober.go:29] interesting pod/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-7w9dq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body= Oct 31 00:03:05.853400 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:05.846504 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-7w9dq" podUID="d93f8ab4-0aff-4eba-843d-2c07ab53d3bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" Oct 31 00:03:05.877669 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:05.877632 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ada7676a-5992-46c1-8a41-870d23a74253-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ada7676a-5992-46c1-8a41-870d23a74253-maas-api-openshift-ca\") pod \"ada7676a-5992-46c1-8a41-870d23a74253\" (UID: \"ada7676a-5992-46c1-8a41-870d23a74253\") " Oct 31 00:03:05.877837 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:05.877710 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ada7676a-5992-46c1-8a41-870d23a74253-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ada7676a-5992-46c1-8a41-870d23a74253-maas-api-xks-ca\") pod \"ada7676a-5992-46c1-8a41-870d23a74253\" (UID: \"ada7676a-5992-46c1-8a41-870d23a74253\") " Oct 31 00:03:05.877900 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:05.877887 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ada7676a-5992-46c1-8a41-870d23a74253-kube-api-access-gz49n\" (UniqueName: \"kubernetes.io/projected/ada7676a-5992-46c1-8a41-870d23a74253-kube-api-access-gz49n\") pod \"ada7676a-5992-46c1-8a41-870d23a74253\" (UID: \"ada7676a-5992-46c1-8a41-870d23a74253\") " Oct 31 00:03:05.878046 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:05.878023 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ada7676a-5992-46c1-8a41-870d23a74253-maas-api-openshift-ca" pod "ada7676a-5992-46c1-8a41-870d23a74253" (UID: "ada7676a-5992-46c1-8a41-870d23a74253"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:03:05.878386 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:05.878364 4927 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ada7676a-5992-46c1-8a41-870d23a74253-maas-api-openshift-ca\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:03:05.879831 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:05.879802 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ada7676a-5992-46c1-8a41-870d23a74253-maas-api-xks-ca" pod "ada7676a-5992-46c1-8a41-870d23a74253" (UID: "ada7676a-5992-46c1-8a41-870d23a74253"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:03:05.879928 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:05.879897 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ada7676a-5992-46c1-8a41-870d23a74253-kube-api-access-gz49n" pod "ada7676a-5992-46c1-8a41-870d23a74253" (UID: "ada7676a-5992-46c1-8a41-870d23a74253"). InnerVolumeSpecName "kube-api-access-gz49n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:03:05.980200 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:05.980163 4927 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ada7676a-5992-46c1-8a41-870d23a74253-maas-api-xks-ca\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:03:05.980200 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:05.980200 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gz49n\" (UniqueName: \"kubernetes.io/projected/ada7676a-5992-46c1-8a41-870d23a74253-kube-api-access-gz49n\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:03:06.226358 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:06.210958 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcc30ad5-tjf6s" event={"ID":"ada7676a-5992-46c1-8a41-870d23a74253","Type":"ContainerDied","Data":"2de2822c82e2240a51776ea30b3c840780dc4e037f62f1107fb463253816a6ef"} Oct 31 00:03:06.226358 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:06.211041 4927 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2de2822c82e2240a51776ea30b3c840780dc4e037f62f1107fb463253816a6ef" Oct 31 00:03:06.226358 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:06.211141 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcc30ad5-tjf6s" Oct 31 00:03:06.235353 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:06.231447 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79cd756e-jfn6r" event={"ID":"6b3f747b-72be-4a13-a670-e150d716f39e","Type":"ContainerStarted","Data":"80f0db48ea0e62be8d260e2ccc7b732ea90b42c34fc00bdba30e4f6bfa502288"} Oct 31 00:03:06.235353 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:06.231520 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79cd756e-jfn6r" event={"ID":"6b3f747b-72be-4a13-a670-e150d716f39e","Type":"ContainerStarted","Data":"daf8e555e326ccfa9dd6bc78003c8f4b76ab4edd70e15133c4e43dfd2f89c294"} Oct 31 00:03:06.375539 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:06.375314 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79cd756e-jfn6r" podStartSLOduration=2.375294433 podStartE2EDuration="2.375294433s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:03:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:03:06.371766095 +0000 UTC m=+1299.408096948" watchObservedRunningTime="2026-10-31 00:03:06.375294433 +0000 UTC m=+1299.411625256" Oct 31 00:03:06.485839 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:06.485755 4927 patch_prober.go:29] interesting pod/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-vb9r5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body= Oct 31 00:03:06.485839 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:06.485811 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-vb9r5" podUID="334f41bf-6d81-474d-bddc-d38c3492d586" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" Oct 31 00:03:06.857150 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:06.855636 4927 patch_prober.go:29] interesting pod/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-7w9dq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body= Oct 31 00:03:06.857150 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:06.855710 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-7w9dq" podUID="d93f8ab4-0aff-4eba-843d-2c07ab53d3bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" Oct 31 00:03:07.259463 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:07.259423 4927 generic.go:356] "Generic (PLEG): container finished" podID="6b3f747b-72be-4a13-a670-e150d716f39e" containerID="80f0db48ea0e62be8d260e2ccc7b732ea90b42c34fc00bdba30e4f6bfa502288" exitCode=0 Oct 31 00:03:07.260445 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:07.259647 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79cd756e-jfn6r" event={"ID":"6b3f747b-72be-4a13-a670-e150d716f39e","Type":"ContainerDied","Data":"80f0db48ea0e62be8d260e2ccc7b732ea90b42c34fc00bdba30e4f6bfa502288"} Oct 31 00:03:07.486172 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:07.486134 4927 patch_prober.go:29] interesting pod/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-vb9r5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body= Oct 31 00:03:07.486401 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:07.486194 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-vb9r5" podUID="334f41bf-6d81-474d-bddc-d38c3492d586" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" Oct 31 00:03:07.955267 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:07.955229 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8b7fb02e-q6fx9"] Oct 31 00:03:07.955909 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:07.955892 4927 cpu_manager.go:414] "containerMap: removing container" podUID="ada7676a-5992-46c1-8a41-870d23a74253" containerName="revoke-keys" Oct 31 00:03:07.955909 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:07.955908 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ada7676a-5992-46c1-8a41-870d23a74253" containerName="revoke-keys" Oct 31 00:03:07.956064 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:07.956044 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="ada7676a-5992-46c1-8a41-870d23a74253" containerName="revoke-keys" Oct 31 00:03:07.959834 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:07.959814 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b7fb02e-q6fx9" Oct 31 00:03:08.020297 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:08.015704 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f00f65ee-2871-4e51-bcde-123f07c1d865-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8b7fb02e-q6fx9\" (UID: \"f00f65ee-2871-4e51-bcde-123f07c1d865\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b7fb02e-q6fx9" Oct 31 00:03:08.020297 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:08.015763 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f00f65ee-2871-4e51-bcde-123f07c1d865-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8b7fb02e-q6fx9\" (UID: \"f00f65ee-2871-4e51-bcde-123f07c1d865\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b7fb02e-q6fx9" Oct 31 00:03:08.020297 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:08.015825 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hlv9j\" (UniqueName: \"kubernetes.io/projected/f00f65ee-2871-4e51-bcde-123f07c1d865-kube-api-access-hlv9j\") pod \"maas-api-revoke-sub-8b7fb02e-q6fx9\" (UID: \"f00f65ee-2871-4e51-bcde-123f07c1d865\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b7fb02e-q6fx9" Oct 31 00:03:08.029662 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:08.026954 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8b7fb02e-q6fx9"] Oct 31 00:03:08.117684 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:08.117623 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f00f65ee-2871-4e51-bcde-123f07c1d865-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8b7fb02e-q6fx9\" (UID: \"f00f65ee-2871-4e51-bcde-123f07c1d865\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b7fb02e-q6fx9" Oct 31 00:03:08.117894 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:08.117731 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f00f65ee-2871-4e51-bcde-123f07c1d865-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8b7fb02e-q6fx9\" (UID: \"f00f65ee-2871-4e51-bcde-123f07c1d865\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b7fb02e-q6fx9" Oct 31 00:03:08.117894 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:08.117797 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hlv9j\" (UniqueName: \"kubernetes.io/projected/f00f65ee-2871-4e51-bcde-123f07c1d865-kube-api-access-hlv9j\") pod \"maas-api-revoke-sub-8b7fb02e-q6fx9\" (UID: \"f00f65ee-2871-4e51-bcde-123f07c1d865\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b7fb02e-q6fx9" Oct 31 00:03:08.119325 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:08.119292 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f00f65ee-2871-4e51-bcde-123f07c1d865-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8b7fb02e-q6fx9\" (UID: \"f00f65ee-2871-4e51-bcde-123f07c1d865\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b7fb02e-q6fx9" Oct 31 00:03:08.121743 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:08.121716 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f00f65ee-2871-4e51-bcde-123f07c1d865-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8b7fb02e-q6fx9\" (UID: \"f00f65ee-2871-4e51-bcde-123f07c1d865\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b7fb02e-q6fx9" Oct 31 00:03:08.180960 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:08.180913 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hlv9j\" (UniqueName: \"kubernetes.io/projected/f00f65ee-2871-4e51-bcde-123f07c1d865-kube-api-access-hlv9j\") pod \"maas-api-revoke-sub-8b7fb02e-q6fx9\" (UID: \"f00f65ee-2871-4e51-bcde-123f07c1d865\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b7fb02e-q6fx9" Oct 31 00:03:08.277644 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:08.277606 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b7fb02e-q6fx9" Oct 31 00:03:08.289894 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:08.289858 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-7w9dq" Oct 31 00:03:08.291088 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:08.291061 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-7w9dq" Oct 31 00:03:08.296547 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:08.295589 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-7w9dq" Oct 31 00:03:08.524426 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:08.522287 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79cd756e-jfn6r" Oct 31 00:03:08.530294 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:08.530245 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b3f747b-72be-4a13-a670-e150d716f39e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b3f747b-72be-4a13-a670-e150d716f39e-maas-api-openshift-ca\") pod \"6b3f747b-72be-4a13-a670-e150d716f39e\" (UID: \"6b3f747b-72be-4a13-a670-e150d716f39e\") " Oct 31 00:03:08.530456 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:08.530372 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b3f747b-72be-4a13-a670-e150d716f39e-kube-api-access-c6ffz\" (UniqueName: \"kubernetes.io/projected/6b3f747b-72be-4a13-a670-e150d716f39e-kube-api-access-c6ffz\") pod \"6b3f747b-72be-4a13-a670-e150d716f39e\" (UID: \"6b3f747b-72be-4a13-a670-e150d716f39e\") " Oct 31 00:03:08.530532 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:08.530466 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6b3f747b-72be-4a13-a670-e150d716f39e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b3f747b-72be-4a13-a670-e150d716f39e-maas-api-xks-ca\") pod \"6b3f747b-72be-4a13-a670-e150d716f39e\" (UID: \"6b3f747b-72be-4a13-a670-e150d716f39e\") " Oct 31 00:03:08.530681 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:08.530639 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b3f747b-72be-4a13-a670-e150d716f39e-maas-api-openshift-ca" pod "6b3f747b-72be-4a13-a670-e150d716f39e" (UID: "6b3f747b-72be-4a13-a670-e150d716f39e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:03:08.530968 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:08.530942 4927 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b3f747b-72be-4a13-a670-e150d716f39e-maas-api-openshift-ca\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:03:08.532693 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:08.532664 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b3f747b-72be-4a13-a670-e150d716f39e-kube-api-access-c6ffz" pod "6b3f747b-72be-4a13-a670-e150d716f39e" (UID: "6b3f747b-72be-4a13-a670-e150d716f39e"). InnerVolumeSpecName "kube-api-access-c6ffz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:03:08.533139 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:08.533104 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b3f747b-72be-4a13-a670-e150d716f39e-maas-api-xks-ca" pod "6b3f747b-72be-4a13-a670-e150d716f39e" (UID: "6b3f747b-72be-4a13-a670-e150d716f39e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:03:08.632908 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:08.632854 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c6ffz\" (UniqueName: \"kubernetes.io/projected/6b3f747b-72be-4a13-a670-e150d716f39e-kube-api-access-c6ffz\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:03:08.632908 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:08.632900 4927 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b3f747b-72be-4a13-a670-e150d716f39e-maas-api-xks-ca\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:03:08.720401 ip-10-0-141-180 kubenswrapper[4927]: W1031 00:03:08.720344 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf00f65ee_2871_4e51_bcde_123f07c1d865.slice/crio-0d544bcffe3d75dcbb7c67143e6959b18bccc7fab7b5aab922fca39066b500e3 WatchSource:0}: Error finding container 0d544bcffe3d75dcbb7c67143e6959b18bccc7fab7b5aab922fca39066b500e3: Status 404 returned error can't find the container with id 0d544bcffe3d75dcbb7c67143e6959b18bccc7fab7b5aab922fca39066b500e3 Oct 31 00:03:08.723731 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:08.723702 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8b7fb02e-q6fx9"] Oct 31 00:03:08.874403 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:08.869535 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-vb9r5" Oct 31 00:03:08.874403 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:08.871036 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-vb9r5" Oct 31 00:03:08.874403 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:08.872546 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-vb9r5" Oct 31 00:03:09.283706 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:09.283672 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79cd756e-jfn6r" Oct 31 00:03:09.283706 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:09.283692 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79cd756e-jfn6r" event={"ID":"6b3f747b-72be-4a13-a670-e150d716f39e","Type":"ContainerDied","Data":"daf8e555e326ccfa9dd6bc78003c8f4b76ab4edd70e15133c4e43dfd2f89c294"} Oct 31 00:03:09.284242 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:09.283755 4927 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="daf8e555e326ccfa9dd6bc78003c8f4b76ab4edd70e15133c4e43dfd2f89c294" Oct 31 00:03:09.285641 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:09.285613 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b7fb02e-q6fx9" event={"ID":"f00f65ee-2871-4e51-bcde-123f07c1d865","Type":"ContainerStarted","Data":"c73aec392b2d8c6904a0b5f15b7b111f6ddbe9408e1930737fe4e1791d47d3f0"} Oct 31 00:03:09.285847 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:09.285829 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b7fb02e-q6fx9" event={"ID":"f00f65ee-2871-4e51-bcde-123f07c1d865","Type":"ContainerStarted","Data":"0d544bcffe3d75dcbb7c67143e6959b18bccc7fab7b5aab922fca39066b500e3"} Oct 31 00:03:09.407785 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:09.407728 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b7fb02e-q6fx9" podStartSLOduration=2.407714569 podStartE2EDuration="2.407714569s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:03:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:03:09.405421077 +0000 UTC m=+1302.441751930" watchObservedRunningTime="2026-10-31 00:03:09.407714569 +0000 UTC m=+1302.444045419" Oct 31 00:03:10.009019 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:10.008930 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:03:10.295946 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:10.295857 4927 generic.go:356] "Generic (PLEG): container finished" podID="f00f65ee-2871-4e51-bcde-123f07c1d865" containerID="c73aec392b2d8c6904a0b5f15b7b111f6ddbe9408e1930737fe4e1791d47d3f0" exitCode=0 Oct 31 00:03:10.295946 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:10.295920 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b7fb02e-q6fx9" event={"ID":"f00f65ee-2871-4e51-bcde-123f07c1d865","Type":"ContainerDied","Data":"c73aec392b2d8c6904a0b5f15b7b111f6ddbe9408e1930737fe4e1791d47d3f0"} Oct 31 00:03:11.497921 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:11.497896 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b7fb02e-q6fx9" Oct 31 00:03:11.606902 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:11.606814 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f00f65ee-2871-4e51-bcde-123f07c1d865-kube-api-access-hlv9j\" (UniqueName: \"kubernetes.io/projected/f00f65ee-2871-4e51-bcde-123f07c1d865-kube-api-access-hlv9j\") pod \"f00f65ee-2871-4e51-bcde-123f07c1d865\" (UID: \"f00f65ee-2871-4e51-bcde-123f07c1d865\") " Oct 31 00:03:11.607064 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:11.607004 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f00f65ee-2871-4e51-bcde-123f07c1d865-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f00f65ee-2871-4e51-bcde-123f07c1d865-maas-api-openshift-ca\") pod \"f00f65ee-2871-4e51-bcde-123f07c1d865\" (UID: \"f00f65ee-2871-4e51-bcde-123f07c1d865\") " Oct 31 00:03:11.607140 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:11.607124 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f00f65ee-2871-4e51-bcde-123f07c1d865-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f00f65ee-2871-4e51-bcde-123f07c1d865-maas-api-xks-ca\") pod \"f00f65ee-2871-4e51-bcde-123f07c1d865\" (UID: \"f00f65ee-2871-4e51-bcde-123f07c1d865\") " Oct 31 00:03:11.607410 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:11.607375 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f00f65ee-2871-4e51-bcde-123f07c1d865-maas-api-openshift-ca" pod "f00f65ee-2871-4e51-bcde-123f07c1d865" (UID: "f00f65ee-2871-4e51-bcde-123f07c1d865"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:03:11.607800 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:11.607779 4927 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f00f65ee-2871-4e51-bcde-123f07c1d865-maas-api-openshift-ca\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:03:11.609154 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:11.609123 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f00f65ee-2871-4e51-bcde-123f07c1d865-kube-api-access-hlv9j" pod "f00f65ee-2871-4e51-bcde-123f07c1d865" (UID: "f00f65ee-2871-4e51-bcde-123f07c1d865"). InnerVolumeSpecName "kube-api-access-hlv9j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:03:11.609259 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:11.609177 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f00f65ee-2871-4e51-bcde-123f07c1d865-maas-api-xks-ca" pod "f00f65ee-2871-4e51-bcde-123f07c1d865" (UID: "f00f65ee-2871-4e51-bcde-123f07c1d865"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:03:11.709369 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:11.709326 4927 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f00f65ee-2871-4e51-bcde-123f07c1d865-maas-api-xks-ca\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:03:11.709369 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:11.709365 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hlv9j\" (UniqueName: \"kubernetes.io/projected/f00f65ee-2871-4e51-bcde-123f07c1d865-kube-api-access-hlv9j\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:03:12.319638 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:12.319594 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b7fb02e-q6fx9" event={"ID":"f00f65ee-2871-4e51-bcde-123f07c1d865","Type":"ContainerDied","Data":"0d544bcffe3d75dcbb7c67143e6959b18bccc7fab7b5aab922fca39066b500e3"} Oct 31 00:03:12.319839 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:12.319664 4927 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0d544bcffe3d75dcbb7c67143e6959b18bccc7fab7b5aab922fca39066b500e3" Oct 31 00:03:12.319839 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:12.319740 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b7fb02e-q6fx9" Oct 31 00:03:12.663282 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:12.663229 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0ef2f748-jlssh"] Oct 31 00:03:12.664279 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:12.664252 4927 cpu_manager.go:414] "containerMap: removing container" podUID="6b3f747b-72be-4a13-a670-e150d716f39e" containerName="revoke-keys" Oct 31 00:03:12.664279 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:12.664276 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b3f747b-72be-4a13-a670-e150d716f39e" containerName="revoke-keys" Oct 31 00:03:12.664499 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:12.664319 4927 cpu_manager.go:414] "containerMap: removing container" podUID="f00f65ee-2871-4e51-bcde-123f07c1d865" containerName="revoke-keys" Oct 31 00:03:12.664499 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:12.664324 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f00f65ee-2871-4e51-bcde-123f07c1d865" containerName="revoke-keys" Oct 31 00:03:12.664499 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:12.664485 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="f00f65ee-2871-4e51-bcde-123f07c1d865" containerName="revoke-keys" Oct 31 00:03:12.664499 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:12.664498 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b3f747b-72be-4a13-a670-e150d716f39e" containerName="revoke-keys" Oct 31 00:03:12.669035 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:12.669014 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ef2f748-jlssh" Oct 31 00:03:12.722925 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:12.722889 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:03:12.723103 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:12.723019 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4fdzs\"" Oct 31 00:03:12.726105 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:12.726073 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0ef2f748-jlssh"] Oct 31 00:03:12.741820 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:12.741791 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ctk47\" (UniqueName: \"kubernetes.io/projected/289b001c-26fa-413b-957e-d7928ca24c1b-kube-api-access-ctk47\") pod \"maas-api-revoke-sub-0ef2f748-jlssh\" (UID: \"289b001c-26fa-413b-957e-d7928ca24c1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ef2f748-jlssh" Oct 31 00:03:12.741975 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:12.741927 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/289b001c-26fa-413b-957e-d7928ca24c1b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0ef2f748-jlssh\" (UID: \"289b001c-26fa-413b-957e-d7928ca24c1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ef2f748-jlssh" Oct 31 00:03:12.742183 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:12.742165 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/289b001c-26fa-413b-957e-d7928ca24c1b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0ef2f748-jlssh\" (UID: \"289b001c-26fa-413b-957e-d7928ca24c1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ef2f748-jlssh" Oct 31 00:03:12.844298 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:12.844253 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/289b001c-26fa-413b-957e-d7928ca24c1b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0ef2f748-jlssh\" (UID: \"289b001c-26fa-413b-957e-d7928ca24c1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ef2f748-jlssh" Oct 31 00:03:12.844516 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:12.844381 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ctk47\" (UniqueName: \"kubernetes.io/projected/289b001c-26fa-413b-957e-d7928ca24c1b-kube-api-access-ctk47\") pod \"maas-api-revoke-sub-0ef2f748-jlssh\" (UID: \"289b001c-26fa-413b-957e-d7928ca24c1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ef2f748-jlssh" Oct 31 00:03:12.844516 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:12.844458 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/289b001c-26fa-413b-957e-d7928ca24c1b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0ef2f748-jlssh\" (UID: \"289b001c-26fa-413b-957e-d7928ca24c1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ef2f748-jlssh" Oct 31 00:03:12.845125 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:12.845091 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/289b001c-26fa-413b-957e-d7928ca24c1b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0ef2f748-jlssh\" (UID: \"289b001c-26fa-413b-957e-d7928ca24c1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ef2f748-jlssh" Oct 31 00:03:12.846710 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:12.846688 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/289b001c-26fa-413b-957e-d7928ca24c1b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0ef2f748-jlssh\" (UID: \"289b001c-26fa-413b-957e-d7928ca24c1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ef2f748-jlssh" Oct 31 00:03:12.906498 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:12.906471 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ctk47\" (UniqueName: \"kubernetes.io/projected/289b001c-26fa-413b-957e-d7928ca24c1b-kube-api-access-ctk47\") pod \"maas-api-revoke-sub-0ef2f748-jlssh\" (UID: \"289b001c-26fa-413b-957e-d7928ca24c1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ef2f748-jlssh" Oct 31 00:03:12.988101 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:12.988020 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ef2f748-jlssh" Oct 31 00:03:13.376606 ip-10-0-141-180 kubenswrapper[4927]: W1031 00:03:13.376562 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod289b001c_26fa_413b_957e_d7928ca24c1b.slice/crio-02e11bf4502307b2e353233942ac3a2b64e6a29ce78333371acce45025c9f001 WatchSource:0}: Error finding container 02e11bf4502307b2e353233942ac3a2b64e6a29ce78333371acce45025c9f001: Status 404 returned error can't find the container with id 02e11bf4502307b2e353233942ac3a2b64e6a29ce78333371acce45025c9f001 Oct 31 00:03:13.377190 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:13.377145 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0ef2f748-jlssh"] Oct 31 00:03:14.346430 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:14.346382 4927 generic.go:356] "Generic (PLEG): container finished" podID="289b001c-26fa-413b-957e-d7928ca24c1b" containerID="78a5aeab69e4ddad53aaece7dc68148ed1bf2f4b6f7658b4723f8b9d7f0a1959" exitCode=0 Oct 31 00:03:14.346801 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:14.346449 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ef2f748-jlssh" event={"ID":"289b001c-26fa-413b-957e-d7928ca24c1b","Type":"ContainerDied","Data":"78a5aeab69e4ddad53aaece7dc68148ed1bf2f4b6f7658b4723f8b9d7f0a1959"} Oct 31 00:03:14.346801 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:14.346504 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ef2f748-jlssh" event={"ID":"289b001c-26fa-413b-957e-d7928ca24c1b","Type":"ContainerStarted","Data":"02e11bf4502307b2e353233942ac3a2b64e6a29ce78333371acce45025c9f001"} Oct 31 00:03:14.354227 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:14.354192 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4d65879a-l2xz9"] Oct 31 00:03:14.358874 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:14.358848 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d65879a-l2xz9" Oct 31 00:03:14.458683 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:14.458647 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4d65879a-l2xz9"] Oct 31 00:03:14.469028 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:14.468993 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6c30728-965b-4edf-a59b-38d13d7459ea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4d65879a-l2xz9\" (UID: \"b6c30728-965b-4edf-a59b-38d13d7459ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d65879a-l2xz9" Oct 31 00:03:14.469238 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:14.469215 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6c30728-965b-4edf-a59b-38d13d7459ea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4d65879a-l2xz9\" (UID: \"b6c30728-965b-4edf-a59b-38d13d7459ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d65879a-l2xz9" Oct 31 00:03:14.469439 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:14.469413 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6tpq\" (UniqueName: \"kubernetes.io/projected/b6c30728-965b-4edf-a59b-38d13d7459ea-kube-api-access-j6tpq\") pod \"maas-api-revoke-sub-4d65879a-l2xz9\" (UID: \"b6c30728-965b-4edf-a59b-38d13d7459ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d65879a-l2xz9" Oct 31 00:03:14.570808 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:14.570721 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6tpq\" (UniqueName: \"kubernetes.io/projected/b6c30728-965b-4edf-a59b-38d13d7459ea-kube-api-access-j6tpq\") pod \"maas-api-revoke-sub-4d65879a-l2xz9\" (UID: \"b6c30728-965b-4edf-a59b-38d13d7459ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d65879a-l2xz9" Oct 31 00:03:14.570808 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:14.570800 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6c30728-965b-4edf-a59b-38d13d7459ea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4d65879a-l2xz9\" (UID: \"b6c30728-965b-4edf-a59b-38d13d7459ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d65879a-l2xz9" Oct 31 00:03:14.571056 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:14.570929 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6c30728-965b-4edf-a59b-38d13d7459ea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4d65879a-l2xz9\" (UID: \"b6c30728-965b-4edf-a59b-38d13d7459ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d65879a-l2xz9" Oct 31 00:03:14.571677 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:14.571657 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6c30728-965b-4edf-a59b-38d13d7459ea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4d65879a-l2xz9\" (UID: \"b6c30728-965b-4edf-a59b-38d13d7459ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d65879a-l2xz9" Oct 31 00:03:14.573199 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:14.573176 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6c30728-965b-4edf-a59b-38d13d7459ea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4d65879a-l2xz9\" (UID: \"b6c30728-965b-4edf-a59b-38d13d7459ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d65879a-l2xz9" Oct 31 00:03:14.582797 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:14.582752 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:03:14.662409 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:14.658253 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6tpq\" (UniqueName: \"kubernetes.io/projected/b6c30728-965b-4edf-a59b-38d13d7459ea-kube-api-access-j6tpq\") pod \"maas-api-revoke-sub-4d65879a-l2xz9\" (UID: \"b6c30728-965b-4edf-a59b-38d13d7459ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d65879a-l2xz9" Oct 31 00:03:14.693389 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:14.681151 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d65879a-l2xz9" Oct 31 00:03:15.154304 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:15.154160 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4d65879a-l2xz9"] Oct 31 00:03:15.358204 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:15.358169 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d65879a-l2xz9" event={"ID":"b6c30728-965b-4edf-a59b-38d13d7459ea","Type":"ContainerStarted","Data":"78df793445b0bcfda7912ea447be95d7e6cb815cf36764545e24976300e10714"} Oct 31 00:03:15.552171 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:15.552142 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ef2f748-jlssh" Oct 31 00:03:15.614665 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:15.614631 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/289b001c-26fa-413b-957e-d7928ca24c1b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/289b001c-26fa-413b-957e-d7928ca24c1b-maas-api-xks-ca\") pod \"289b001c-26fa-413b-957e-d7928ca24c1b\" (UID: \"289b001c-26fa-413b-957e-d7928ca24c1b\") " Oct 31 00:03:15.614850 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:15.614826 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/289b001c-26fa-413b-957e-d7928ca24c1b-kube-api-access-ctk47\" (UniqueName: \"kubernetes.io/projected/289b001c-26fa-413b-957e-d7928ca24c1b-kube-api-access-ctk47\") pod \"289b001c-26fa-413b-957e-d7928ca24c1b\" (UID: \"289b001c-26fa-413b-957e-d7928ca24c1b\") " Oct 31 00:03:15.614979 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:15.614863 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/289b001c-26fa-413b-957e-d7928ca24c1b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/289b001c-26fa-413b-957e-d7928ca24c1b-maas-api-openshift-ca\") pod \"289b001c-26fa-413b-957e-d7928ca24c1b\" (UID: \"289b001c-26fa-413b-957e-d7928ca24c1b\") " Oct 31 00:03:15.615295 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:15.615257 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/289b001c-26fa-413b-957e-d7928ca24c1b-maas-api-openshift-ca" pod "289b001c-26fa-413b-957e-d7928ca24c1b" (UID: "289b001c-26fa-413b-957e-d7928ca24c1b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:03:15.615478 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:15.615457 4927 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/289b001c-26fa-413b-957e-d7928ca24c1b-maas-api-openshift-ca\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:03:15.617157 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:15.617132 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/289b001c-26fa-413b-957e-d7928ca24c1b-kube-api-access-ctk47" pod "289b001c-26fa-413b-957e-d7928ca24c1b" (UID: "289b001c-26fa-413b-957e-d7928ca24c1b"). InnerVolumeSpecName "kube-api-access-ctk47". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:03:15.617238 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:15.617192 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/289b001c-26fa-413b-957e-d7928ca24c1b-maas-api-xks-ca" pod "289b001c-26fa-413b-957e-d7928ca24c1b" (UID: "289b001c-26fa-413b-957e-d7928ca24c1b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:03:15.698620 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:15.698223 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-47cd94c7-bvjxk"] Oct 31 00:03:15.699850 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:15.699173 4927 cpu_manager.go:414] "containerMap: removing container" podUID="289b001c-26fa-413b-957e-d7928ca24c1b" containerName="revoke-keys" Oct 31 00:03:15.699850 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:15.699192 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="289b001c-26fa-413b-957e-d7928ca24c1b" containerName="revoke-keys" Oct 31 00:03:15.699850 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:15.699352 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="289b001c-26fa-413b-957e-d7928ca24c1b" containerName="revoke-keys" Oct 31 00:03:15.703067 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:15.703049 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47cd94c7-bvjxk" Oct 31 00:03:15.716801 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:15.716770 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4v46\" (UniqueName: \"kubernetes.io/projected/5655083c-8498-4488-8ec7-49e4de5d40fd-kube-api-access-d4v46\") pod \"maas-api-revoke-sub-47cd94c7-bvjxk\" (UID: \"5655083c-8498-4488-8ec7-49e4de5d40fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47cd94c7-bvjxk" Oct 31 00:03:15.716990 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:15.716966 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5655083c-8498-4488-8ec7-49e4de5d40fd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-47cd94c7-bvjxk\" (UID: \"5655083c-8498-4488-8ec7-49e4de5d40fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47cd94c7-bvjxk" Oct 31 00:03:15.717142 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:15.717127 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5655083c-8498-4488-8ec7-49e4de5d40fd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-47cd94c7-bvjxk\" (UID: \"5655083c-8498-4488-8ec7-49e4de5d40fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47cd94c7-bvjxk" Oct 31 00:03:15.717264 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:15.717251 4927 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/289b001c-26fa-413b-957e-d7928ca24c1b-maas-api-xks-ca\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:03:15.717322 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:15.717271 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ctk47\" (UniqueName: \"kubernetes.io/projected/289b001c-26fa-413b-957e-d7928ca24c1b-kube-api-access-ctk47\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:03:15.762444 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:15.762414 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-47cd94c7-bvjxk"] Oct 31 00:03:15.818554 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:15.818518 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5655083c-8498-4488-8ec7-49e4de5d40fd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-47cd94c7-bvjxk\" (UID: \"5655083c-8498-4488-8ec7-49e4de5d40fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47cd94c7-bvjxk" Oct 31 00:03:15.818723 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:15.818611 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4v46\" (UniqueName: \"kubernetes.io/projected/5655083c-8498-4488-8ec7-49e4de5d40fd-kube-api-access-d4v46\") pod \"maas-api-revoke-sub-47cd94c7-bvjxk\" (UID: \"5655083c-8498-4488-8ec7-49e4de5d40fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47cd94c7-bvjxk" Oct 31 00:03:15.818807 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:15.818724 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5655083c-8498-4488-8ec7-49e4de5d40fd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-47cd94c7-bvjxk\" (UID: \"5655083c-8498-4488-8ec7-49e4de5d40fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47cd94c7-bvjxk" Oct 31 00:03:15.819546 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:15.819517 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5655083c-8498-4488-8ec7-49e4de5d40fd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-47cd94c7-bvjxk\" (UID: \"5655083c-8498-4488-8ec7-49e4de5d40fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47cd94c7-bvjxk" Oct 31 00:03:15.821382 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:15.821355 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5655083c-8498-4488-8ec7-49e4de5d40fd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-47cd94c7-bvjxk\" (UID: \"5655083c-8498-4488-8ec7-49e4de5d40fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47cd94c7-bvjxk" Oct 31 00:03:15.882014 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:15.881978 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4v46\" (UniqueName: \"kubernetes.io/projected/5655083c-8498-4488-8ec7-49e4de5d40fd-kube-api-access-d4v46\") pod \"maas-api-revoke-sub-47cd94c7-bvjxk\" (UID: \"5655083c-8498-4488-8ec7-49e4de5d40fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47cd94c7-bvjxk" Oct 31 00:03:16.021929 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:16.021827 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47cd94c7-bvjxk" Oct 31 00:03:16.369248 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:16.369207 4927 generic.go:356] "Generic (PLEG): container finished" podID="b6c30728-965b-4edf-a59b-38d13d7459ea" containerID="afed5fcb9c4eb942eb2f49c6fe69ed7635be16c3dfc799753d6351802d44ee13" exitCode=0 Oct 31 00:03:16.369726 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:16.369300 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d65879a-l2xz9" event={"ID":"b6c30728-965b-4edf-a59b-38d13d7459ea","Type":"ContainerDied","Data":"afed5fcb9c4eb942eb2f49c6fe69ed7635be16c3dfc799753d6351802d44ee13"} Oct 31 00:03:16.370877 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:16.370857 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ef2f748-jlssh" Oct 31 00:03:16.370997 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:16.370892 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ef2f748-jlssh" event={"ID":"289b001c-26fa-413b-957e-d7928ca24c1b","Type":"ContainerDied","Data":"02e11bf4502307b2e353233942ac3a2b64e6a29ce78333371acce45025c9f001"} Oct 31 00:03:16.370997 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:16.370931 4927 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="02e11bf4502307b2e353233942ac3a2b64e6a29ce78333371acce45025c9f001" Oct 31 00:03:16.412044 ip-10-0-141-180 kubenswrapper[4927]: W1031 00:03:16.412016 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5655083c_8498_4488_8ec7_49e4de5d40fd.slice/crio-d0449ff842ffee0039795da47aeb02f90c8bfa51b7b64c405c3888aa3c5a8dbd WatchSource:0}: Error finding container d0449ff842ffee0039795da47aeb02f90c8bfa51b7b64c405c3888aa3c5a8dbd: Status 404 returned error can't find the container with id d0449ff842ffee0039795da47aeb02f90c8bfa51b7b64c405c3888aa3c5a8dbd Oct 31 00:03:16.414495 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:16.414466 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-47cd94c7-bvjxk"] Oct 31 00:03:17.092349 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:17.092307 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:03:17.302536 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:17.302499 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:03:17.381991 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:17.381903 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47cd94c7-bvjxk" event={"ID":"5655083c-8498-4488-8ec7-49e4de5d40fd","Type":"ContainerStarted","Data":"62b7d68856a267415e32f509eabc516f5b0aef9c2c6522dc60570b7509d9d223"} Oct 31 00:03:17.381991 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:17.381959 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47cd94c7-bvjxk" event={"ID":"5655083c-8498-4488-8ec7-49e4de5d40fd","Type":"ContainerStarted","Data":"d0449ff842ffee0039795da47aeb02f90c8bfa51b7b64c405c3888aa3c5a8dbd"} Oct 31 00:03:17.502682 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:17.502626 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47cd94c7-bvjxk" podStartSLOduration=2.502611414 podStartE2EDuration="2.502611414s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:03:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:03:17.50031086 +0000 UTC m=+1310.536641683" watchObservedRunningTime="2026-10-31 00:03:17.502611414 +0000 UTC m=+1310.538942236" Oct 31 00:03:17.565224 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:17.565198 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d65879a-l2xz9" Oct 31 00:03:17.661547 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:17.661517 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b6c30728-965b-4edf-a59b-38d13d7459ea-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6c30728-965b-4edf-a59b-38d13d7459ea-maas-api-openshift-ca\") pod \"b6c30728-965b-4edf-a59b-38d13d7459ea\" (UID: \"b6c30728-965b-4edf-a59b-38d13d7459ea\") " Oct 31 00:03:17.661817 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:17.661797 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b6c30728-965b-4edf-a59b-38d13d7459ea-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6c30728-965b-4edf-a59b-38d13d7459ea-maas-api-xks-ca\") pod \"b6c30728-965b-4edf-a59b-38d13d7459ea\" (UID: \"b6c30728-965b-4edf-a59b-38d13d7459ea\") " Oct 31 00:03:17.661930 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:17.661913 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6c30728-965b-4edf-a59b-38d13d7459ea-kube-api-access-j6tpq\" (UniqueName: \"kubernetes.io/projected/b6c30728-965b-4edf-a59b-38d13d7459ea-kube-api-access-j6tpq\") pod \"b6c30728-965b-4edf-a59b-38d13d7459ea\" (UID: \"b6c30728-965b-4edf-a59b-38d13d7459ea\") " Oct 31 00:03:17.661974 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:17.661924 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6c30728-965b-4edf-a59b-38d13d7459ea-maas-api-openshift-ca" pod "b6c30728-965b-4edf-a59b-38d13d7459ea" (UID: "b6c30728-965b-4edf-a59b-38d13d7459ea"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:03:17.662409 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:17.662391 4927 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6c30728-965b-4edf-a59b-38d13d7459ea-maas-api-openshift-ca\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:03:17.663855 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:17.663831 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6c30728-965b-4edf-a59b-38d13d7459ea-maas-api-xks-ca" pod "b6c30728-965b-4edf-a59b-38d13d7459ea" (UID: "b6c30728-965b-4edf-a59b-38d13d7459ea"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:03:17.663948 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:17.663898 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6c30728-965b-4edf-a59b-38d13d7459ea-kube-api-access-j6tpq" pod "b6c30728-965b-4edf-a59b-38d13d7459ea" (UID: "b6c30728-965b-4edf-a59b-38d13d7459ea"). InnerVolumeSpecName "kube-api-access-j6tpq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:03:17.763801 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:17.763765 4927 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6c30728-965b-4edf-a59b-38d13d7459ea-maas-api-xks-ca\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:03:17.763801 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:17.763796 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j6tpq\" (UniqueName: \"kubernetes.io/projected/b6c30728-965b-4edf-a59b-38d13d7459ea-kube-api-access-j6tpq\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:03:18.038491 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:18.038323 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:03:18.464803 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:18.464760 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d65879a-l2xz9" event={"ID":"b6c30728-965b-4edf-a59b-38d13d7459ea","Type":"ContainerDied","Data":"78df793445b0bcfda7912ea447be95d7e6cb815cf36764545e24976300e10714"} Oct 31 00:03:18.465309 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:18.464832 4927 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="78df793445b0bcfda7912ea447be95d7e6cb815cf36764545e24976300e10714" Oct 31 00:03:18.465309 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:18.464904 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d65879a-l2xz9" Oct 31 00:03:18.469560 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:18.468519 4927 generic.go:356] "Generic (PLEG): container finished" podID="5655083c-8498-4488-8ec7-49e4de5d40fd" containerID="62b7d68856a267415e32f509eabc516f5b0aef9c2c6522dc60570b7509d9d223" exitCode=0 Oct 31 00:03:18.469560 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:18.468585 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47cd94c7-bvjxk" event={"ID":"5655083c-8498-4488-8ec7-49e4de5d40fd","Type":"ContainerDied","Data":"62b7d68856a267415e32f509eabc516f5b0aef9c2c6522dc60570b7509d9d223"} Oct 31 00:03:19.289355 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:19.285488 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:03:19.733435 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:19.730020 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:03:19.848858 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:03:19.848817 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sqv69" containerName="extractor" Oct 31 00:03:19.999533 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:19.999437 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fe01fa/distinct-2-w5-fe01fa-kserve-79b99966d8-dkhzd" podUID="d5c1c9e8-ce47-4868-860d-b8413ff74ab5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.111:8000/ready\": context deadline exceeded" Oct 31 00:03:20.564494 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:20.563954 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5c3374/premium-w1-5c3374-kserve-6c56d4dcd-llwcl" podUID="dc813f02-e077-4251-8c1e-fea0986b0f36" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded" Oct 31 00:03:20.682680 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:20.679478 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-35a187/distinct-2-w6-35a187-kserve-7ff4b6dc55-944g6" podUID="429ee092-42d8-475e-b0c6-b5a9930766fc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.113:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:03:20.747761 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:20.747718 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-46b586/simulator-w0-46b586-kserve-67588b8dbd-z8xmn" podUID="9a25e252-223d-41f2-abf7-632067c77e0b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded" Oct 31 00:03:20.929923 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:20.914009 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-27e82b/distinct-w2-27e82b-kserve-75f475fcd9-mccsz" podUID="4ba8ced4-e93f-46ac-87c5-3d0b7fadc02d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:03:20.929923 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:20.915856 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:03:20.932101 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:20.932069 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-0d96587a-openshift-default-bcfbd49cc-xrrp4"] Oct 31 00:03:20.944347 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:20.944303 4927 cpu_manager.go:414] "containerMap: removing container" podUID="b6c30728-965b-4edf-a59b-38d13d7459ea" containerName="revoke-keys" Oct 31 00:03:20.945438 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:20.945412 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6c30728-965b-4edf-a59b-38d13d7459ea" containerName="revoke-keys" Oct 31 00:03:20.945944 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:20.945927 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6c30728-965b-4edf-a59b-38d13d7459ea" containerName="revoke-keys" Oct 31 00:03:21.007372 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:21.007138 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fe01fa/distinct-2-w5-fe01fa-kserve-79b99966d8-dkhzd" podUID="d5c1c9e8-ce47-4868-860d-b8413ff74ab5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.111:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:03:21.038265 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:21.038230 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-fe01fa/distinct-2-w5-fe01fa-kserve-79b99966d8-dkhzd" Oct 31 00:03:21.040159 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:21.038699 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-0d96587a-openshift-default-bcfbd49cc-xrrp4" Oct 31 00:03:21.063356 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:21.059909 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-0d96587a-openshift-default-bcfbd49cc-xrrp4"] Oct 31 00:03:21.111775 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:21.110552 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-986d37/premium-w3-986d37-kserve-5c7c7757fb-r99j7" podUID="f4e38df7-355f-4b47-a0b5-7e0b907f3326" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded" Oct 31 00:03:21.142569 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:21.142040 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-0d96587a-openshift-default-dockercfg-zf8rs\"" Oct 31 00:03:21.312894 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:21.261206 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d310a62f-9f54-4000-b280-0581fd91192f-istio-envoy\") pod \"e2e-ait-0d96587a-openshift-default-bcfbd49cc-xrrp4\" (UID: \"d310a62f-9f54-4000-b280-0581fd91192f\") " pod="openshift-ingress/e2e-ait-0d96587a-openshift-default-bcfbd49cc-xrrp4" Oct 31 00:03:21.312894 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:21.261249 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d310a62f-9f54-4000-b280-0581fd91192f-credential-socket\") pod \"e2e-ait-0d96587a-openshift-default-bcfbd49cc-xrrp4\" (UID: \"d310a62f-9f54-4000-b280-0581fd91192f\") " pod="openshift-ingress/e2e-ait-0d96587a-openshift-default-bcfbd49cc-xrrp4" Oct 31 00:03:21.312894 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:21.288253 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d310a62f-9f54-4000-b280-0581fd91192f-istio-podinfo\") pod \"e2e-ait-0d96587a-openshift-default-bcfbd49cc-xrrp4\" (UID: \"d310a62f-9f54-4000-b280-0581fd91192f\") " pod="openshift-ingress/e2e-ait-0d96587a-openshift-default-bcfbd49cc-xrrp4" Oct 31 00:03:21.312894 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:21.288947 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d310a62f-9f54-4000-b280-0581fd91192f-istio-token\") pod \"e2e-ait-0d96587a-openshift-default-bcfbd49cc-xrrp4\" (UID: \"d310a62f-9f54-4000-b280-0581fd91192f\") " pod="openshift-ingress/e2e-ait-0d96587a-openshift-default-bcfbd49cc-xrrp4" Oct 31 00:03:21.312894 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:21.289185 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d310a62f-9f54-4000-b280-0581fd91192f-workload-socket\") pod \"e2e-ait-0d96587a-openshift-default-bcfbd49cc-xrrp4\" (UID: \"d310a62f-9f54-4000-b280-0581fd91192f\") " pod="openshift-ingress/e2e-ait-0d96587a-openshift-default-bcfbd49cc-xrrp4" Oct 31 00:03:21.312894 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:21.289246 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d310a62f-9f54-4000-b280-0581fd91192f-istio-data\") pod \"e2e-ait-0d96587a-openshift-default-bcfbd49cc-xrrp4\" (UID: \"d310a62f-9f54-4000-b280-0581fd91192f\") " pod="openshift-ingress/e2e-ait-0d96587a-openshift-default-bcfbd49cc-xrrp4" Oct 31 00:03:21.312894 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:21.289433 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d310a62f-9f54-4000-b280-0581fd91192f-workload-certs\") pod \"e2e-ait-0d96587a-openshift-default-bcfbd49cc-xrrp4\" (UID: \"d310a62f-9f54-4000-b280-0581fd91192f\") " pod="openshift-ingress/e2e-ait-0d96587a-openshift-default-bcfbd49cc-xrrp4" Oct 31 00:03:21.312894 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:21.289479 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d310a62f-9f54-4000-b280-0581fd91192f-istiod-ca-cert\") pod \"e2e-ait-0d96587a-openshift-default-bcfbd49cc-xrrp4\" (UID: \"d310a62f-9f54-4000-b280-0581fd91192f\") " pod="openshift-ingress/e2e-ait-0d96587a-openshift-default-bcfbd49cc-xrrp4" Oct 31 00:03:21.312894 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:21.289513 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dmdsb\" (UniqueName: \"kubernetes.io/projected/d310a62f-9f54-4000-b280-0581fd91192f-kube-api-access-dmdsb\") pod \"e2e-ait-0d96587a-openshift-default-bcfbd49cc-xrrp4\" (UID: \"d310a62f-9f54-4000-b280-0581fd91192f\") " pod="openshift-ingress/e2e-ait-0d96587a-openshift-default-bcfbd49cc-xrrp4" Oct 31 00:03:21.336732 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:21.336422 4927 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-27e82b/distinct-w2-27e82b-kserve-75f475fcd9-mccsz" podUID="4ba8ced4-e93f-46ac-87c5-3d0b7fadc02d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:03:21.404710 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:21.395763 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d310a62f-9f54-4000-b280-0581fd91192f-workload-certs\") pod \"e2e-ait-0d96587a-openshift-default-bcfbd49cc-xrrp4\" (UID: \"d310a62f-9f54-4000-b280-0581fd91192f\") " pod="openshift-ingress/e2e-ait-0d96587a-openshift-default-bcfbd49cc-xrrp4" Oct 31 00:03:21.404710 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:21.395821 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d310a62f-9f54-4000-b280-0581fd91192f-istiod-ca-cert\") pod \"e2e-ait-0d96587a-openshift-default-bcfbd49cc-xrrp4\" (UID: \"d310a62f-9f54-4000-b280-0581fd91192f\") " pod="openshift-ingress/e2e-ait-0d96587a-openshift-default-bcfbd49cc-xrrp4" Oct 31 00:03:21.404710 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:21.395856 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmdsb\" (UniqueName: \"kubernetes.io/projected/d310a62f-9f54-4000-b280-0581fd91192f-kube-api-access-dmdsb\") pod \"e2e-ait-0d96587a-openshift-default-bcfbd49cc-xrrp4\" (UID: \"d310a62f-9f54-4000-b280-0581fd91192f\") " pod="openshift-ingress/e2e-ait-0d96587a-openshift-default-bcfbd49cc-xrrp4" Oct 31 00:03:21.404710 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:21.395935 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d310a62f-9f54-4000-b280-0581fd91192f-istio-envoy\") pod \"e2e-ait-0d96587a-openshift-default-bcfbd49cc-xrrp4\" (UID: \"d310a62f-9f54-4000-b280-0581fd91192f\") " pod="openshift-ingress/e2e-ait-0d96587a-openshift-default-bcfbd49cc-xrrp4" Oct 31 00:03:21.404710 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:21.395963 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d310a62f-9f54-4000-b280-0581fd91192f-credential-socket\") pod \"e2e-ait-0d96587a-openshift-default-bcfbd49cc-xrrp4\" (UID: \"d310a62f-9f54-4000-b280-0581fd91192f\") " pod="openshift-ingress/e2e-ait-0d96587a-openshift-default-bcfbd49cc-xrrp4" Oct 31 00:03:21.404710 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:21.396061 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d310a62f-9f54-4000-b280-0581fd91192f-istio-podinfo\") pod \"e2e-ait-0d96587a-openshift-default-bcfbd49cc-xrrp4\" (UID: \"d310a62f-9f54-4000-b280-0581fd91192f\") " pod="openshift-ingress/e2e-ait-0d96587a-openshift-default-bcfbd49cc-xrrp4" Oct 31 00:03:21.404710 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:21.396111 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d310a62f-9f54-4000-b280-0581fd91192f-istio-token\") pod \"e2e-ait-0d96587a-openshift-default-bcfbd49cc-xrrp4\" (UID: \"d310a62f-9f54-4000-b280-0581fd91192f\") " pod="openshift-ingress/e2e-ait-0d96587a-openshift-default-bcfbd49cc-xrrp4" Oct 31 00:03:21.404710 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:21.396251 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d310a62f-9f54-4000-b280-0581fd91192f-workload-socket\") pod \"e2e-ait-0d96587a-openshift-default-bcfbd49cc-xrrp4\" (UID: \"d310a62f-9f54-4000-b280-0581fd91192f\") " pod="openshift-ingress/e2e-ait-0d96587a-openshift-default-bcfbd49cc-xrrp4" Oct 31 00:03:21.404710 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:21.396294 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d310a62f-9f54-4000-b280-0581fd91192f-istio-data\") pod \"e2e-ait-0d96587a-openshift-default-bcfbd49cc-xrrp4\" (UID: \"d310a62f-9f54-4000-b280-0581fd91192f\") " pod="openshift-ingress/e2e-ait-0d96587a-openshift-default-bcfbd49cc-xrrp4" Oct 31 00:03:21.430166 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:21.406009 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d310a62f-9f54-4000-b280-0581fd91192f-istio-data\") pod \"e2e-ait-0d96587a-openshift-default-bcfbd49cc-xrrp4\" (UID: \"d310a62f-9f54-4000-b280-0581fd91192f\") " pod="openshift-ingress/e2e-ait-0d96587a-openshift-default-bcfbd49cc-xrrp4" Oct 31 00:03:21.430166 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:21.406298 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d310a62f-9f54-4000-b280-0581fd91192f-credential-socket\") pod \"e2e-ait-0d96587a-openshift-default-bcfbd49cc-xrrp4\" (UID: \"d310a62f-9f54-4000-b280-0581fd91192f\") " pod="openshift-ingress/e2e-ait-0d96587a-openshift-default-bcfbd49cc-xrrp4" Oct 31 00:03:21.430166 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:21.429200 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d310a62f-9f54-4000-b280-0581fd91192f-istiod-ca-cert\") pod \"e2e-ait-0d96587a-openshift-default-bcfbd49cc-xrrp4\" (UID: \"d310a62f-9f54-4000-b280-0581fd91192f\") " pod="openshift-ingress/e2e-ait-0d96587a-openshift-default-bcfbd49cc-xrrp4" Oct 31 00:03:21.438733 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:21.438683 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d310a62f-9f54-4000-b280-0581fd91192f-workload-socket\") pod \"e2e-ait-0d96587a-openshift-default-bcfbd49cc-xrrp4\" (UID: \"d310a62f-9f54-4000-b280-0581fd91192f\") " pod="openshift-ingress/e2e-ait-0d96587a-openshift-default-bcfbd49cc-xrrp4" Oct 31 00:03:21.454269 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:21.450723 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d310a62f-9f54-4000-b280-0581fd91192f-workload-certs\") pod \"e2e-ait-0d96587a-openshift-default-bcfbd49cc-xrrp4\" (UID: \"d310a62f-9f54-4000-b280-0581fd91192f\") " pod="openshift-ingress/e2e-ait-0d96587a-openshift-default-bcfbd49cc-xrrp4" Oct 31 00:03:21.499140 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:21.494566 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47cd94c7-bvjxk" Oct 31 00:03:21.547389 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:21.540547 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d310a62f-9f54-4000-b280-0581fd91192f-istio-envoy\") pod \"e2e-ait-0d96587a-openshift-default-bcfbd49cc-xrrp4\" (UID: \"d310a62f-9f54-4000-b280-0581fd91192f\") " pod="openshift-ingress/e2e-ait-0d96587a-openshift-default-bcfbd49cc-xrrp4" Oct 31 00:03:21.569180 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:21.568439 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5c3374/premium-w1-5c3374-kserve-6c56d4dcd-llwcl" podUID="dc813f02-e077-4251-8c1e-fea0986b0f36" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded" Oct 31 00:03:21.600431 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:21.569805 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5c3374/premium-w1-5c3374-kserve-6c56d4dcd-llwcl" Oct 31 00:03:21.600431 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:21.571781 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d310a62f-9f54-4000-b280-0581fd91192f-istio-podinfo\") pod \"e2e-ait-0d96587a-openshift-default-bcfbd49cc-xrrp4\" (UID: \"d310a62f-9f54-4000-b280-0581fd91192f\") " pod="openshift-ingress/e2e-ait-0d96587a-openshift-default-bcfbd49cc-xrrp4" Oct 31 00:03:21.600431 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:21.575183 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d310a62f-9f54-4000-b280-0581fd91192f-istio-token\") pod \"e2e-ait-0d96587a-openshift-default-bcfbd49cc-xrrp4\" (UID: \"d310a62f-9f54-4000-b280-0581fd91192f\") " pod="openshift-ingress/e2e-ait-0d96587a-openshift-default-bcfbd49cc-xrrp4" Oct 31 00:03:21.606359 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:21.602428 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dmdsb\" (UniqueName: \"kubernetes.io/projected/d310a62f-9f54-4000-b280-0581fd91192f-kube-api-access-dmdsb\") pod \"e2e-ait-0d96587a-openshift-default-bcfbd49cc-xrrp4\" (UID: \"d310a62f-9f54-4000-b280-0581fd91192f\") " pod="openshift-ingress/e2e-ait-0d96587a-openshift-default-bcfbd49cc-xrrp4" Oct 31 00:03:21.638045 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:21.627519 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5655083c-8498-4488-8ec7-49e4de5d40fd-kube-api-access-d4v46\" (UniqueName: \"kubernetes.io/projected/5655083c-8498-4488-8ec7-49e4de5d40fd-kube-api-access-d4v46\") pod \"5655083c-8498-4488-8ec7-49e4de5d40fd\" (UID: \"5655083c-8498-4488-8ec7-49e4de5d40fd\") " Oct 31 00:03:21.638045 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:21.627680 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5655083c-8498-4488-8ec7-49e4de5d40fd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5655083c-8498-4488-8ec7-49e4de5d40fd-maas-api-xks-ca\") pod \"5655083c-8498-4488-8ec7-49e4de5d40fd\" (UID: \"5655083c-8498-4488-8ec7-49e4de5d40fd\") " Oct 31 00:03:21.638045 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:21.627781 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5655083c-8498-4488-8ec7-49e4de5d40fd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5655083c-8498-4488-8ec7-49e4de5d40fd-maas-api-openshift-ca\") pod \"5655083c-8498-4488-8ec7-49e4de5d40fd\" (UID: \"5655083c-8498-4488-8ec7-49e4de5d40fd\") " Oct 31 00:03:21.638045 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:21.637714 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5655083c-8498-4488-8ec7-49e4de5d40fd-maas-api-openshift-ca" pod "5655083c-8498-4488-8ec7-49e4de5d40fd" (UID: "5655083c-8498-4488-8ec7-49e4de5d40fd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:03:21.657357 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:21.654531 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5655083c-8498-4488-8ec7-49e4de5d40fd-maas-api-xks-ca" pod "5655083c-8498-4488-8ec7-49e4de5d40fd" (UID: "5655083c-8498-4488-8ec7-49e4de5d40fd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:03:21.709362 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:21.676922 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5655083c-8498-4488-8ec7-49e4de5d40fd-kube-api-access-d4v46" pod "5655083c-8498-4488-8ec7-49e4de5d40fd" (UID: "5655083c-8498-4488-8ec7-49e4de5d40fd"). InnerVolumeSpecName "kube-api-access-d4v46". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:03:21.729367 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:21.721444 4927 patch_prober.go:29] interesting pod/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-6gwbm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:03:21.729367 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:21.721525 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-6gwbm" podUID="c4095540-8f80-4704-9146-06c5d2cab638" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:03:21.784268 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:21.743493 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-35a187/distinct-2-w6-35a187-kserve-7ff4b6dc55-944g6" podUID="429ee092-42d8-475e-b0c6-b5a9930766fc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.113:8000/ready\": context deadline exceeded" Oct 31 00:03:21.802381 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:21.784771 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-35a187/distinct-2-w6-35a187-kserve-7ff4b6dc55-944g6" Oct 31 00:03:21.802381 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:21.785712 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-0d96587a-openshift-default-bcfbd49cc-xrrp4" Oct 31 00:03:21.802381 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:21.785800 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d4v46\" (UniqueName: \"kubernetes.io/projected/5655083c-8498-4488-8ec7-49e4de5d40fd-kube-api-access-d4v46\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:03:21.802381 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:21.785824 4927 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5655083c-8498-4488-8ec7-49e4de5d40fd-maas-api-xks-ca\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:03:21.802381 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:21.785839 4927 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5655083c-8498-4488-8ec7-49e4de5d40fd-maas-api-openshift-ca\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:03:22.006674 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:21.922696 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-46b586/simulator-w0-46b586-kserve-67588b8dbd-z8xmn" podUID="9a25e252-223d-41f2-abf7-632067c77e0b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded" Oct 31 00:03:22.006674 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:21.923017 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-27e82b/distinct-w2-27e82b-kserve-75f475fcd9-mccsz" podUID="4ba8ced4-e93f-46ac-87c5-3d0b7fadc02d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/ready\": context deadline exceeded" Oct 31 00:03:22.006674 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:21.923314 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-46b586/simulator-w0-46b586-kserve-67588b8dbd-z8xmn" Oct 31 00:03:22.006674 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:21.923390 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-27e82b/distinct-w2-27e82b-kserve-75f475fcd9-mccsz" Oct 31 00:03:22.009624 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:22.009585 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fe01fa/distinct-2-w5-fe01fa-kserve-79b99966d8-dkhzd" podUID="d5c1c9e8-ce47-4868-860d-b8413ff74ab5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.111:8000/ready\": context deadline exceeded" Oct 31 00:03:22.118370 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:22.114619 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-986d37/premium-w3-986d37-kserve-5c7c7757fb-r99j7" podUID="f4e38df7-355f-4b47-a0b5-7e0b907f3326" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:03:22.828464 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:22.744085 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-35a187/distinct-2-w6-35a187-kserve-7ff4b6dc55-944g6" podUID="429ee092-42d8-475e-b0c6-b5a9930766fc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.113:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:03:23.048360 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:22.608551 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5c3374/premium-w1-5c3374-kserve-6c56d4dcd-llwcl" podUID="dc813f02-e077-4251-8c1e-fea0986b0f36" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:03:23.141412 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:23.114986 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-986d37/premium-w3-986d37-kserve-5c7c7757fb-r99j7" Oct 31 00:03:23.141412 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:23.119523 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fe01fa/distinct-2-w5-fe01fa-kserve-79b99966d8-dkhzd" podUID="d5c1c9e8-ce47-4868-860d-b8413ff74ab5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.111:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:03:23.196773 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:23.196729 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-27e82b/distinct-w2-27e82b-kserve-75f475fcd9-mccsz" podUID="4ba8ced4-e93f-46ac-87c5-3d0b7fadc02d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/ready\": context deadline exceeded" Oct 31 00:03:23.229503 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:23.196767 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-46b586/simulator-w0-46b586-kserve-67588b8dbd-z8xmn" podUID="9a25e252-223d-41f2-abf7-632067c77e0b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded" Oct 31 00:03:23.546858 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:23.519126 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:03:23.726358 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:23.636260 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-986d37/premium-w3-986d37-kserve-5c7c7757fb-r99j7" podUID="f4e38df7-355f-4b47-a0b5-7e0b907f3326" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:03:23.782356 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:23.729028 4927 patch_prober.go:29] interesting pod/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-q7dtb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:03:23.782356 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:23.772814 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-q7dtb" podUID="1f3dbfce-11a6-4176-9a52-c9772a349dbe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:03:23.782356 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:23.756513 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-35a187/distinct-2-w6-35a187-kserve-7ff4b6dc55-944g6" podUID="429ee092-42d8-475e-b0c6-b5a9930766fc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.113:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:03:23.838406 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:23.836454 4927 patch_prober.go:29] interesting pod/e2e-worker-w6-35a187-openshift-default-5b54c65f75-klh6t container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:03:23.838406 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:23.836520 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-klh6t" podUID="b7eb6748-2997-4527-91b3-86900ea70eed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:03:23.978686 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:23.978642 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47cd94c7-bvjxk" event={"ID":"5655083c-8498-4488-8ec7-49e4de5d40fd","Type":"ContainerDied","Data":"d0449ff842ffee0039795da47aeb02f90c8bfa51b7b64c405c3888aa3c5a8dbd"} Oct 31 00:03:24.040377 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:23.982407 4927 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d0449ff842ffee0039795da47aeb02f90c8bfa51b7b64c405c3888aa3c5a8dbd" Oct 31 00:03:24.040377 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:23.979037 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47cd94c7-bvjxk" Oct 31 00:03:24.236266 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:24.119951 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fe01fa/distinct-2-w5-fe01fa-kserve-79b99966d8-dkhzd" podUID="d5c1c9e8-ce47-4868-860d-b8413ff74ab5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.111:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:03:24.508419 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:24.481137 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-46b586/simulator-w0-46b586-kserve-67588b8dbd-z8xmn" podUID="9a25e252-223d-41f2-abf7-632067c77e0b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded" Oct 31 00:03:24.508419 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:24.481555 4927 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-986d37/premium-w3-986d37-kserve-5c7c7757fb-r99j7" podUID="f4e38df7-355f-4b47-a0b5-7e0b907f3326" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/health\": context deadline exceeded" Oct 31 00:03:24.508419 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:24.482306 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-27e82b/distinct-w2-27e82b-kserve-75f475fcd9-mccsz" podUID="4ba8ced4-e93f-46ac-87c5-3d0b7fadc02d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/ready\": context deadline exceeded" Oct 31 00:03:24.508419 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:24.483243 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5c3374/premium-w1-5c3374-kserve-6c56d4dcd-llwcl" podUID="dc813f02-e077-4251-8c1e-fea0986b0f36" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:03:24.508419 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:24.497287 4927 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-35a187/distinct-2-w6-35a187-kserve-7ff4b6dc55-944g6" podUID="429ee092-42d8-475e-b0c6-b5a9930766fc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.113:8000/health\": context deadline exceeded" Oct 31 00:03:24.555357 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:24.547579 4927 patch_prober.go:29] interesting pod/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-d44pq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:03:24.555357 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:24.547643 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-d44pq" podUID="f08dd946-14b1-4546-8fae-01564f3c6908" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:03:24.572435 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:24.569680 4927 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-fe01fa/distinct-2-w5-fe01fa-kserve-79b99966d8-dkhzd" podUID="d5c1c9e8-ce47-4868-860d-b8413ff74ab5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.111:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:03:24.820407 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:24.811511 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-986d37/premium-w3-986d37-kserve-5c7c7757fb-r99j7" podUID="f4e38df7-355f-4b47-a0b5-7e0b907f3326" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded" Oct 31 00:03:24.847852 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:24.820644 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-35a187/distinct-2-w6-35a187-kserve-7ff4b6dc55-944g6" podUID="429ee092-42d8-475e-b0c6-b5a9930766fc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.113:8000/ready\": context deadline exceeded" Oct 31 00:03:25.063403 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:25.048079 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-27e82b/distinct-w2-27e82b-kserve-75f475fcd9-mccsz" Oct 31 00:03:25.149557 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:25.120771 4927 patch_prober.go:29] interesting pod/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-7w9dq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:03:25.243847 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:25.243804 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-7w9dq" podUID="d93f8ab4-0aff-4eba-843d-2c07ab53d3bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:03:25.348221 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:25.314282 4927 patch_prober.go:29] interesting pod/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-vb9r5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:03:25.348221 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:25.314363 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-vb9r5" podUID="334f41bf-6d81-474d-bddc-d38c3492d586" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:03:25.515356 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:25.490061 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5c3374/premium-w1-5c3374-kserve-6c56d4dcd-llwcl" podUID="dc813f02-e077-4251-8c1e-fea0986b0f36" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:03:25.748790 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:25.490737 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-46b586/simulator-w0-46b586-kserve-67588b8dbd-z8xmn" podUID="9a25e252-223d-41f2-abf7-632067c77e0b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:03:25.751376 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:25.751349 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:03:25.863467 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:25.858927 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-35a187/distinct-2-w6-35a187-kserve-7ff4b6dc55-944g6" podUID="429ee092-42d8-475e-b0c6-b5a9930766fc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.113:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:03:25.863467 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:25.859297 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-986d37/premium-w3-986d37-kserve-5c7c7757fb-r99j7" podUID="f4e38df7-355f-4b47-a0b5-7e0b907f3326" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded" Oct 31 00:03:26.148699 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:26.148609 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:03:26.190211 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:26.169622 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:03:26.196692 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:26.190551 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:03:26.197005 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:26.196950 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:03:26.320497 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:26.320450 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fe01fa/distinct-2-w5-fe01fa-kserve-79b99966d8-dkhzd" podUID="d5c1c9e8-ce47-4868-860d-b8413ff74ab5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.111:8000/ready\": context deadline exceeded" Oct 31 00:03:26.456660 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:26.453673 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:03:26.484704 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:03:26.484667 4927 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.395s" Oct 31 00:03:26.484966 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:26.484947 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-0d96587a-openshift-default-bcfbd49cc-xrrp4"] Oct 31 00:03:26.485170 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:26.485154 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-fe01fa/distinct-2-w5-fe01fa-kserve-79b99966d8-dkhzd" Oct 31 00:03:26.523845 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:26.523800 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-27e82b/distinct-w2-27e82b-kserve-75f475fcd9-mccsz" podUID="4ba8ced4-e93f-46ac-87c5-3d0b7fadc02d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:03:26.551359 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:26.535316 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c9c66457-4rw24"] Oct 31 00:03:26.551359 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:26.536495 4927 cpu_manager.go:414] "containerMap: removing container" podUID="5655083c-8498-4488-8ec7-49e4de5d40fd" containerName="revoke-keys" Oct 31 00:03:26.551359 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:26.536517 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5655083c-8498-4488-8ec7-49e4de5d40fd" containerName="revoke-keys" Oct 31 00:03:26.551359 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:26.536701 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="5655083c-8498-4488-8ec7-49e4de5d40fd" containerName="revoke-keys" Oct 31 00:03:26.598578 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:26.598543 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c9c66457-4rw24"] Oct 31 00:03:26.598917 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:26.598897 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9c66457-4rw24" Oct 31 00:03:26.711407 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:26.701108 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4fdzs\"" Oct 31 00:03:26.845358 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:26.796714 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:03:26.845793 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:26.845755 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5c3374/premium-w1-5c3374-kserve-6c56d4dcd-llwcl" podUID="dc813f02-e077-4251-8c1e-fea0986b0f36" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded" Oct 31 00:03:26.871356 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:26.850838 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-46b586/simulator-w0-46b586-kserve-67588b8dbd-z8xmn" podUID="9a25e252-223d-41f2-abf7-632067c77e0b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:03:26.872198 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:26.872151 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-986d37/premium-w3-986d37-kserve-5c7c7757fb-r99j7" podUID="f4e38df7-355f-4b47-a0b5-7e0b907f3326" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:03:26.883386 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:26.872724 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-35a187/distinct-2-w6-35a187-kserve-7ff4b6dc55-944g6" podUID="429ee092-42d8-475e-b0c6-b5a9930766fc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.113:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:03:27.010014 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:27.009915 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dmsdc\" (UniqueName: \"kubernetes.io/projected/351c1f92-30d9-4b47-ad7e-70f102f0a280-kube-api-access-dmsdc\") pod \"maas-api-revoke-sub-c9c66457-4rw24\" (UID: \"351c1f92-30d9-4b47-ad7e-70f102f0a280\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9c66457-4rw24" Oct 31 00:03:27.213677 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:27.211508 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/351c1f92-30d9-4b47-ad7e-70f102f0a280-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c9c66457-4rw24\" (UID: \"351c1f92-30d9-4b47-ad7e-70f102f0a280\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9c66457-4rw24" Oct 31 00:03:27.213677 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:27.211697 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/351c1f92-30d9-4b47-ad7e-70f102f0a280-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c9c66457-4rw24\" (UID: \"351c1f92-30d9-4b47-ad7e-70f102f0a280\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9c66457-4rw24" Oct 31 00:03:27.389357 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:27.388939 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fe01fa/distinct-2-w5-fe01fa-kserve-79b99966d8-dkhzd" podUID="d5c1c9e8-ce47-4868-860d-b8413ff74ab5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.111:8000/ready\": context deadline exceeded" Oct 31 00:03:27.609413 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:27.546621 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-27e82b/distinct-w2-27e82b-kserve-75f475fcd9-mccsz" podUID="4ba8ced4-e93f-46ac-87c5-3d0b7fadc02d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:03:27.644358 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:27.634134 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmsdc\" (UniqueName: \"kubernetes.io/projected/351c1f92-30d9-4b47-ad7e-70f102f0a280-kube-api-access-dmsdc\") pod \"maas-api-revoke-sub-c9c66457-4rw24\" (UID: \"351c1f92-30d9-4b47-ad7e-70f102f0a280\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9c66457-4rw24" Oct 31 00:03:27.644358 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:27.634462 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/351c1f92-30d9-4b47-ad7e-70f102f0a280-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c9c66457-4rw24\" (UID: \"351c1f92-30d9-4b47-ad7e-70f102f0a280\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9c66457-4rw24" Oct 31 00:03:27.679602 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:27.651350 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/351c1f92-30d9-4b47-ad7e-70f102f0a280-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c9c66457-4rw24\" (UID: \"351c1f92-30d9-4b47-ad7e-70f102f0a280\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9c66457-4rw24" Oct 31 00:03:27.743368 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:27.720798 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/351c1f92-30d9-4b47-ad7e-70f102f0a280-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c9c66457-4rw24\" (UID: \"351c1f92-30d9-4b47-ad7e-70f102f0a280\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9c66457-4rw24" Oct 31 00:03:27.876399 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:27.822911 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:03:27.876399 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:27.834803 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dmsdc\" (UniqueName: \"kubernetes.io/projected/351c1f92-30d9-4b47-ad7e-70f102f0a280-kube-api-access-dmsdc\") pod \"maas-api-revoke-sub-c9c66457-4rw24\" (UID: \"351c1f92-30d9-4b47-ad7e-70f102f0a280\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9c66457-4rw24" Oct 31 00:03:28.021599 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:28.021554 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/351c1f92-30d9-4b47-ad7e-70f102f0a280-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c9c66457-4rw24\" (UID: \"351c1f92-30d9-4b47-ad7e-70f102f0a280\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9c66457-4rw24" Oct 31 00:03:28.184892 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:28.071393 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-35a187/distinct-2-w6-35a187-kserve-7ff4b6dc55-944g6" podUID="429ee092-42d8-475e-b0c6-b5a9930766fc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.113:8000/ready\": context deadline exceeded" Oct 31 00:03:28.185723 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:28.071935 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-986d37/premium-w3-986d37-kserve-5c7c7757fb-r99j7" podUID="f4e38df7-355f-4b47-a0b5-7e0b907f3326" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded" Oct 31 00:03:28.186140 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:28.071958 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-46b586/simulator-w0-46b586-kserve-67588b8dbd-z8xmn" podUID="9a25e252-223d-41f2-abf7-632067c77e0b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:03:28.186520 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:03:28.112560 4927 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.01s" Oct 31 00:03:28.186768 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:28.186750 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5c3374/premium-w1-5c3374-kserve-6c56d4dcd-llwcl" Oct 31 00:03:28.186931 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:28.186916 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-fe01fa/distinct-2-w5-fe01fa-kserve-79b99966d8-dkhzd" Oct 31 00:03:28.187062 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:28.187051 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-27e82b/distinct-w2-27e82b-kserve-75f475fcd9-mccsz" Oct 31 00:03:28.187176 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:28.187164 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-fe01fa/distinct-2-w5-fe01fa-kserve-79b99966d8-dkhzd" Oct 31 00:03:28.187290 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:28.187279 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-27e82b/distinct-w2-27e82b-kserve-75f475fcd9-mccsz" Oct 31 00:03:28.187406 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:28.130915 4927 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-46b586/simulator-w0-46b586-kserve-67588b8dbd-z8xmn" podUID="9a25e252-223d-41f2-abf7-632067c77e0b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/health\": context deadline exceeded" Oct 31 00:03:28.295782 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:28.295748 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-35a187/distinct-2-w6-35a187-kserve-7ff4b6dc55-944g6" Oct 31 00:03:28.334382 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:28.328892 4927 scope.go:122] "RemoveContainer" containerID="4e0f6edfeb5185a6651f45cd23c9c0058c57e87df15da1cf5972af1226159634" Oct 31 00:03:28.401260 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:28.399470 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-986d37/premium-w3-986d37-kserve-5c7c7757fb-r99j7" Oct 31 00:03:28.401260 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:28.399528 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0d96587a-openshift-default-bcfbd49cc-xrrp4" event={"ID":"d310a62f-9f54-4000-b280-0581fd91192f","Type":"ContainerStarted","Data":"ae5b79a50331596a3b0abc27a755878cbe38e756b0abf2963eab10d18de553a0"} Oct 31 00:03:28.401260 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:28.399563 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0d96587a-openshift-default-bcfbd49cc-xrrp4" event={"ID":"d310a62f-9f54-4000-b280-0581fd91192f","Type":"ContainerStarted","Data":"d331d7d5f45c04477f2bf720d3faa58524ddad8fff465202e2042c77af99fc8e"} Oct 31 00:03:28.449290 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:28.447500 4927 scope.go:122] "RemoveContainer" containerID="6dd79938b7269d8bd8ed7db80315d358fcef08e7223fec10844476c8687f3e58" Oct 31 00:03:28.457794 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:28.449787 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:03:28.477124 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:28.477083 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-46b586/simulator-w0-46b586-kserve-67588b8dbd-z8xmn" Oct 31 00:03:28.482368 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:28.480325 4927 scope.go:122] "RemoveContainer" containerID="59199777c694eb7ed10cbe8e1086608848bb52e103b9d2a524c675897b93d072" Oct 31 00:03:28.502838 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:28.502812 4927 scope.go:122] "RemoveContainer" containerID="beb361bcaf638007b3141d2c5af0c4969c65f6cc9e687f0327387847362fdece" Oct 31 00:03:28.519291 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:28.519261 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4fdzs\"" Oct 31 00:03:28.525018 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:28.524993 4927 scope.go:122] "RemoveContainer" containerID="888de4f50d24dcad4afad7f9ff2a9763835d2b6dd87987c07328775148cc797b" Oct 31 00:03:28.527111 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:28.527088 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9c66457-4rw24" Oct 31 00:03:28.559917 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:28.557988 4927 scope.go:122] "RemoveContainer" containerID="32f0b272a7ef173c0be0f24ba3215a3f055cbd986f858eb00d742eec60bf56a3" Oct 31 00:03:28.576532 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:28.576326 4927 scope.go:122] "RemoveContainer" containerID="bc7456083fece25a72cbce4e4227c27fd540933b478ba3e6b5b25ba1e46aa2b2" Oct 31 00:03:28.617167 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:28.616778 4927 scope.go:122] "RemoveContainer" containerID="3c50a1544fbd51c805356bf11bbd843333ac2484aa92df7735634b4ab1997ea4" Oct 31 00:03:28.633618 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:28.633196 4927 scope.go:122] "RemoveContainer" containerID="a5dd2ee348c74a1893b88d729dc14b12a7b0ddbf0a953f09df7cdc6d019b6790" Oct 31 00:03:28.652237 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:28.652197 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:03:28.670658 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:28.670631 4927 scope.go:122] "RemoveContainer" containerID="4b3ed1ea4226b920b263173e2ecf9695600e86c822d53648628e23647825caab" Oct 31 00:03:28.684738 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:28.684710 4927 scope.go:122] "RemoveContainer" containerID="6422c129c8065907b4ea48d95adf2c558278dfa9428ae3496d96e4329b817e48" Oct 31 00:03:28.786761 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:28.786738 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-0d96587a-openshift-default-bcfbd49cc-xrrp4" Oct 31 00:03:28.792642 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:28.792620 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-0d96587a-openshift-default-bcfbd49cc-xrrp4" Oct 31 00:03:28.995433 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:28.994263 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c9c66457-4rw24"] Oct 31 00:03:29.050997 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:29.050965 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:03:29.254608 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:29.254508 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:03:29.279548 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:29.279494 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-0d96587a-openshift-default-bcfbd49cc-xrrp4" podStartSLOduration=9.279480494 podStartE2EDuration="9.279480494s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:03:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:03:29.272828689 +0000 UTC m=+1322.309159550" watchObservedRunningTime="2026-10-31 00:03:29.279480494 +0000 UTC m=+1322.315811316" Oct 31 00:03:29.404935 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:29.404902 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9c66457-4rw24" event={"ID":"351c1f92-30d9-4b47-ad7e-70f102f0a280","Type":"ContainerStarted","Data":"775927206e5998415138a1bd3b478886ef84ba2e77211934c5c734789f7cf7bb"} Oct 31 00:03:29.405279 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:29.405264 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-0d96587a-openshift-default-bcfbd49cc-xrrp4" Oct 31 00:03:29.406157 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:29.406138 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-0d96587a-openshift-default-bcfbd49cc-xrrp4" Oct 31 00:03:29.518523 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:29.518426 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:03:29.802250 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:29.802151 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:03:29.903751 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:29.871477 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2609fef-pwzkp"] Oct 31 00:03:29.906536 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:29.906157 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2609fef-pwzkp" Oct 31 00:03:29.938505 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:29.922376 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7kf6\" (UniqueName: \"kubernetes.io/projected/adaebfaf-92b7-459d-ae77-0d78fb8a46f5-kube-api-access-t7kf6\") pod \"maas-api-revoke-sub-d2609fef-pwzkp\" (UID: \"adaebfaf-92b7-459d-ae77-0d78fb8a46f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2609fef-pwzkp" Oct 31 00:03:29.938505 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:29.922594 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/adaebfaf-92b7-459d-ae77-0d78fb8a46f5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2609fef-pwzkp\" (UID: \"adaebfaf-92b7-459d-ae77-0d78fb8a46f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2609fef-pwzkp" Oct 31 00:03:29.938505 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:29.922720 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/adaebfaf-92b7-459d-ae77-0d78fb8a46f5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2609fef-pwzkp\" (UID: \"adaebfaf-92b7-459d-ae77-0d78fb8a46f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2609fef-pwzkp" Oct 31 00:03:29.953784 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:29.948090 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2609fef-pwzkp"] Oct 31 00:03:29.979357 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:29.972646 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:03:30.023658 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:30.023628 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/adaebfaf-92b7-459d-ae77-0d78fb8a46f5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2609fef-pwzkp\" (UID: \"adaebfaf-92b7-459d-ae77-0d78fb8a46f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2609fef-pwzkp" Oct 31 00:03:30.024920 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:30.024900 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/adaebfaf-92b7-459d-ae77-0d78fb8a46f5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2609fef-pwzkp\" (UID: \"adaebfaf-92b7-459d-ae77-0d78fb8a46f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2609fef-pwzkp" Oct 31 00:03:30.025193 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:30.024780 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/adaebfaf-92b7-459d-ae77-0d78fb8a46f5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2609fef-pwzkp\" (UID: \"adaebfaf-92b7-459d-ae77-0d78fb8a46f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2609fef-pwzkp" Oct 31 00:03:30.025293 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:30.025209 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7kf6\" (UniqueName: \"kubernetes.io/projected/adaebfaf-92b7-459d-ae77-0d78fb8a46f5-kube-api-access-t7kf6\") pod \"maas-api-revoke-sub-d2609fef-pwzkp\" (UID: \"adaebfaf-92b7-459d-ae77-0d78fb8a46f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2609fef-pwzkp" Oct 31 00:03:30.027862 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:30.027835 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/adaebfaf-92b7-459d-ae77-0d78fb8a46f5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2609fef-pwzkp\" (UID: \"adaebfaf-92b7-459d-ae77-0d78fb8a46f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2609fef-pwzkp" Oct 31 00:03:30.087183 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:30.087110 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7kf6\" (UniqueName: \"kubernetes.io/projected/adaebfaf-92b7-459d-ae77-0d78fb8a46f5-kube-api-access-t7kf6\") pod \"maas-api-revoke-sub-d2609fef-pwzkp\" (UID: \"adaebfaf-92b7-459d-ae77-0d78fb8a46f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2609fef-pwzkp" Oct 31 00:03:30.273355 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:30.273298 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2609fef-pwzkp" Oct 31 00:03:30.427121 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:30.427083 4927 generic.go:356] "Generic (PLEG): container finished" podID="351c1f92-30d9-4b47-ad7e-70f102f0a280" containerID="b64527904348f4d5e2d655dfac4d8fda191042979ea4c2768e86372c9963b6a9" exitCode=0 Oct 31 00:03:30.427356 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:30.427276 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9c66457-4rw24" event={"ID":"351c1f92-30d9-4b47-ad7e-70f102f0a280","Type":"ContainerDied","Data":"b64527904348f4d5e2d655dfac4d8fda191042979ea4c2768e86372c9963b6a9"} Oct 31 00:03:30.669918 ip-10-0-141-180 kubenswrapper[4927]: W1031 00:03:30.669888 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadaebfaf_92b7_459d_ae77_0d78fb8a46f5.slice/crio-68dcab3659fa07b0f257b94da89bf1f7c8a291fe64dbcbb70e2d290c6da40372 WatchSource:0}: Error finding container 68dcab3659fa07b0f257b94da89bf1f7c8a291fe64dbcbb70e2d290c6da40372: Status 404 returned error can't find the container with id 68dcab3659fa07b0f257b94da89bf1f7c8a291fe64dbcbb70e2d290c6da40372 Oct 31 00:03:30.671827 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:30.671801 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2609fef-pwzkp"] Oct 31 00:03:30.782576 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:30.782545 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:03:31.441228 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:31.441173 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2609fef-pwzkp" event={"ID":"adaebfaf-92b7-459d-ae77-0d78fb8a46f5","Type":"ContainerStarted","Data":"929566ae98b0eb7a0ca1bb9d566a078364e0ed57cd4a769374a3e9fc79214289"} Oct 31 00:03:31.441768 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:31.441245 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2609fef-pwzkp" event={"ID":"adaebfaf-92b7-459d-ae77-0d78fb8a46f5","Type":"ContainerStarted","Data":"68dcab3659fa07b0f257b94da89bf1f7c8a291fe64dbcbb70e2d290c6da40372"} Oct 31 00:03:31.560957 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:31.560861 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2609fef-pwzkp" podStartSLOduration=2.560845778 podStartE2EDuration="2.560845778s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:03:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:03:31.560325779 +0000 UTC m=+1324.596656603" watchObservedRunningTime="2026-10-31 00:03:31.560845778 +0000 UTC m=+1324.597176600" Oct 31 00:03:31.643952 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:31.643919 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9c66457-4rw24" Oct 31 00:03:31.651715 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:31.651680 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/351c1f92-30d9-4b47-ad7e-70f102f0a280-kube-api-access-dmsdc\" (UniqueName: \"kubernetes.io/projected/351c1f92-30d9-4b47-ad7e-70f102f0a280-kube-api-access-dmsdc\") pod \"351c1f92-30d9-4b47-ad7e-70f102f0a280\" (UID: \"351c1f92-30d9-4b47-ad7e-70f102f0a280\") " Oct 31 00:03:31.651868 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:31.651773 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/351c1f92-30d9-4b47-ad7e-70f102f0a280-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/351c1f92-30d9-4b47-ad7e-70f102f0a280-maas-api-xks-ca\") pod \"351c1f92-30d9-4b47-ad7e-70f102f0a280\" (UID: \"351c1f92-30d9-4b47-ad7e-70f102f0a280\") " Oct 31 00:03:31.652028 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:31.651998 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/351c1f92-30d9-4b47-ad7e-70f102f0a280-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/351c1f92-30d9-4b47-ad7e-70f102f0a280-maas-api-openshift-ca\") pod \"351c1f92-30d9-4b47-ad7e-70f102f0a280\" (UID: \"351c1f92-30d9-4b47-ad7e-70f102f0a280\") " Oct 31 00:03:31.652406 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:31.652371 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/351c1f92-30d9-4b47-ad7e-70f102f0a280-maas-api-openshift-ca" pod "351c1f92-30d9-4b47-ad7e-70f102f0a280" (UID: "351c1f92-30d9-4b47-ad7e-70f102f0a280"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:03:31.652820 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:31.652794 4927 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/351c1f92-30d9-4b47-ad7e-70f102f0a280-maas-api-openshift-ca\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:03:31.653966 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:31.653935 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/351c1f92-30d9-4b47-ad7e-70f102f0a280-kube-api-access-dmsdc" pod "351c1f92-30d9-4b47-ad7e-70f102f0a280" (UID: "351c1f92-30d9-4b47-ad7e-70f102f0a280"). InnerVolumeSpecName "kube-api-access-dmsdc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:03:31.654418 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:31.654385 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/351c1f92-30d9-4b47-ad7e-70f102f0a280-maas-api-xks-ca" pod "351c1f92-30d9-4b47-ad7e-70f102f0a280" (UID: "351c1f92-30d9-4b47-ad7e-70f102f0a280"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:03:31.754576 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:31.754540 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dmsdc\" (UniqueName: \"kubernetes.io/projected/351c1f92-30d9-4b47-ad7e-70f102f0a280-kube-api-access-dmsdc\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:03:31.754576 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:31.754572 4927 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/351c1f92-30d9-4b47-ad7e-70f102f0a280-maas-api-xks-ca\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:03:32.344044 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:32.343954 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:03:32.459167 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:32.459131 4927 generic.go:356] "Generic (PLEG): container finished" podID="adaebfaf-92b7-459d-ae77-0d78fb8a46f5" containerID="929566ae98b0eb7a0ca1bb9d566a078364e0ed57cd4a769374a3e9fc79214289" exitCode=0 Oct 31 00:03:32.459634 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:32.459220 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2609fef-pwzkp" event={"ID":"adaebfaf-92b7-459d-ae77-0d78fb8a46f5","Type":"ContainerDied","Data":"929566ae98b0eb7a0ca1bb9d566a078364e0ed57cd4a769374a3e9fc79214289"} Oct 31 00:03:32.460940 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:32.460911 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9c66457-4rw24" Oct 31 00:03:32.461091 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:32.460955 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9c66457-4rw24" event={"ID":"351c1f92-30d9-4b47-ad7e-70f102f0a280","Type":"ContainerDied","Data":"775927206e5998415138a1bd3b478886ef84ba2e77211934c5c734789f7cf7bb"} Oct 31 00:03:32.461091 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:32.460993 4927 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="775927206e5998415138a1bd3b478886ef84ba2e77211934c5c734789f7cf7bb" Oct 31 00:03:32.572464 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:32.572422 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:03:32.760898 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:32.760865 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:03:32.920565 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:32.920524 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:03:33.233355 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:33.229892 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:03:33.480355 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:33.475503 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-8e7d0098-6794f6ff67-429f5"] Oct 31 00:03:33.480355 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:33.476521 4927 cpu_manager.go:414] "containerMap: removing container" podUID="351c1f92-30d9-4b47-ad7e-70f102f0a280" containerName="revoke-keys" Oct 31 00:03:33.480355 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:33.476542 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="351c1f92-30d9-4b47-ad7e-70f102f0a280" containerName="revoke-keys" Oct 31 00:03:33.480355 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:33.476747 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="351c1f92-30d9-4b47-ad7e-70f102f0a280" containerName="revoke-keys" Oct 31 00:03:33.549760 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:33.528643 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-8e7d0098-6794f6ff67-429f5" Oct 31 00:03:33.570408 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:33.562409 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-8e7d0098-6794f6ff67-429f5"] Oct 31 00:03:33.688123 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:33.684285 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-8e7d0098-dockercfg-xhh8c\"" Oct 31 00:03:33.688123 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:33.684401 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-8e7d0098\"" Oct 31 00:03:33.699876 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:33.696393 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5779f4e6-fab2-4060-a6e6-b64f4756d7a5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-8e7d0098-6794f6ff67-429f5\" (UID: \"5779f4e6-fab2-4060-a6e6-b64f4756d7a5\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-8e7d0098-6794f6ff67-429f5" Oct 31 00:03:33.699876 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:33.696501 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lp9tn\" (UniqueName: \"kubernetes.io/projected/5779f4e6-fab2-4060-a6e6-b64f4756d7a5-kube-api-access-lp9tn\") pod \"payload-pre-processing-e2e-ait-8e7d0098-6794f6ff67-429f5\" (UID: \"5779f4e6-fab2-4060-a6e6-b64f4756d7a5\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-8e7d0098-6794f6ff67-429f5" Oct 31 00:03:33.756867 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:33.756773 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-8e7d0098-8f775757b-4mt6m"] Oct 31 00:03:33.805871 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:33.802819 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5779f4e6-fab2-4060-a6e6-b64f4756d7a5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-8e7d0098-6794f6ff67-429f5\" (UID: \"5779f4e6-fab2-4060-a6e6-b64f4756d7a5\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-8e7d0098-6794f6ff67-429f5" Oct 31 00:03:33.805871 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:33.802910 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lp9tn\" (UniqueName: \"kubernetes.io/projected/5779f4e6-fab2-4060-a6e6-b64f4756d7a5-kube-api-access-lp9tn\") pod \"payload-pre-processing-e2e-ait-8e7d0098-6794f6ff67-429f5\" (UID: \"5779f4e6-fab2-4060-a6e6-b64f4756d7a5\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-8e7d0098-6794f6ff67-429f5" Oct 31 00:03:33.806216 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:33.806140 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5779f4e6-fab2-4060-a6e6-b64f4756d7a5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-8e7d0098-6794f6ff67-429f5\" (UID: \"5779f4e6-fab2-4060-a6e6-b64f4756d7a5\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-8e7d0098-6794f6ff67-429f5" Oct 31 00:03:33.809356 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:33.808910 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-8e7d0098-8f775757b-4mt6m"] Oct 31 00:03:33.809356 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:33.809033 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-8e7d0098-8f775757b-4mt6m" Oct 31 00:03:33.900366 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:33.897074 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lp9tn\" (UniqueName: \"kubernetes.io/projected/5779f4e6-fab2-4060-a6e6-b64f4756d7a5-kube-api-access-lp9tn\") pod \"payload-pre-processing-e2e-ait-8e7d0098-6794f6ff67-429f5\" (UID: \"5779f4e6-fab2-4060-a6e6-b64f4756d7a5\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-8e7d0098-6794f6ff67-429f5" Oct 31 00:03:33.906355 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:33.904531 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wl9zg\" (UniqueName: \"kubernetes.io/projected/daee2336-91a5-4c33-8bdf-b1c0e5538c35-kube-api-access-wl9zg\") pod \"payload-processing-e2e-ait-8e7d0098-8f775757b-4mt6m\" (UID: \"daee2336-91a5-4c33-8bdf-b1c0e5538c35\") " pod="openshift-ingress/payload-processing-e2e-ait-8e7d0098-8f775757b-4mt6m" Oct 31 00:03:33.906355 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:33.904626 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/daee2336-91a5-4c33-8bdf-b1c0e5538c35-plugins-config-volume\") pod \"payload-processing-e2e-ait-8e7d0098-8f775757b-4mt6m\" (UID: \"daee2336-91a5-4c33-8bdf-b1c0e5538c35\") " pod="openshift-ingress/payload-processing-e2e-ait-8e7d0098-8f775757b-4mt6m" Oct 31 00:03:33.935222 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:33.933974 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-8e7d0098-6794f6ff67-429f5" Oct 31 00:03:34.006079 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:34.006028 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wl9zg\" (UniqueName: \"kubernetes.io/projected/daee2336-91a5-4c33-8bdf-b1c0e5538c35-kube-api-access-wl9zg\") pod \"payload-processing-e2e-ait-8e7d0098-8f775757b-4mt6m\" (UID: \"daee2336-91a5-4c33-8bdf-b1c0e5538c35\") " pod="openshift-ingress/payload-processing-e2e-ait-8e7d0098-8f775757b-4mt6m" Oct 31 00:03:34.006295 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:34.006144 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/daee2336-91a5-4c33-8bdf-b1c0e5538c35-plugins-config-volume\") pod \"payload-processing-e2e-ait-8e7d0098-8f775757b-4mt6m\" (UID: \"daee2336-91a5-4c33-8bdf-b1c0e5538c35\") " pod="openshift-ingress/payload-processing-e2e-ait-8e7d0098-8f775757b-4mt6m" Oct 31 00:03:34.007222 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:34.007190 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/daee2336-91a5-4c33-8bdf-b1c0e5538c35-plugins-config-volume\") pod \"payload-processing-e2e-ait-8e7d0098-8f775757b-4mt6m\" (UID: \"daee2336-91a5-4c33-8bdf-b1c0e5538c35\") " pod="openshift-ingress/payload-processing-e2e-ait-8e7d0098-8f775757b-4mt6m" Oct 31 00:03:34.069239 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:34.069159 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wl9zg\" (UniqueName: \"kubernetes.io/projected/daee2336-91a5-4c33-8bdf-b1c0e5538c35-kube-api-access-wl9zg\") pod \"payload-processing-e2e-ait-8e7d0098-8f775757b-4mt6m\" (UID: \"daee2336-91a5-4c33-8bdf-b1c0e5538c35\") " pod="openshift-ingress/payload-processing-e2e-ait-8e7d0098-8f775757b-4mt6m" Oct 31 00:03:34.110799 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:34.110771 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2609fef-pwzkp" Oct 31 00:03:34.144448 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:34.144420 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-8e7d0098-8f775757b-4mt6m" Oct 31 00:03:34.208488 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:34.208465 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/adaebfaf-92b7-459d-ae77-0d78fb8a46f5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/adaebfaf-92b7-459d-ae77-0d78fb8a46f5-maas-api-openshift-ca\") pod \"adaebfaf-92b7-459d-ae77-0d78fb8a46f5\" (UID: \"adaebfaf-92b7-459d-ae77-0d78fb8a46f5\") " Oct 31 00:03:34.208606 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:34.208573 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/adaebfaf-92b7-459d-ae77-0d78fb8a46f5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/adaebfaf-92b7-459d-ae77-0d78fb8a46f5-maas-api-xks-ca\") pod \"adaebfaf-92b7-459d-ae77-0d78fb8a46f5\" (UID: \"adaebfaf-92b7-459d-ae77-0d78fb8a46f5\") " Oct 31 00:03:34.208688 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:34.208665 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/adaebfaf-92b7-459d-ae77-0d78fb8a46f5-kube-api-access-t7kf6\" (UniqueName: \"kubernetes.io/projected/adaebfaf-92b7-459d-ae77-0d78fb8a46f5-kube-api-access-t7kf6\") pod \"adaebfaf-92b7-459d-ae77-0d78fb8a46f5\" (UID: \"adaebfaf-92b7-459d-ae77-0d78fb8a46f5\") " Oct 31 00:03:34.209837 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:34.209797 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/adaebfaf-92b7-459d-ae77-0d78fb8a46f5-maas-api-openshift-ca" pod "adaebfaf-92b7-459d-ae77-0d78fb8a46f5" (UID: "adaebfaf-92b7-459d-ae77-0d78fb8a46f5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:03:34.212143 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:34.212114 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/adaebfaf-92b7-459d-ae77-0d78fb8a46f5-maas-api-xks-ca" pod "adaebfaf-92b7-459d-ae77-0d78fb8a46f5" (UID: "adaebfaf-92b7-459d-ae77-0d78fb8a46f5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:03:34.212294 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:34.212268 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/adaebfaf-92b7-459d-ae77-0d78fb8a46f5-kube-api-access-t7kf6" pod "adaebfaf-92b7-459d-ae77-0d78fb8a46f5" (UID: "adaebfaf-92b7-459d-ae77-0d78fb8a46f5"). InnerVolumeSpecName "kube-api-access-t7kf6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:03:34.310637 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:34.310599 4927 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/adaebfaf-92b7-459d-ae77-0d78fb8a46f5-maas-api-xks-ca\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:03:34.310637 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:34.310633 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t7kf6\" (UniqueName: \"kubernetes.io/projected/adaebfaf-92b7-459d-ae77-0d78fb8a46f5-kube-api-access-t7kf6\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:03:34.310840 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:34.310652 4927 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/adaebfaf-92b7-459d-ae77-0d78fb8a46f5-maas-api-openshift-ca\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:03:34.349961 ip-10-0-141-180 kubenswrapper[4927]: W1031 00:03:34.349921 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5779f4e6_fab2_4060_a6e6_b64f4756d7a5.slice/crio-4dc4a9688df47d0033355b06bc79f8f5a904886816347686ce9ae04f9fa14dbc WatchSource:0}: Error finding container 4dc4a9688df47d0033355b06bc79f8f5a904886816347686ce9ae04f9fa14dbc: Status 404 returned error can't find the container with id 4dc4a9688df47d0033355b06bc79f8f5a904886816347686ce9ae04f9fa14dbc Oct 31 00:03:34.350465 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:34.350439 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-8e7d0098-6794f6ff67-429f5"] Oct 31 00:03:34.533092 ip-10-0-141-180 kubenswrapper[4927]: W1031 00:03:34.533059 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddaee2336_91a5_4c33_8bdf_b1c0e5538c35.slice/crio-f79ced99e90bcecaa11a4aee39003cf3c3316e2c7c7c72500e30a621ada3d7b3 WatchSource:0}: Error finding container f79ced99e90bcecaa11a4aee39003cf3c3316e2c7c7c72500e30a621ada3d7b3: Status 404 returned error can't find the container with id f79ced99e90bcecaa11a4aee39003cf3c3316e2c7c7c72500e30a621ada3d7b3 Oct 31 00:03:34.534821 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:34.534734 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-8e7d0098-8f775757b-4mt6m"] Oct 31 00:03:34.723663 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:34.723632 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2609fef-pwzkp" Oct 31 00:03:34.723663 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:34.723642 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2609fef-pwzkp" event={"ID":"adaebfaf-92b7-459d-ae77-0d78fb8a46f5","Type":"ContainerDied","Data":"68dcab3659fa07b0f257b94da89bf1f7c8a291fe64dbcbb70e2d290c6da40372"} Oct 31 00:03:34.723923 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:34.723693 4927 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="68dcab3659fa07b0f257b94da89bf1f7c8a291fe64dbcbb70e2d290c6da40372" Oct 31 00:03:34.725399 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:34.725370 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-8e7d0098-8f775757b-4mt6m" event={"ID":"daee2336-91a5-4c33-8bdf-b1c0e5538c35","Type":"ContainerStarted","Data":"c80e4c0cfb6a79697cd6d5bfb8b83d4e602034d065f2423ad284d2f5a43b9a85"} Oct 31 00:03:34.725549 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:34.725417 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-8e7d0098-8f775757b-4mt6m" event={"ID":"daee2336-91a5-4c33-8bdf-b1c0e5538c35","Type":"ContainerStarted","Data":"f79ced99e90bcecaa11a4aee39003cf3c3316e2c7c7c72500e30a621ada3d7b3"} Oct 31 00:03:34.725549 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:34.725445 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-8e7d0098-8f775757b-4mt6m" Oct 31 00:03:34.727154 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:34.727130 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-8e7d0098-6794f6ff67-429f5" event={"ID":"5779f4e6-fab2-4060-a6e6-b64f4756d7a5","Type":"ContainerStarted","Data":"d7e2809ce27ccd8fc6ea633bc9188efa8209d8d8f5e88252632b3c8c99397927"} Oct 31 00:03:34.727273 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:34.727183 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-8e7d0098-6794f6ff67-429f5" Oct 31 00:03:34.727273 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:34.727209 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-8e7d0098-6794f6ff67-429f5" event={"ID":"5779f4e6-fab2-4060-a6e6-b64f4756d7a5","Type":"ContainerStarted","Data":"4dc4a9688df47d0033355b06bc79f8f5a904886816347686ce9ae04f9fa14dbc"} Oct 31 00:03:34.852562 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:34.852502 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-8e7d0098-8f775757b-4mt6m" podStartSLOduration=1.852482698 podStartE2EDuration="1.852482698s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:03:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:03:34.843019228 +0000 UTC m=+1327.879350050" watchObservedRunningTime="2026-10-31 00:03:34.852482698 +0000 UTC m=+1327.888813522" Oct 31 00:03:35.027448 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:35.026834 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-8e7d0098-6794f6ff67-429f5" podStartSLOduration=2.026812174 podStartE2EDuration="2.026812174s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:03:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:03:35.014630011 +0000 UTC m=+1328.050960834" watchObservedRunningTime="2026-10-31 00:03:35.026812174 +0000 UTC m=+1328.063143001" Oct 31 00:03:35.831288 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:35.831255 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:03:37.476310 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:37.476272 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:03:37.683403 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:37.670523 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:03:38.223432 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:38.223390 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-0d96587a-584d7b5dff-ccjj6"] Oct 31 00:03:38.263357 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:38.249644 4927 cpu_manager.go:414] "containerMap: removing container" podUID="adaebfaf-92b7-459d-ae77-0d78fb8a46f5" containerName="revoke-keys" Oct 31 00:03:38.327844 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:38.327812 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="adaebfaf-92b7-459d-ae77-0d78fb8a46f5" containerName="revoke-keys" Oct 31 00:03:38.328373 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:38.328348 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="adaebfaf-92b7-459d-ae77-0d78fb8a46f5" containerName="revoke-keys" Oct 31 00:03:38.365356 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:38.356818 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-0d96587a-584d7b5dff-ccjj6" Oct 31 00:03:38.389981 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:38.389567 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-0d96587a-584d7b5dff-ccjj6"] Oct 31 00:03:38.414989 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:38.414956 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-0d96587a\"" Oct 31 00:03:38.435743 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:38.435606 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-0d96587a-58d9d8846-j62hj"] Oct 31 00:03:38.447910 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:38.447884 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-0d96587a-58d9d8846-j62hj"] Oct 31 00:03:38.448167 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:38.448144 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-0d96587a-58d9d8846-j62hj" Oct 31 00:03:38.467483 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:38.466139 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-0d96587a-dockercfg-2zdct\"" Oct 31 00:03:38.470995 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:38.470961 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46996ba2-e5f5-4d2a-897b-7d82db8ba74f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-0d96587a-584d7b5dff-ccjj6\" (UID: \"46996ba2-e5f5-4d2a-897b-7d82db8ba74f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-0d96587a-584d7b5dff-ccjj6" Oct 31 00:03:38.471941 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:38.471272 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4mdbm\" (UniqueName: \"kubernetes.io/projected/46996ba2-e5f5-4d2a-897b-7d82db8ba74f-kube-api-access-4mdbm\") pod \"payload-pre-processing-e2e-ait-0d96587a-584d7b5dff-ccjj6\" (UID: \"46996ba2-e5f5-4d2a-897b-7d82db8ba74f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-0d96587a-584d7b5dff-ccjj6" Oct 31 00:03:38.573439 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:38.573323 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mz695\" (UniqueName: \"kubernetes.io/projected/0668f592-496f-412b-92c6-23b551aff729-kube-api-access-mz695\") pod \"payload-processing-e2e-ait-0d96587a-58d9d8846-j62hj\" (UID: \"0668f592-496f-412b-92c6-23b551aff729\") " pod="openshift-ingress/payload-processing-e2e-ait-0d96587a-58d9d8846-j62hj" Oct 31 00:03:38.573794 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:38.573497 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mdbm\" (UniqueName: \"kubernetes.io/projected/46996ba2-e5f5-4d2a-897b-7d82db8ba74f-kube-api-access-4mdbm\") pod \"payload-pre-processing-e2e-ait-0d96587a-584d7b5dff-ccjj6\" (UID: \"46996ba2-e5f5-4d2a-897b-7d82db8ba74f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-0d96587a-584d7b5dff-ccjj6" Oct 31 00:03:38.573794 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:38.573531 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0668f592-496f-412b-92c6-23b551aff729-plugins-config-volume\") pod \"payload-processing-e2e-ait-0d96587a-58d9d8846-j62hj\" (UID: \"0668f592-496f-412b-92c6-23b551aff729\") " pod="openshift-ingress/payload-processing-e2e-ait-0d96587a-58d9d8846-j62hj" Oct 31 00:03:38.573794 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:38.573762 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46996ba2-e5f5-4d2a-897b-7d82db8ba74f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-0d96587a-584d7b5dff-ccjj6\" (UID: \"46996ba2-e5f5-4d2a-897b-7d82db8ba74f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-0d96587a-584d7b5dff-ccjj6" Oct 31 00:03:38.574470 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:38.574446 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46996ba2-e5f5-4d2a-897b-7d82db8ba74f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-0d96587a-584d7b5dff-ccjj6\" (UID: \"46996ba2-e5f5-4d2a-897b-7d82db8ba74f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-0d96587a-584d7b5dff-ccjj6" Oct 31 00:03:38.636775 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:38.636746 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4mdbm\" (UniqueName: \"kubernetes.io/projected/46996ba2-e5f5-4d2a-897b-7d82db8ba74f-kube-api-access-4mdbm\") pod \"payload-pre-processing-e2e-ait-0d96587a-584d7b5dff-ccjj6\" (UID: \"46996ba2-e5f5-4d2a-897b-7d82db8ba74f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-0d96587a-584d7b5dff-ccjj6" Oct 31 00:03:38.675478 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:38.675448 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mz695\" (UniqueName: \"kubernetes.io/projected/0668f592-496f-412b-92c6-23b551aff729-kube-api-access-mz695\") pod \"payload-processing-e2e-ait-0d96587a-58d9d8846-j62hj\" (UID: \"0668f592-496f-412b-92c6-23b551aff729\") " pod="openshift-ingress/payload-processing-e2e-ait-0d96587a-58d9d8846-j62hj" Oct 31 00:03:38.675632 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:38.675534 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0668f592-496f-412b-92c6-23b551aff729-plugins-config-volume\") pod \"payload-processing-e2e-ait-0d96587a-58d9d8846-j62hj\" (UID: \"0668f592-496f-412b-92c6-23b551aff729\") " pod="openshift-ingress/payload-processing-e2e-ait-0d96587a-58d9d8846-j62hj" Oct 31 00:03:38.676242 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:38.676215 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0668f592-496f-412b-92c6-23b551aff729-plugins-config-volume\") pod \"payload-processing-e2e-ait-0d96587a-58d9d8846-j62hj\" (UID: \"0668f592-496f-412b-92c6-23b551aff729\") " pod="openshift-ingress/payload-processing-e2e-ait-0d96587a-58d9d8846-j62hj" Oct 31 00:03:38.725078 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:38.725046 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-0d96587a-584d7b5dff-ccjj6" Oct 31 00:03:38.738396 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:38.738370 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mz695\" (UniqueName: \"kubernetes.io/projected/0668f592-496f-412b-92c6-23b551aff729-kube-api-access-mz695\") pod \"payload-processing-e2e-ait-0d96587a-58d9d8846-j62hj\" (UID: \"0668f592-496f-412b-92c6-23b551aff729\") " pod="openshift-ingress/payload-processing-e2e-ait-0d96587a-58d9d8846-j62hj" Oct 31 00:03:38.768521 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:38.768486 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-0d96587a-58d9d8846-j62hj" Oct 31 00:03:38.956873 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:38.956838 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:03:39.112494 ip-10-0-141-180 kubenswrapper[4927]: W1031 00:03:39.112449 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46996ba2_e5f5_4d2a_897b_7d82db8ba74f.slice/crio-f3a20c1bcac87e6971f0015d417fcd4e50966a88be6d456f22a794a424a0100d WatchSource:0}: Error finding container f3a20c1bcac87e6971f0015d417fcd4e50966a88be6d456f22a794a424a0100d: Status 404 returned error can't find the container with id f3a20c1bcac87e6971f0015d417fcd4e50966a88be6d456f22a794a424a0100d Oct 31 00:03:39.114314 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:39.114290 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-0d96587a-584d7b5dff-ccjj6"] Oct 31 00:03:39.227751 ip-10-0-141-180 kubenswrapper[4927]: W1031 00:03:39.227720 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0668f592_496f_412b_92c6_23b551aff729.slice/crio-c49e2ec0600c34aad0df0e25037a218d4cbb33e172bac9dd72734cb23275c2f4 WatchSource:0}: Error finding container c49e2ec0600c34aad0df0e25037a218d4cbb33e172bac9dd72734cb23275c2f4: Status 404 returned error can't find the container with id c49e2ec0600c34aad0df0e25037a218d4cbb33e172bac9dd72734cb23275c2f4 Oct 31 00:03:39.229250 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:39.229222 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-0d96587a-58d9d8846-j62hj"] Oct 31 00:03:39.605819 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:39.603602 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:03:39.867403 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:39.861034 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:03:40.064352 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:40.052252 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-0d96587a-584d7b5dff-ccjj6" event={"ID":"46996ba2-e5f5-4d2a-897b-7d82db8ba74f","Type":"ContainerStarted","Data":"9cc22c403958fcddd41f8cdf1b9e66e82f32ab9313cd99ed5e0cf4f054cd0e7a"} Oct 31 00:03:40.064352 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:40.052320 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-0d96587a-584d7b5dff-ccjj6" event={"ID":"46996ba2-e5f5-4d2a-897b-7d82db8ba74f","Type":"ContainerStarted","Data":"f3a20c1bcac87e6971f0015d417fcd4e50966a88be6d456f22a794a424a0100d"} Oct 31 00:03:40.064352 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:40.053537 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-0d96587a-584d7b5dff-ccjj6" Oct 31 00:03:40.064352 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:40.055872 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-0d96587a-58d9d8846-j62hj" event={"ID":"0668f592-496f-412b-92c6-23b551aff729","Type":"ContainerStarted","Data":"a4dd6e554576bacab85bf0e0513a9cac19874cd497b4ebed88e29c72ac90bb3f"} Oct 31 00:03:40.064352 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:40.055923 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-0d96587a-58d9d8846-j62hj" event={"ID":"0668f592-496f-412b-92c6-23b551aff729","Type":"ContainerStarted","Data":"c49e2ec0600c34aad0df0e25037a218d4cbb33e172bac9dd72734cb23275c2f4"} Oct 31 00:03:40.064352 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:40.056874 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-0d96587a-58d9d8846-j62hj" Oct 31 00:03:40.184804 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:40.184754 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-0d96587a-584d7b5dff-ccjj6" podStartSLOduration=3.184739155 podStartE2EDuration="3.184739155s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:03:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:03:40.17994594 +0000 UTC m=+1333.216276781" watchObservedRunningTime="2026-10-31 00:03:40.184739155 +0000 UTC m=+1333.221069976" Oct 31 00:03:40.288873 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:40.288838 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:03:40.354358 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:40.341503 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-0d96587a-58d9d8846-j62hj" podStartSLOduration=3.341480834 podStartE2EDuration="3.341480834s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:03:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:03:40.333252914 +0000 UTC m=+1333.369583782" watchObservedRunningTime="2026-10-31 00:03:40.341480834 +0000 UTC m=+1333.377811657" Oct 31 00:03:40.582831 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:40.582750 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ea831844-gf8cw"] Oct 31 00:03:40.587642 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:40.587622 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea831844-gf8cw" Oct 31 00:03:40.629259 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:40.629019 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ea831844-gf8cw"] Oct 31 00:03:40.634718 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:40.634680 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:03:40.641503 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:40.641465 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:03:40.641714 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:40.641689 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4fdzs\"" Oct 31 00:03:40.710575 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:40.709647 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n9zd8\" (UniqueName: \"kubernetes.io/projected/55ce234b-b2e4-48ca-ae22-57598d4bd8db-kube-api-access-n9zd8\") pod \"maas-api-revoke-sub-ea831844-gf8cw\" (UID: \"55ce234b-b2e4-48ca-ae22-57598d4bd8db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea831844-gf8cw" Oct 31 00:03:40.710575 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:40.709763 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55ce234b-b2e4-48ca-ae22-57598d4bd8db-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ea831844-gf8cw\" (UID: \"55ce234b-b2e4-48ca-ae22-57598d4bd8db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea831844-gf8cw" Oct 31 00:03:40.710575 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:40.709832 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55ce234b-b2e4-48ca-ae22-57598d4bd8db-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ea831844-gf8cw\" (UID: \"55ce234b-b2e4-48ca-ae22-57598d4bd8db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea831844-gf8cw" Oct 31 00:03:40.811309 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:40.811273 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9zd8\" (UniqueName: \"kubernetes.io/projected/55ce234b-b2e4-48ca-ae22-57598d4bd8db-kube-api-access-n9zd8\") pod \"maas-api-revoke-sub-ea831844-gf8cw\" (UID: \"55ce234b-b2e4-48ca-ae22-57598d4bd8db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea831844-gf8cw" Oct 31 00:03:40.811488 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:40.811397 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55ce234b-b2e4-48ca-ae22-57598d4bd8db-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ea831844-gf8cw\" (UID: \"55ce234b-b2e4-48ca-ae22-57598d4bd8db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea831844-gf8cw" Oct 31 00:03:40.811488 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:40.811466 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55ce234b-b2e4-48ca-ae22-57598d4bd8db-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ea831844-gf8cw\" (UID: \"55ce234b-b2e4-48ca-ae22-57598d4bd8db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea831844-gf8cw" Oct 31 00:03:40.812190 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:40.812159 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55ce234b-b2e4-48ca-ae22-57598d4bd8db-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ea831844-gf8cw\" (UID: \"55ce234b-b2e4-48ca-ae22-57598d4bd8db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea831844-gf8cw" Oct 31 00:03:40.813978 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:40.813948 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55ce234b-b2e4-48ca-ae22-57598d4bd8db-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ea831844-gf8cw\" (UID: \"55ce234b-b2e4-48ca-ae22-57598d4bd8db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea831844-gf8cw" Oct 31 00:03:40.874278 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:40.874198 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9zd8\" (UniqueName: \"kubernetes.io/projected/55ce234b-b2e4-48ca-ae22-57598d4bd8db-kube-api-access-n9zd8\") pod \"maas-api-revoke-sub-ea831844-gf8cw\" (UID: \"55ce234b-b2e4-48ca-ae22-57598d4bd8db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea831844-gf8cw" Oct 31 00:03:40.903054 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:40.903020 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea831844-gf8cw" Oct 31 00:03:41.297982 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:41.295503 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ea831844-gf8cw"] Oct 31 00:03:41.297982 ip-10-0-141-180 kubenswrapper[4927]: W1031 00:03:41.297900 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55ce234b_b2e4_48ca_ae22_57598d4bd8db.slice/crio-54bd331b7a80db4f10438991a65de7fa4c910452e982a77de8ec264d3049b4b1 WatchSource:0}: Error finding container 54bd331b7a80db4f10438991a65de7fa4c910452e982a77de8ec264d3049b4b1: Status 404 returned error can't find the container with id 54bd331b7a80db4f10438991a65de7fa4c910452e982a77de8ec264d3049b4b1 Oct 31 00:03:42.076184 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:42.076145 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea831844-gf8cw" event={"ID":"55ce234b-b2e4-48ca-ae22-57598d4bd8db","Type":"ContainerStarted","Data":"114568ad1163e197c82f77d023d51d1b633d9e71899ffd33099b988dc40bc7b1"} Oct 31 00:03:42.076634 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:42.076208 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea831844-gf8cw" event={"ID":"55ce234b-b2e4-48ca-ae22-57598d4bd8db","Type":"ContainerStarted","Data":"54bd331b7a80db4f10438991a65de7fa4c910452e982a77de8ec264d3049b4b1"} Oct 31 00:03:42.196815 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:42.196766 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea831844-gf8cw" podStartSLOduration=2.1967523350000002 podStartE2EDuration="2.196752335s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:03:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:03:42.193792228 +0000 UTC m=+1335.230123051" watchObservedRunningTime="2026-10-31 00:03:42.196752335 +0000 UTC m=+1335.233083156" Oct 31 00:03:43.087850 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:43.087811 4927 generic.go:356] "Generic (PLEG): container finished" podID="55ce234b-b2e4-48ca-ae22-57598d4bd8db" containerID="114568ad1163e197c82f77d023d51d1b633d9e71899ffd33099b988dc40bc7b1" exitCode=0 Oct 31 00:03:43.088632 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:43.088607 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea831844-gf8cw" event={"ID":"55ce234b-b2e4-48ca-ae22-57598d4bd8db","Type":"ContainerDied","Data":"114568ad1163e197c82f77d023d51d1b633d9e71899ffd33099b988dc40bc7b1"} Oct 31 00:03:44.285709 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:44.285686 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea831844-gf8cw" Oct 31 00:03:44.362717 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:44.362635 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/55ce234b-b2e4-48ca-ae22-57598d4bd8db-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55ce234b-b2e4-48ca-ae22-57598d4bd8db-maas-api-openshift-ca\") pod \"55ce234b-b2e4-48ca-ae22-57598d4bd8db\" (UID: \"55ce234b-b2e4-48ca-ae22-57598d4bd8db\") " Oct 31 00:03:44.362863 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:44.362851 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55ce234b-b2e4-48ca-ae22-57598d4bd8db-kube-api-access-n9zd8\" (UniqueName: \"kubernetes.io/projected/55ce234b-b2e4-48ca-ae22-57598d4bd8db-kube-api-access-n9zd8\") pod \"55ce234b-b2e4-48ca-ae22-57598d4bd8db\" (UID: \"55ce234b-b2e4-48ca-ae22-57598d4bd8db\") " Oct 31 00:03:44.363044 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:44.363028 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/55ce234b-b2e4-48ca-ae22-57598d4bd8db-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55ce234b-b2e4-48ca-ae22-57598d4bd8db-maas-api-xks-ca\") pod \"55ce234b-b2e4-48ca-ae22-57598d4bd8db\" (UID: \"55ce234b-b2e4-48ca-ae22-57598d4bd8db\") " Oct 31 00:03:44.363181 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:44.363025 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55ce234b-b2e4-48ca-ae22-57598d4bd8db-maas-api-openshift-ca" pod "55ce234b-b2e4-48ca-ae22-57598d4bd8db" (UID: "55ce234b-b2e4-48ca-ae22-57598d4bd8db"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:03:44.363613 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:44.363592 4927 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55ce234b-b2e4-48ca-ae22-57598d4bd8db-maas-api-openshift-ca\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:03:44.365018 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:44.364991 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55ce234b-b2e4-48ca-ae22-57598d4bd8db-kube-api-access-n9zd8" pod "55ce234b-b2e4-48ca-ae22-57598d4bd8db" (UID: "55ce234b-b2e4-48ca-ae22-57598d4bd8db"). InnerVolumeSpecName "kube-api-access-n9zd8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:03:44.365134 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:44.364992 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/55ce234b-b2e4-48ca-ae22-57598d4bd8db-maas-api-xks-ca" pod "55ce234b-b2e4-48ca-ae22-57598d4bd8db" (UID: "55ce234b-b2e4-48ca-ae22-57598d4bd8db"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:03:44.465079 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:44.465042 4927 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55ce234b-b2e4-48ca-ae22-57598d4bd8db-maas-api-xks-ca\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:03:44.465079 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:44.465073 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n9zd8\" (UniqueName: \"kubernetes.io/projected/55ce234b-b2e4-48ca-ae22-57598d4bd8db-kube-api-access-n9zd8\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:03:44.670705 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:44.670674 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:03:44.885841 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:44.885784 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:03:44.931554 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:44.931465 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-pc4b9"] Oct 31 00:03:44.932244 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:44.932228 4927 cpu_manager.go:414] "containerMap: removing container" podUID="55ce234b-b2e4-48ca-ae22-57598d4bd8db" containerName="revoke-keys" Oct 31 00:03:44.932307 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:44.932245 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55ce234b-b2e4-48ca-ae22-57598d4bd8db" containerName="revoke-keys" Oct 31 00:03:44.932395 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:44.932369 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="55ce234b-b2e4-48ca-ae22-57598d4bd8db" containerName="revoke-keys" Oct 31 00:03:44.936637 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:44.936619 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-pc4b9" Oct 31 00:03:44.950077 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:44.950054 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-hs4pf"] Oct 31 00:03:44.954716 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:44.954691 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-hs4pf" Oct 31 00:03:44.971521 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:44.971496 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83ebf014-df38-445b-b3df-d2b9e8af0595-istio-podinfo\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-pc4b9\" (UID: \"83ebf014-df38-445b-b3df-d2b9e8af0595\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-pc4b9" Oct 31 00:03:44.971694 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:44.971679 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83ebf014-df38-445b-b3df-d2b9e8af0595-istio-data\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-pc4b9\" (UID: \"83ebf014-df38-445b-b3df-d2b9e8af0595\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-pc4b9" Oct 31 00:03:44.971770 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:44.971725 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83ebf014-df38-445b-b3df-d2b9e8af0595-istiod-ca-cert\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-pc4b9\" (UID: \"83ebf014-df38-445b-b3df-d2b9e8af0595\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-pc4b9" Oct 31 00:03:44.971841 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:44.971825 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83ebf014-df38-445b-b3df-d2b9e8af0595-workload-certs\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-pc4b9\" (UID: \"83ebf014-df38-445b-b3df-d2b9e8af0595\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-pc4b9" Oct 31 00:03:44.971902 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:44.971857 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83ebf014-df38-445b-b3df-d2b9e8af0595-istio-token\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-pc4b9\" (UID: \"83ebf014-df38-445b-b3df-d2b9e8af0595\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-pc4b9" Oct 31 00:03:44.971959 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:44.971939 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83ebf014-df38-445b-b3df-d2b9e8af0595-istio-envoy\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-pc4b9\" (UID: \"83ebf014-df38-445b-b3df-d2b9e8af0595\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-pc4b9" Oct 31 00:03:44.972043 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:44.972025 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83ebf014-df38-445b-b3df-d2b9e8af0595-workload-socket\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-pc4b9\" (UID: \"83ebf014-df38-445b-b3df-d2b9e8af0595\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-pc4b9" Oct 31 00:03:44.972137 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:44.972121 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7hmq\" (UniqueName: \"kubernetes.io/projected/83ebf014-df38-445b-b3df-d2b9e8af0595-kube-api-access-m7hmq\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-pc4b9\" (UID: \"83ebf014-df38-445b-b3df-d2b9e8af0595\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-pc4b9" Oct 31 00:03:44.972192 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:44.972162 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83ebf014-df38-445b-b3df-d2b9e8af0595-credential-socket\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-pc4b9\" (UID: \"83ebf014-df38-445b-b3df-d2b9e8af0595\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-pc4b9" Oct 31 00:03:45.057392 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:45.047388 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-pc4b9"] Oct 31 00:03:45.073176 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:45.073144 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jxbr7\" (UniqueName: \"kubernetes.io/projected/9625c102-c118-48b2-9c66-4373c5d839eb-kube-api-access-jxbr7\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-hs4pf\" (UID: \"9625c102-c118-48b2-9c66-4373c5d839eb\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-hs4pf" Oct 31 00:03:45.073373 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:45.073194 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83ebf014-df38-445b-b3df-d2b9e8af0595-istio-podinfo\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-pc4b9\" (UID: \"83ebf014-df38-445b-b3df-d2b9e8af0595\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-pc4b9" Oct 31 00:03:45.073451 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:45.073376 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9625c102-c118-48b2-9c66-4373c5d839eb-istio-token\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-hs4pf\" (UID: \"9625c102-c118-48b2-9c66-4373c5d839eb\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-hs4pf" Oct 31 00:03:45.073451 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:45.073420 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9625c102-c118-48b2-9c66-4373c5d839eb-istio-data\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-hs4pf\" (UID: \"9625c102-c118-48b2-9c66-4373c5d839eb\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-hs4pf" Oct 31 00:03:45.073571 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:45.073467 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9625c102-c118-48b2-9c66-4373c5d839eb-workload-socket\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-hs4pf\" (UID: \"9625c102-c118-48b2-9c66-4373c5d839eb\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-hs4pf" Oct 31 00:03:45.073571 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:45.073505 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83ebf014-df38-445b-b3df-d2b9e8af0595-istio-data\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-pc4b9\" (UID: \"83ebf014-df38-445b-b3df-d2b9e8af0595\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-pc4b9" Oct 31 00:03:45.073571 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:45.073551 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83ebf014-df38-445b-b3df-d2b9e8af0595-istiod-ca-cert\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-pc4b9\" (UID: \"83ebf014-df38-445b-b3df-d2b9e8af0595\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-pc4b9" Oct 31 00:03:45.073734 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:45.073575 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9625c102-c118-48b2-9c66-4373c5d839eb-istio-envoy\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-hs4pf\" (UID: \"9625c102-c118-48b2-9c66-4373c5d839eb\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-hs4pf" Oct 31 00:03:45.073734 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:45.073628 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9625c102-c118-48b2-9c66-4373c5d839eb-workload-certs\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-hs4pf\" (UID: \"9625c102-c118-48b2-9c66-4373c5d839eb\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-hs4pf" Oct 31 00:03:45.073734 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:45.073658 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83ebf014-df38-445b-b3df-d2b9e8af0595-workload-certs\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-pc4b9\" (UID: \"83ebf014-df38-445b-b3df-d2b9e8af0595\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-pc4b9" Oct 31 00:03:45.073734 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:45.073688 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83ebf014-df38-445b-b3df-d2b9e8af0595-istio-token\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-pc4b9\" (UID: \"83ebf014-df38-445b-b3df-d2b9e8af0595\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-pc4b9" Oct 31 00:03:45.074545 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:45.073735 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83ebf014-df38-445b-b3df-d2b9e8af0595-istio-envoy\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-pc4b9\" (UID: \"83ebf014-df38-445b-b3df-d2b9e8af0595\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-pc4b9" Oct 31 00:03:45.074545 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:45.073794 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83ebf014-df38-445b-b3df-d2b9e8af0595-workload-socket\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-pc4b9\" (UID: \"83ebf014-df38-445b-b3df-d2b9e8af0595\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-pc4b9" Oct 31 00:03:45.074545 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:45.073824 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9625c102-c118-48b2-9c66-4373c5d839eb-istiod-ca-cert\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-hs4pf\" (UID: \"9625c102-c118-48b2-9c66-4373c5d839eb\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-hs4pf" Oct 31 00:03:45.074545 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:45.073889 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9625c102-c118-48b2-9c66-4373c5d839eb-credential-socket\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-hs4pf\" (UID: \"9625c102-c118-48b2-9c66-4373c5d839eb\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-hs4pf" Oct 31 00:03:45.074545 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:45.073951 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9625c102-c118-48b2-9c66-4373c5d839eb-istio-podinfo\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-hs4pf\" (UID: \"9625c102-c118-48b2-9c66-4373c5d839eb\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-hs4pf" Oct 31 00:03:45.074545 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:45.073979 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7hmq\" (UniqueName: \"kubernetes.io/projected/83ebf014-df38-445b-b3df-d2b9e8af0595-kube-api-access-m7hmq\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-pc4b9\" (UID: \"83ebf014-df38-445b-b3df-d2b9e8af0595\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-pc4b9" Oct 31 00:03:45.074545 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:45.074016 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83ebf014-df38-445b-b3df-d2b9e8af0595-credential-socket\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-pc4b9\" (UID: \"83ebf014-df38-445b-b3df-d2b9e8af0595\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-pc4b9" Oct 31 00:03:45.074930 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:45.074672 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83ebf014-df38-445b-b3df-d2b9e8af0595-credential-socket\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-pc4b9\" (UID: \"83ebf014-df38-445b-b3df-d2b9e8af0595\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-pc4b9" Oct 31 00:03:45.074930 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:45.074737 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83ebf014-df38-445b-b3df-d2b9e8af0595-workload-certs\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-pc4b9\" (UID: \"83ebf014-df38-445b-b3df-d2b9e8af0595\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-pc4b9" Oct 31 00:03:45.075658 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:45.075630 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83ebf014-df38-445b-b3df-d2b9e8af0595-istio-data\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-pc4b9\" (UID: \"83ebf014-df38-445b-b3df-d2b9e8af0595\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-pc4b9" Oct 31 00:03:45.075658 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:45.075649 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83ebf014-df38-445b-b3df-d2b9e8af0595-istiod-ca-cert\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-pc4b9\" (UID: \"83ebf014-df38-445b-b3df-d2b9e8af0595\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-pc4b9" Oct 31 00:03:45.075876 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:45.075733 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83ebf014-df38-445b-b3df-d2b9e8af0595-workload-socket\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-pc4b9\" (UID: \"83ebf014-df38-445b-b3df-d2b9e8af0595\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-pc4b9" Oct 31 00:03:45.084280 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:45.078583 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83ebf014-df38-445b-b3df-d2b9e8af0595-istio-envoy\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-pc4b9\" (UID: \"83ebf014-df38-445b-b3df-d2b9e8af0595\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-pc4b9" Oct 31 00:03:45.084280 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:45.079097 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83ebf014-df38-445b-b3df-d2b9e8af0595-istio-podinfo\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-pc4b9\" (UID: \"83ebf014-df38-445b-b3df-d2b9e8af0595\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-pc4b9" Oct 31 00:03:45.099692 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:45.099661 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-hs4pf"] Oct 31 00:03:45.113901 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:45.113879 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea831844-gf8cw" Oct 31 00:03:45.114018 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:45.113897 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea831844-gf8cw" event={"ID":"55ce234b-b2e4-48ca-ae22-57598d4bd8db","Type":"ContainerDied","Data":"54bd331b7a80db4f10438991a65de7fa4c910452e982a77de8ec264d3049b4b1"} Oct 31 00:03:45.114018 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:45.113951 4927 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="54bd331b7a80db4f10438991a65de7fa4c910452e982a77de8ec264d3049b4b1" Oct 31 00:03:45.170186 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:45.170151 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7hmq\" (UniqueName: \"kubernetes.io/projected/83ebf014-df38-445b-b3df-d2b9e8af0595-kube-api-access-m7hmq\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-pc4b9\" (UID: \"83ebf014-df38-445b-b3df-d2b9e8af0595\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-pc4b9" Oct 31 00:03:45.170557 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:45.170534 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83ebf014-df38-445b-b3df-d2b9e8af0595-istio-token\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-pc4b9\" (UID: \"83ebf014-df38-445b-b3df-d2b9e8af0595\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-pc4b9" Oct 31 00:03:45.174978 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:45.174953 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9625c102-c118-48b2-9c66-4373c5d839eb-istio-data\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-hs4pf\" (UID: \"9625c102-c118-48b2-9c66-4373c5d839eb\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-hs4pf" Oct 31 00:03:45.175091 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:45.174995 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9625c102-c118-48b2-9c66-4373c5d839eb-workload-socket\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-hs4pf\" (UID: \"9625c102-c118-48b2-9c66-4373c5d839eb\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-hs4pf" Oct 31 00:03:45.175091 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:45.175045 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9625c102-c118-48b2-9c66-4373c5d839eb-istio-envoy\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-hs4pf\" (UID: \"9625c102-c118-48b2-9c66-4373c5d839eb\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-hs4pf" Oct 31 00:03:45.175254 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:45.175096 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9625c102-c118-48b2-9c66-4373c5d839eb-workload-certs\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-hs4pf\" (UID: \"9625c102-c118-48b2-9c66-4373c5d839eb\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-hs4pf" Oct 31 00:03:45.175254 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:45.175185 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9625c102-c118-48b2-9c66-4373c5d839eb-istiod-ca-cert\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-hs4pf\" (UID: \"9625c102-c118-48b2-9c66-4373c5d839eb\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-hs4pf" Oct 31 00:03:45.175254 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:45.175236 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9625c102-c118-48b2-9c66-4373c5d839eb-credential-socket\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-hs4pf\" (UID: \"9625c102-c118-48b2-9c66-4373c5d839eb\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-hs4pf" Oct 31 00:03:45.175512 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:45.175291 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9625c102-c118-48b2-9c66-4373c5d839eb-istio-podinfo\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-hs4pf\" (UID: \"9625c102-c118-48b2-9c66-4373c5d839eb\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-hs4pf" Oct 31 00:03:45.175512 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:45.175433 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxbr7\" (UniqueName: \"kubernetes.io/projected/9625c102-c118-48b2-9c66-4373c5d839eb-kube-api-access-jxbr7\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-hs4pf\" (UID: \"9625c102-c118-48b2-9c66-4373c5d839eb\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-hs4pf" Oct 31 00:03:45.175622 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:45.175548 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9625c102-c118-48b2-9c66-4373c5d839eb-workload-certs\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-hs4pf\" (UID: \"9625c102-c118-48b2-9c66-4373c5d839eb\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-hs4pf" Oct 31 00:03:45.175687 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:45.175625 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9625c102-c118-48b2-9c66-4373c5d839eb-istio-token\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-hs4pf\" (UID: \"9625c102-c118-48b2-9c66-4373c5d839eb\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-hs4pf" Oct 31 00:03:45.175687 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:45.175677 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9625c102-c118-48b2-9c66-4373c5d839eb-workload-socket\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-hs4pf\" (UID: \"9625c102-c118-48b2-9c66-4373c5d839eb\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-hs4pf" Oct 31 00:03:45.176411 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:45.176385 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9625c102-c118-48b2-9c66-4373c5d839eb-credential-socket\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-hs4pf\" (UID: \"9625c102-c118-48b2-9c66-4373c5d839eb\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-hs4pf" Oct 31 00:03:45.176965 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:45.176939 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9625c102-c118-48b2-9c66-4373c5d839eb-istiod-ca-cert\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-hs4pf\" (UID: \"9625c102-c118-48b2-9c66-4373c5d839eb\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-hs4pf" Oct 31 00:03:45.177077 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:45.177059 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9625c102-c118-48b2-9c66-4373c5d839eb-istio-data\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-hs4pf\" (UID: \"9625c102-c118-48b2-9c66-4373c5d839eb\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-hs4pf" Oct 31 00:03:45.178148 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:45.178131 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9625c102-c118-48b2-9c66-4373c5d839eb-istio-envoy\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-hs4pf\" (UID: \"9625c102-c118-48b2-9c66-4373c5d839eb\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-hs4pf" Oct 31 00:03:45.178283 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:45.178264 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9625c102-c118-48b2-9c66-4373c5d839eb-istio-podinfo\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-hs4pf\" (UID: \"9625c102-c118-48b2-9c66-4373c5d839eb\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-hs4pf" Oct 31 00:03:45.209848 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:45.209769 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-mlkdz"] Oct 31 00:03:45.215891 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:45.215871 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-mlkdz" Oct 31 00:03:45.249593 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:45.249564 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9625c102-c118-48b2-9c66-4373c5d839eb-istio-token\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-hs4pf\" (UID: \"9625c102-c118-48b2-9c66-4373c5d839eb\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-hs4pf" Oct 31 00:03:45.249762 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:45.249648 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxbr7\" (UniqueName: \"kubernetes.io/projected/9625c102-c118-48b2-9c66-4373c5d839eb-kube-api-access-jxbr7\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-hs4pf\" (UID: \"9625c102-c118-48b2-9c66-4373c5d839eb\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-hs4pf" Oct 31 00:03:45.256891 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:45.256863 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-pc4b9" Oct 31 00:03:45.266604 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:45.266567 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-t4ms2"] Oct 31 00:03:45.270204 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:45.270179 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-hs4pf" Oct 31 00:03:45.273639 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:45.273620 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-t4ms2" Oct 31 00:03:45.276882 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:45.276835 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9bfa829-54c3-4c65-8596-dca8560ec433-istio-podinfo\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-mlkdz\" (UID: \"d9bfa829-54c3-4c65-8596-dca8560ec433\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-mlkdz" Oct 31 00:03:45.276989 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:45.276974 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9bfa829-54c3-4c65-8596-dca8560ec433-credential-socket\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-mlkdz\" (UID: \"d9bfa829-54c3-4c65-8596-dca8560ec433\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-mlkdz" Oct 31 00:03:45.277102 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:45.277047 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9bfa829-54c3-4c65-8596-dca8560ec433-workload-certs\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-mlkdz\" (UID: \"d9bfa829-54c3-4c65-8596-dca8560ec433\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-mlkdz" Oct 31 00:03:45.277257 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:45.277105 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9zzv\" (UniqueName: \"kubernetes.io/projected/d9bfa829-54c3-4c65-8596-dca8560ec433-kube-api-access-v9zzv\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-mlkdz\" (UID: \"d9bfa829-54c3-4c65-8596-dca8560ec433\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-mlkdz" Oct 31 00:03:45.277257 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:45.277187 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9bfa829-54c3-4c65-8596-dca8560ec433-istio-envoy\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-mlkdz\" (UID: \"d9bfa829-54c3-4c65-8596-dca8560ec433\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-mlkdz" Oct 31 00:03:45.277403 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:45.277258 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9bfa829-54c3-4c65-8596-dca8560ec433-workload-socket\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-mlkdz\" (UID: \"d9bfa829-54c3-4c65-8596-dca8560ec433\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-mlkdz" Oct 31 00:03:45.277403 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:45.277283 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9bfa829-54c3-4c65-8596-dca8560ec433-istio-token\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-mlkdz\" (UID: \"d9bfa829-54c3-4c65-8596-dca8560ec433\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-mlkdz" Oct 31 00:03:45.277403 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:45.277312 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9bfa829-54c3-4c65-8596-dca8560ec433-istio-data\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-mlkdz\" (UID: \"d9bfa829-54c3-4c65-8596-dca8560ec433\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-mlkdz" Oct 31 00:03:45.277403 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:45.277357 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9bfa829-54c3-4c65-8596-dca8560ec433-istiod-ca-cert\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-mlkdz\" (UID: \"d9bfa829-54c3-4c65-8596-dca8560ec433\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-mlkdz" Oct 31 00:03:45.279294 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:45.279270 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-mlkdz"] Oct 31 00:03:45.375227 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:45.375193 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-t4ms2"] Oct 31 00:03:45.382929 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:45.382898 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1822bec6-cd20-4045-b362-d83766f3337b-workload-socket\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-t4ms2\" (UID: \"1822bec6-cd20-4045-b362-d83766f3337b\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-t4ms2" Oct 31 00:03:45.383064 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:45.382954 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1822bec6-cd20-4045-b362-d83766f3337b-workload-certs\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-t4ms2\" (UID: \"1822bec6-cd20-4045-b362-d83766f3337b\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-t4ms2" Oct 31 00:03:45.383064 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:45.383009 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9bfa829-54c3-4c65-8596-dca8560ec433-credential-socket\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-mlkdz\" (UID: \"d9bfa829-54c3-4c65-8596-dca8560ec433\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-mlkdz" Oct 31 00:03:45.383064 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:45.383036 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1822bec6-cd20-4045-b362-d83766f3337b-istiod-ca-cert\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-t4ms2\" (UID: \"1822bec6-cd20-4045-b362-d83766f3337b\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-t4ms2" Oct 31 00:03:45.383236 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:45.383065 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1822bec6-cd20-4045-b362-d83766f3337b-istio-envoy\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-t4ms2\" (UID: \"1822bec6-cd20-4045-b362-d83766f3337b\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-t4ms2" Oct 31 00:03:45.383236 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:45.383209 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9bfa829-54c3-4c65-8596-dca8560ec433-workload-certs\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-mlkdz\" (UID: \"d9bfa829-54c3-4c65-8596-dca8560ec433\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-mlkdz" Oct 31 00:03:45.383368 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:45.383257 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9zzv\" (UniqueName: \"kubernetes.io/projected/d9bfa829-54c3-4c65-8596-dca8560ec433-kube-api-access-v9zzv\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-mlkdz\" (UID: \"d9bfa829-54c3-4c65-8596-dca8560ec433\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-mlkdz" Oct 31 00:03:45.383434 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:45.383321 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1822bec6-cd20-4045-b362-d83766f3337b-credential-socket\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-t4ms2\" (UID: \"1822bec6-cd20-4045-b362-d83766f3337b\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-t4ms2" Oct 31 00:03:45.383492 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:45.383455 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9bfa829-54c3-4c65-8596-dca8560ec433-istio-envoy\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-mlkdz\" (UID: \"d9bfa829-54c3-4c65-8596-dca8560ec433\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-mlkdz" Oct 31 00:03:45.383547 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:45.383533 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9bfa829-54c3-4c65-8596-dca8560ec433-workload-socket\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-mlkdz\" (UID: \"d9bfa829-54c3-4c65-8596-dca8560ec433\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-mlkdz" Oct 31 00:03:45.383599 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:45.383560 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9bfa829-54c3-4c65-8596-dca8560ec433-istio-token\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-mlkdz\" (UID: \"d9bfa829-54c3-4c65-8596-dca8560ec433\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-mlkdz" Oct 31 00:03:45.383660 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:45.383594 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jvrk\" (UniqueName: \"kubernetes.io/projected/1822bec6-cd20-4045-b362-d83766f3337b-kube-api-access-9jvrk\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-t4ms2\" (UID: \"1822bec6-cd20-4045-b362-d83766f3337b\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-t4ms2" Oct 31 00:03:45.383660 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:45.383622 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9bfa829-54c3-4c65-8596-dca8560ec433-istio-data\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-mlkdz\" (UID: \"d9bfa829-54c3-4c65-8596-dca8560ec433\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-mlkdz" Oct 31 00:03:45.383660 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:45.383649 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1822bec6-cd20-4045-b362-d83766f3337b-istio-data\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-t4ms2\" (UID: \"1822bec6-cd20-4045-b362-d83766f3337b\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-t4ms2" Oct 31 00:03:45.383813 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:45.383677 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9bfa829-54c3-4c65-8596-dca8560ec433-istiod-ca-cert\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-mlkdz\" (UID: \"d9bfa829-54c3-4c65-8596-dca8560ec433\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-mlkdz" Oct 31 00:03:45.383813 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:45.383709 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1822bec6-cd20-4045-b362-d83766f3337b-istio-token\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-t4ms2\" (UID: \"1822bec6-cd20-4045-b362-d83766f3337b\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-t4ms2" Oct 31 00:03:45.383916 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:45.383836 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1822bec6-cd20-4045-b362-d83766f3337b-istio-podinfo\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-t4ms2\" (UID: \"1822bec6-cd20-4045-b362-d83766f3337b\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-t4ms2" Oct 31 00:03:45.383916 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:45.383869 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9bfa829-54c3-4c65-8596-dca8560ec433-istio-podinfo\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-mlkdz\" (UID: \"d9bfa829-54c3-4c65-8596-dca8560ec433\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-mlkdz" Oct 31 00:03:45.386322 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:45.386303 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9bfa829-54c3-4c65-8596-dca8560ec433-credential-socket\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-mlkdz\" (UID: \"d9bfa829-54c3-4c65-8596-dca8560ec433\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-mlkdz" Oct 31 00:03:45.386679 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:45.386665 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9bfa829-54c3-4c65-8596-dca8560ec433-workload-certs\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-mlkdz\" (UID: \"d9bfa829-54c3-4c65-8596-dca8560ec433\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-mlkdz" Oct 31 00:03:45.390861 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:45.390493 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9bfa829-54c3-4c65-8596-dca8560ec433-workload-socket\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-mlkdz\" (UID: \"d9bfa829-54c3-4c65-8596-dca8560ec433\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-mlkdz" Oct 31 00:03:45.391175 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:45.391092 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9bfa829-54c3-4c65-8596-dca8560ec433-istio-envoy\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-mlkdz\" (UID: \"d9bfa829-54c3-4c65-8596-dca8560ec433\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-mlkdz" Oct 31 00:03:45.391676 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:45.391654 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9bfa829-54c3-4c65-8596-dca8560ec433-istio-podinfo\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-mlkdz\" (UID: \"d9bfa829-54c3-4c65-8596-dca8560ec433\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-mlkdz" Oct 31 00:03:45.391808 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:45.391693 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9bfa829-54c3-4c65-8596-dca8560ec433-istio-data\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-mlkdz\" (UID: \"d9bfa829-54c3-4c65-8596-dca8560ec433\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-mlkdz" Oct 31 00:03:45.391965 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:45.391942 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9bfa829-54c3-4c65-8596-dca8560ec433-istiod-ca-cert\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-mlkdz\" (UID: \"d9bfa829-54c3-4c65-8596-dca8560ec433\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-mlkdz" Oct 31 00:03:45.449751 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:45.449699 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9zzv\" (UniqueName: \"kubernetes.io/projected/d9bfa829-54c3-4c65-8596-dca8560ec433-kube-api-access-v9zzv\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-mlkdz\" (UID: \"d9bfa829-54c3-4c65-8596-dca8560ec433\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-mlkdz" Oct 31 00:03:45.458037 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:45.458011 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9bfa829-54c3-4c65-8596-dca8560ec433-istio-token\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-mlkdz\" (UID: \"d9bfa829-54c3-4c65-8596-dca8560ec433\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-mlkdz" Oct 31 00:03:45.476139 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:45.475982 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:03:45.485215 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:45.485178 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1822bec6-cd20-4045-b362-d83766f3337b-istio-podinfo\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-t4ms2\" (UID: \"1822bec6-cd20-4045-b362-d83766f3337b\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-t4ms2" Oct 31 00:03:45.485396 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:45.485293 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1822bec6-cd20-4045-b362-d83766f3337b-workload-socket\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-t4ms2\" (UID: \"1822bec6-cd20-4045-b362-d83766f3337b\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-t4ms2" Oct 31 00:03:45.485396 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:45.485353 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1822bec6-cd20-4045-b362-d83766f3337b-workload-certs\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-t4ms2\" (UID: \"1822bec6-cd20-4045-b362-d83766f3337b\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-t4ms2" Oct 31 00:03:45.485520 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:45.485409 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1822bec6-cd20-4045-b362-d83766f3337b-istiod-ca-cert\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-t4ms2\" (UID: \"1822bec6-cd20-4045-b362-d83766f3337b\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-t4ms2" Oct 31 00:03:45.485520 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:45.485441 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1822bec6-cd20-4045-b362-d83766f3337b-istio-envoy\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-t4ms2\" (UID: \"1822bec6-cd20-4045-b362-d83766f3337b\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-t4ms2" Oct 31 00:03:45.485636 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:45.485562 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1822bec6-cd20-4045-b362-d83766f3337b-credential-socket\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-t4ms2\" (UID: \"1822bec6-cd20-4045-b362-d83766f3337b\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-t4ms2" Oct 31 00:03:45.485732 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:45.485710 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jvrk\" (UniqueName: \"kubernetes.io/projected/1822bec6-cd20-4045-b362-d83766f3337b-kube-api-access-9jvrk\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-t4ms2\" (UID: \"1822bec6-cd20-4045-b362-d83766f3337b\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-t4ms2" Oct 31 00:03:45.485785 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:45.485749 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1822bec6-cd20-4045-b362-d83766f3337b-istio-data\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-t4ms2\" (UID: \"1822bec6-cd20-4045-b362-d83766f3337b\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-t4ms2" Oct 31 00:03:45.485870 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:45.485785 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1822bec6-cd20-4045-b362-d83766f3337b-istio-token\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-t4ms2\" (UID: \"1822bec6-cd20-4045-b362-d83766f3337b\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-t4ms2" Oct 31 00:03:45.495033 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:45.495008 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1822bec6-cd20-4045-b362-d83766f3337b-credential-socket\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-t4ms2\" (UID: \"1822bec6-cd20-4045-b362-d83766f3337b\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-t4ms2" Oct 31 00:03:45.495536 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:45.495471 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1822bec6-cd20-4045-b362-d83766f3337b-workload-certs\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-t4ms2\" (UID: \"1822bec6-cd20-4045-b362-d83766f3337b\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-t4ms2" Oct 31 00:03:45.496105 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:45.495884 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1822bec6-cd20-4045-b362-d83766f3337b-istiod-ca-cert\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-t4ms2\" (UID: \"1822bec6-cd20-4045-b362-d83766f3337b\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-t4ms2" Oct 31 00:03:45.496200 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:45.496125 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1822bec6-cd20-4045-b362-d83766f3337b-workload-socket\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-t4ms2\" (UID: \"1822bec6-cd20-4045-b362-d83766f3337b\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-t4ms2" Oct 31 00:03:45.496367 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:45.496343 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1822bec6-cd20-4045-b362-d83766f3337b-istio-data\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-t4ms2\" (UID: \"1822bec6-cd20-4045-b362-d83766f3337b\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-t4ms2" Oct 31 00:03:45.496822 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:45.496800 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1822bec6-cd20-4045-b362-d83766f3337b-istio-envoy\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-t4ms2\" (UID: \"1822bec6-cd20-4045-b362-d83766f3337b\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-t4ms2" Oct 31 00:03:45.500020 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:45.499995 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1822bec6-cd20-4045-b362-d83766f3337b-istio-podinfo\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-t4ms2\" (UID: \"1822bec6-cd20-4045-b362-d83766f3337b\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-t4ms2" Oct 31 00:03:45.534926 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:45.534644 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-mlkdz" Oct 31 00:03:45.567267 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:45.567196 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1822bec6-cd20-4045-b362-d83766f3337b-istio-token\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-t4ms2\" (UID: \"1822bec6-cd20-4045-b362-d83766f3337b\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-t4ms2" Oct 31 00:03:45.569913 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:45.569856 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jvrk\" (UniqueName: \"kubernetes.io/projected/1822bec6-cd20-4045-b362-d83766f3337b-kube-api-access-9jvrk\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-t4ms2\" (UID: \"1822bec6-cd20-4045-b362-d83766f3337b\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-t4ms2" Oct 31 00:03:45.683392 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:45.683055 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-t4ms2" Oct 31 00:03:45.743403 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:45.742579 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-8e7d0098-6794f6ff67-429f5" Oct 31 00:03:45.743960 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:45.743934 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-8e7d0098-8f775757b-4mt6m" Oct 31 00:03:45.759059 ip-10-0-141-180 kubenswrapper[4927]: W1031 00:03:45.759002 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9625c102_c118_48b2_9c66_4373c5d839eb.slice/crio-9bcfde161ed4022e22e41159824e223c682c2b5568d72534f4598390e7e4e3d5 WatchSource:0}: Error finding container 9bcfde161ed4022e22e41159824e223c682c2b5568d72534f4598390e7e4e3d5: Status 404 returned error can't find the container with id 9bcfde161ed4022e22e41159824e223c682c2b5568d72534f4598390e7e4e3d5 Oct 31 00:03:45.759675 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:45.759646 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-hs4pf"] Oct 31 00:03:45.761769 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:45.761734 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:03:45.761870 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:45.761797 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:03:45.761870 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:45.761827 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:03:45.962143 ip-10-0-141-180 kubenswrapper[4927]: W1031 00:03:45.962094 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83ebf014_df38_445b_b3df_d2b9e8af0595.slice/crio-ce9bb47db9f6682eac44426d6723b40aa4cd90737cc34a6d890129dd822d312f WatchSource:0}: Error finding container ce9bb47db9f6682eac44426d6723b40aa4cd90737cc34a6d890129dd822d312f: Status 404 returned error can't find the container with id ce9bb47db9f6682eac44426d6723b40aa4cd90737cc34a6d890129dd822d312f Oct 31 00:03:45.963841 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:45.963817 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-pc4b9"] Oct 31 00:03:45.965201 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:45.965168 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:03:45.965276 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:45.965234 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:03:45.965276 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:45.965262 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:03:46.048095 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:46.047905 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-mlkdz"] Oct 31 00:03:46.049574 ip-10-0-141-180 kubenswrapper[4927]: W1031 00:03:46.049546 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9bfa829_54c3_4c65_8596_dca8560ec433.slice/crio-571f2163fa283dea21fb7f9d98fa137e1b13780cfcc4bc85e14867a100e047c4 WatchSource:0}: Error finding container 571f2163fa283dea21fb7f9d98fa137e1b13780cfcc4bc85e14867a100e047c4: Status 404 returned error can't find the container with id 571f2163fa283dea21fb7f9d98fa137e1b13780cfcc4bc85e14867a100e047c4 Oct 31 00:03:46.053093 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:46.053047 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:03:46.053236 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:46.053137 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:03:46.053236 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:46.053178 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:03:46.130674 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:46.130640 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-t4ms2"] Oct 31 00:03:46.138116 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:46.137536 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:03:46.138116 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:46.137603 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:03:46.138116 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:46.137635 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:03:46.143499 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:46.142595 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-pc4b9" event={"ID":"83ebf014-df38-445b-b3df-d2b9e8af0595","Type":"ContainerStarted","Data":"e97887a9d8849a5e30b43a09ac63fe852803aa921fd555e35946136986381b7f"} Oct 31 00:03:46.143499 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:46.142645 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-pc4b9" event={"ID":"83ebf014-df38-445b-b3df-d2b9e8af0595","Type":"ContainerStarted","Data":"ce9bb47db9f6682eac44426d6723b40aa4cd90737cc34a6d890129dd822d312f"} Oct 31 00:03:46.146601 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:46.146516 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-mlkdz" event={"ID":"d9bfa829-54c3-4c65-8596-dca8560ec433","Type":"ContainerStarted","Data":"571f2163fa283dea21fb7f9d98fa137e1b13780cfcc4bc85e14867a100e047c4"} Oct 31 00:03:46.146601 ip-10-0-141-180 kubenswrapper[4927]: W1031 00:03:46.146577 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1822bec6_cd20_4045_b362_d83766f3337b.slice/crio-56b5348455bc2c26be8b05bf7068d2b8cb5276653ea8f9a7b2cb8abd0de7ee6d WatchSource:0}: Error finding container 56b5348455bc2c26be8b05bf7068d2b8cb5276653ea8f9a7b2cb8abd0de7ee6d: Status 404 returned error can't find the container with id 56b5348455bc2c26be8b05bf7068d2b8cb5276653ea8f9a7b2cb8abd0de7ee6d Oct 31 00:03:46.149378 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:46.148738 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-hs4pf" event={"ID":"9625c102-c118-48b2-9c66-4373c5d839eb","Type":"ContainerStarted","Data":"7ab257803381068ac6284e73ea5650b7abea666ef936dfb3dbff7ece5af6a0c8"} Oct 31 00:03:46.149378 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:46.148774 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-hs4pf" event={"ID":"9625c102-c118-48b2-9c66-4373c5d839eb","Type":"ContainerStarted","Data":"9bcfde161ed4022e22e41159824e223c682c2b5568d72534f4598390e7e4e3d5"} Oct 31 00:03:46.258671 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:46.257505 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-pc4b9" Oct 31 00:03:46.271900 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:46.271828 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-hs4pf" Oct 31 00:03:46.272208 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:46.272112 4927 patch_prober.go:29] interesting pod/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-hs4pf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body= Oct 31 00:03:46.272208 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:46.272154 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-hs4pf" podUID="9625c102-c118-48b2-9c66-4373c5d839eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" Oct 31 00:03:46.279414 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:46.278432 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-pc4b9" podStartSLOduration=2.278419179 podStartE2EDuration="2.278419179s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:03:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:03:46.277141807 +0000 UTC m=+1339.313472646" watchObservedRunningTime="2026-10-31 00:03:46.278419179 +0000 UTC m=+1339.314750001" Oct 31 00:03:46.430352 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:46.428889 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-hs4pf" podStartSLOduration=2.4288698 podStartE2EDuration="2.4288698s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:03:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:03:46.416849611 +0000 UTC m=+1339.453180450" watchObservedRunningTime="2026-10-31 00:03:46.4288698 +0000 UTC m=+1339.465200622" Oct 31 00:03:47.285357 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:47.281911 4927 patch_prober.go:29] interesting pod/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-hs4pf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body= Oct 31 00:03:47.299355 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:47.291414 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-hs4pf" podUID="9625c102-c118-48b2-9c66-4373c5d839eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" Oct 31 00:03:47.321929 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:47.311765 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-mlkdz" event={"ID":"d9bfa829-54c3-4c65-8596-dca8560ec433","Type":"ContainerStarted","Data":"480c4ab2112a37c5eed8ef1926b62400aa5aa06dee35334733aa6ff8b8c23f55"} Oct 31 00:03:47.321929 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:47.314459 4927 patch_prober.go:29] interesting pod/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-pc4b9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" start-of-body= Oct 31 00:03:47.321929 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:47.314510 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-pc4b9" podUID="83ebf014-df38-445b-b3df-d2b9e8af0595" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" Oct 31 00:03:47.334535 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:47.334499 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-t4ms2" event={"ID":"1822bec6-cd20-4045-b362-d83766f3337b","Type":"ContainerStarted","Data":"52c84a877ba688ff497a00fa0c370da126a9e699c2f21bd029c9a50e48ec50cf"} Oct 31 00:03:47.334790 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:47.334775 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-t4ms2" event={"ID":"1822bec6-cd20-4045-b362-d83766f3337b","Type":"ContainerStarted","Data":"56b5348455bc2c26be8b05bf7068d2b8cb5276653ea8f9a7b2cb8abd0de7ee6d"} Oct 31 00:03:47.477468 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:47.477407 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-mlkdz" podStartSLOduration=2.477384188 podStartE2EDuration="2.477384188s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:03:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:03:47.454580177 +0000 UTC m=+1340.490911011" watchObservedRunningTime="2026-10-31 00:03:47.477384188 +0000 UTC m=+1340.513715016" Oct 31 00:03:47.538018 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:47.537884 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-mlkdz" Oct 31 00:03:47.544193 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:47.543861 4927 patch_prober.go:29] interesting pod/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-mlkdz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" start-of-body= Oct 31 00:03:47.544193 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:47.543922 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-mlkdz" podUID="d9bfa829-54c3-4c65-8596-dca8560ec433" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" Oct 31 00:03:47.626281 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:47.622027 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-t4ms2" podStartSLOduration=2.622001403 podStartE2EDuration="2.622001403s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:03:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:03:47.593007526 +0000 UTC m=+1340.629338369" watchObservedRunningTime="2026-10-31 00:03:47.622001403 +0000 UTC m=+1340.658332228" Oct 31 00:03:47.687356 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:47.686856 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-t4ms2" Oct 31 00:03:47.687356 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:47.687020 4927 patch_prober.go:29] interesting pod/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-t4ms2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" start-of-body= Oct 31 00:03:47.687356 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:47.687065 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-t4ms2" podUID="1822bec6-cd20-4045-b362-d83766f3337b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" Oct 31 00:03:48.294235 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:48.294187 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:03:48.602086 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:48.601989 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fe01fa/distinct-2-w5-fe01fa-kserve-79b99966d8-dkhzd" podUID="d5c1c9e8-ce47-4868-860d-b8413ff74ab5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.111:8000/ready\": context deadline exceeded" Oct 31 00:03:48.734367 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:48.702556 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5c3374/premium-w1-5c3374-kserve-6c56d4dcd-llwcl" podUID="dc813f02-e077-4251-8c1e-fea0986b0f36" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded" Oct 31 00:03:48.834356 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:48.759013 4927 patch_prober.go:29] interesting pod/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-pc4b9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" start-of-body= Oct 31 00:03:48.834356 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:48.823295 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-pc4b9" podUID="83ebf014-df38-445b-b3df-d2b9e8af0595" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" Oct 31 00:03:48.834356 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:48.759294 4927 patch_prober.go:29] interesting pod/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-mlkdz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" start-of-body= Oct 31 00:03:48.834356 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:48.823407 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-mlkdz" podUID="d9bfa829-54c3-4c65-8596-dca8560ec433" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" Oct 31 00:03:49.040358 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:49.019824 4927 patch_prober.go:29] interesting pod/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-t4ms2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" start-of-body= Oct 31 00:03:49.041476 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:49.041430 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-t4ms2" podUID="1822bec6-cd20-4045-b362-d83766f3337b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" Oct 31 00:03:49.042107 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:48.961210 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-46b586/simulator-w0-46b586-kserve-67588b8dbd-z8xmn" podUID="9a25e252-223d-41f2-abf7-632067c77e0b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:03:49.043579 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:49.043553 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-986d37/premium-w3-986d37-kserve-5c7c7757fb-r99j7" podUID="f4e38df7-355f-4b47-a0b5-7e0b907f3326" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:03:49.051646 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:49.051614 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-35a187/distinct-2-w6-35a187-kserve-7ff4b6dc55-944g6" podUID="429ee092-42d8-475e-b0c6-b5a9930766fc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.113:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 00:03:49.200529 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:49.200485 4927 patch_prober.go:29] interesting pod/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-hs4pf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body= Oct 31 00:03:49.225581 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:49.225511 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-hs4pf" podUID="9625c102-c118-48b2-9c66-4373c5d839eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" Oct 31 00:03:49.280512 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:49.280476 4927 patch_prober.go:29] interesting pod/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-pc4b9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" start-of-body= Oct 31 00:03:49.280780 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:49.280752 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-pc4b9" podUID="83ebf014-df38-445b-b3df-d2b9e8af0595" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" Oct 31 00:03:49.308432 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:49.285964 4927 patch_prober.go:29] interesting pod/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-hs4pf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body= Oct 31 00:03:49.308432 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:49.286019 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-hs4pf" podUID="9625c102-c118-48b2-9c66-4373c5d839eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" Oct 31 00:03:49.603409 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:49.599858 4927 patch_prober.go:29] interesting pod/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-mlkdz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" start-of-body= Oct 31 00:03:49.603409 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:49.599930 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-mlkdz" podUID="d9bfa829-54c3-4c65-8596-dca8560ec433" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" Oct 31 00:03:49.664125 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:49.663426 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-27e82b/distinct-w2-27e82b-kserve-75f475fcd9-mccsz" podUID="4ba8ced4-e93f-46ac-87c5-3d0b7fadc02d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 00:03:49.693365 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:49.690454 4927 patch_prober.go:29] interesting pod/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-t4ms2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" start-of-body= Oct 31 00:03:49.693365 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:49.690533 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-t4ms2" podUID="1822bec6-cd20-4045-b362-d83766f3337b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" Oct 31 00:03:49.747082 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:49.747046 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-e0751b65-7d7cd79459-76xvm"] Oct 31 00:03:49.783370 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:49.776809 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-e0751b65-7d7cd79459-76xvm" Oct 31 00:03:49.828354 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:49.822069 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5c3374/premium-w1-5c3374-kserve-6c56d4dcd-llwcl" podUID="dc813f02-e077-4251-8c1e-fea0986b0f36" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded" Oct 31 00:03:49.849993 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:49.828802 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5c3374/premium-w1-5c3374-kserve-6c56d4dcd-llwcl" Oct 31 00:03:49.872721 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:49.872623 4927 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-5c3374/premium-w1-5c3374-kserve-6c56d4dcd-llwcl" podUID="dc813f02-e077-4251-8c1e-fea0986b0f36" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:03:49.889502 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:49.889472 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-e0751b65\"" Oct 31 00:03:49.895097 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:49.895056 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-e0751b65-dockercfg-rxqnn\"" Oct 31 00:03:49.896197 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:49.896164 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gszwr\" (UniqueName: \"kubernetes.io/projected/ecce302f-aa28-48f3-86b1-374399cd6540-kube-api-access-gszwr\") pod \"payload-pre-processing-e2e-mt-e0751b65-7d7cd79459-76xvm\" (UID: \"ecce302f-aa28-48f3-86b1-374399cd6540\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-e0751b65-7d7cd79459-76xvm" Oct 31 00:03:49.896354 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:49.896278 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ecce302f-aa28-48f3-86b1-374399cd6540-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-e0751b65-7d7cd79459-76xvm\" (UID: \"ecce302f-aa28-48f3-86b1-374399cd6540\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-e0751b65-7d7cd79459-76xvm" Oct 31 00:03:49.968359 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:49.967071 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-e0751b65-7d7cd79459-76xvm"] Oct 31 00:03:49.975375 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:49.973477 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fe01fa/distinct-2-w5-fe01fa-kserve-79b99966d8-dkhzd" podUID="d5c1c9e8-ce47-4868-860d-b8413ff74ab5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.111:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 00:03:49.975375 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:49.973882 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-fe01fa/distinct-2-w5-fe01fa-kserve-79b99966d8-dkhzd" Oct 31 00:03:49.997925 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:49.997887 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gszwr\" (UniqueName: \"kubernetes.io/projected/ecce302f-aa28-48f3-86b1-374399cd6540-kube-api-access-gszwr\") pod \"payload-pre-processing-e2e-mt-e0751b65-7d7cd79459-76xvm\" (UID: \"ecce302f-aa28-48f3-86b1-374399cd6540\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-e0751b65-7d7cd79459-76xvm" Oct 31 00:03:49.998511 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:49.998485 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ecce302f-aa28-48f3-86b1-374399cd6540-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-e0751b65-7d7cd79459-76xvm\" (UID: \"ecce302f-aa28-48f3-86b1-374399cd6540\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-e0751b65-7d7cd79459-76xvm" Oct 31 00:03:50.000764 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:50.000741 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ecce302f-aa28-48f3-86b1-374399cd6540-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-e0751b65-7d7cd79459-76xvm\" (UID: \"ecce302f-aa28-48f3-86b1-374399cd6540\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-e0751b65-7d7cd79459-76xvm" Oct 31 00:03:50.043754 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:50.043708 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-46b586/simulator-w0-46b586-kserve-67588b8dbd-z8xmn" podUID="9a25e252-223d-41f2-abf7-632067c77e0b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:03:50.044070 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:50.044048 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-986d37/premium-w3-986d37-kserve-5c7c7757fb-r99j7" podUID="f4e38df7-355f-4b47-a0b5-7e0b907f3326" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:03:50.044582 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:50.044562 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-46b586/simulator-w0-46b586-kserve-67588b8dbd-z8xmn" Oct 31 00:03:50.044750 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:50.044739 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-986d37/premium-w3-986d37-kserve-5c7c7757fb-r99j7" Oct 31 00:03:50.060351 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:50.055407 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-35a187/distinct-2-w6-35a187-kserve-7ff4b6dc55-944g6" podUID="429ee092-42d8-475e-b0c6-b5a9930766fc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.113:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:03:50.060351 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:50.055771 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-35a187/distinct-2-w6-35a187-kserve-7ff4b6dc55-944g6" Oct 31 00:03:50.123712 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:50.123633 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gszwr\" (UniqueName: \"kubernetes.io/projected/ecce302f-aa28-48f3-86b1-374399cd6540-kube-api-access-gszwr\") pod \"payload-pre-processing-e2e-mt-e0751b65-7d7cd79459-76xvm\" (UID: \"ecce302f-aa28-48f3-86b1-374399cd6540\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-e0751b65-7d7cd79459-76xvm" Oct 31 00:03:50.137365 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:50.128170 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-e0751b65-7d7cd79459-76xvm" Oct 31 00:03:50.305357 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:50.281795 4927 patch_prober.go:29] interesting pod/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-pc4b9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" start-of-body= Oct 31 00:03:50.305357 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:50.281868 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-pc4b9" podUID="83ebf014-df38-445b-b3df-d2b9e8af0595" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" Oct 31 00:03:50.393409 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:50.357013 4927 patch_prober.go:29] interesting pod/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-hs4pf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body= Oct 31 00:03:50.393409 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:50.357069 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-hs4pf" podUID="9625c102-c118-48b2-9c66-4373c5d839eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" Oct 31 00:03:50.536817 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:50.536780 4927 patch_prober.go:29] interesting pod/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-mlkdz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" start-of-body= Oct 31 00:03:50.558410 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:50.537093 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-mlkdz" podUID="d9bfa829-54c3-4c65-8596-dca8560ec433" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" Oct 31 00:03:50.666471 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:50.666428 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-27e82b/distinct-w2-27e82b-kserve-75f475fcd9-mccsz" podUID="4ba8ced4-e93f-46ac-87c5-3d0b7fadc02d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:03:50.667195 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:50.667169 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-27e82b/distinct-w2-27e82b-kserve-75f475fcd9-mccsz" Oct 31 00:03:50.714381 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:50.705446 4927 patch_prober.go:29] interesting pod/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-t4ms2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" start-of-body= Oct 31 00:03:50.714381 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:50.705509 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-t4ms2" podUID="1822bec6-cd20-4045-b362-d83766f3337b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" Oct 31 00:03:50.829691 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:50.829633 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5c3374/premium-w1-5c3374-kserve-6c56d4dcd-llwcl" podUID="dc813f02-e077-4251-8c1e-fea0986b0f36" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:03:50.944438 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:50.943852 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-fe01fa/distinct-2-w5-fe01fa-kserve-79b99966d8-dkhzd" Oct 31 00:03:51.048367 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:51.044175 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-46b586/simulator-w0-46b586-kserve-67588b8dbd-z8xmn" podUID="9a25e252-223d-41f2-abf7-632067c77e0b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:03:51.104442 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:51.048448 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-986d37/premium-w3-986d37-kserve-5c7c7757fb-r99j7" podUID="f4e38df7-355f-4b47-a0b5-7e0b907f3326" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:03:51.162608 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:51.162490 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-35a187/distinct-2-w6-35a187-kserve-7ff4b6dc55-944g6" podUID="429ee092-42d8-475e-b0c6-b5a9930766fc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.113:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 00:03:51.410771 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:51.377366 4927 patch_prober.go:29] interesting pod/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-pc4b9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" start-of-body= Oct 31 00:03:51.410771 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:51.377463 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-pc4b9" podUID="83ebf014-df38-445b-b3df-d2b9e8af0595" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" Oct 31 00:03:51.446357 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:51.436823 4927 patch_prober.go:29] interesting pod/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-hs4pf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body= Oct 31 00:03:51.446357 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:51.436884 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-hs4pf" podUID="9625c102-c118-48b2-9c66-4373c5d839eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" Oct 31 00:03:51.446357 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:51.443982 4927 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-27e82b/distinct-w2-27e82b-kserve-75f475fcd9-mccsz" podUID="4ba8ced4-e93f-46ac-87c5-3d0b7fadc02d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/health\": context deadline exceeded" Oct 31 00:03:51.446357 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:03:51.445730 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sqv69" containerName="extractor" Oct 31 00:03:51.661688 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:51.656781 4927 patch_prober.go:29] interesting pod/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-mlkdz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" start-of-body= Oct 31 00:03:51.671277 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:51.671233 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-mlkdz" podUID="d9bfa829-54c3-4c65-8596-dca8560ec433" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" Oct 31 00:03:51.671952 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:51.671914 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-27e82b/distinct-w2-27e82b-kserve-75f475fcd9-mccsz" podUID="4ba8ced4-e93f-46ac-87c5-3d0b7fadc02d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:03:51.697997 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:51.691105 4927 patch_prober.go:29] interesting pod/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-t4ms2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" start-of-body= Oct 31 00:03:51.697997 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:51.691198 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-t4ms2" podUID="1822bec6-cd20-4045-b362-d83766f3337b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" Oct 31 00:03:51.908473 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:51.904479 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5c3374/premium-w1-5c3374-kserve-6c56d4dcd-llwcl" podUID="dc813f02-e077-4251-8c1e-fea0986b0f36" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:03:52.148982 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:52.089641 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fe01fa/distinct-2-w5-fe01fa-kserve-79b99966d8-dkhzd" podUID="d5c1c9e8-ce47-4868-860d-b8413ff74ab5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.111:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:03:52.148982 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:52.114233 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-986d37/premium-w3-986d37-kserve-5c7c7757fb-r99j7" podUID="f4e38df7-355f-4b47-a0b5-7e0b907f3326" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:03:52.179513 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:52.179446 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-35a187/distinct-2-w6-35a187-kserve-7ff4b6dc55-944g6" podUID="429ee092-42d8-475e-b0c6-b5a9930766fc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.113:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:03:52.228373 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:03:52.210069 4927 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.118s" Oct 31 00:03:52.228373 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:52.210175 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-0d96587a-584d7b5dff-ccjj6" Oct 31 00:03:52.303301 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:52.294810 4927 patch_prober.go:29] interesting pod/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-pc4b9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" start-of-body= Oct 31 00:03:52.303301 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:52.294874 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-pc4b9" podUID="83ebf014-df38-445b-b3df-d2b9e8af0595" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" Oct 31 00:03:52.444906 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:52.444768 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-46b586/simulator-w0-46b586-kserve-67588b8dbd-z8xmn" podUID="9a25e252-223d-41f2-abf7-632067c77e0b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:03:52.671923 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:52.671887 4927 patch_prober.go:29] interesting pod/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-hs4pf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body= Oct 31 00:03:52.672563 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:52.672527 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-hs4pf" podUID="9625c102-c118-48b2-9c66-4373c5d839eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" Oct 31 00:03:52.873175 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:52.704843 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-0d96587a-58d9d8846-j62hj" Oct 31 00:03:52.873175 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:52.716782 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-27e82b/distinct-w2-27e82b-kserve-75f475fcd9-mccsz" Oct 31 00:03:52.873175 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:52.838158 4927 patch_prober.go:29] interesting pod/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-t4ms2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" start-of-body= Oct 31 00:03:52.873175 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:52.838214 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-t4ms2" podUID="1822bec6-cd20-4045-b362-d83766f3337b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" Oct 31 00:03:52.884461 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:52.884389 4927 patch_prober.go:29] interesting pod/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-mlkdz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" start-of-body= Oct 31 00:03:53.010816 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:52.981186 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-mlkdz" podUID="d9bfa829-54c3-4c65-8596-dca8560ec433" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" Oct 31 00:03:53.113200 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:53.104445 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fe01fa/distinct-2-w5-fe01fa-kserve-79b99966d8-dkhzd" podUID="d5c1c9e8-ce47-4868-860d-b8413ff74ab5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.111:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:03:53.120066 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:53.119942 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-986d37/premium-w3-986d37-kserve-5c7c7757fb-r99j7" podUID="f4e38df7-355f-4b47-a0b5-7e0b907f3326" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:03:53.222450 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:53.218700 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-35a187/distinct-2-w6-35a187-kserve-7ff4b6dc55-944g6" podUID="429ee092-42d8-475e-b0c6-b5a9930766fc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.113:8000/ready\": context deadline exceeded" Oct 31 00:03:53.295587 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:53.287882 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5c3374/premium-w1-5c3374-kserve-6c56d4dcd-llwcl" podUID="dc813f02-e077-4251-8c1e-fea0986b0f36" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:03:53.379360 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:53.297279 4927 patch_prober.go:29] interesting pod/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-pc4b9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" start-of-body= Oct 31 00:03:53.379360 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:53.305474 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-fe01fa/distinct-2-w5-fe01fa-kserve-79b99966d8-dkhzd" Oct 31 00:03:53.379360 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:53.308540 4927 patch_prober.go:29] interesting pod/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-hs4pf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body= Oct 31 00:03:53.379360 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:53.308622 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-hs4pf" podUID="9625c102-c118-48b2-9c66-4373c5d839eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" Oct 31 00:03:53.379360 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:53.341274 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-8s8j7" podUID="2d879158-1e38-47e0-8f8e-9f91e5d213ca" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:03:53.379360 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:53.378661 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-pc4b9" podUID="83ebf014-df38-445b-b3df-d2b9e8af0595" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" Oct 31 00:03:53.685805 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:53.685773 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5c3374/premium-w1-5c3374-kserve-6c56d4dcd-llwcl" Oct 31 00:03:53.748422 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:53.730432 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-46b586/simulator-w0-46b586-kserve-67588b8dbd-z8xmn" podUID="9a25e252-223d-41f2-abf7-632067c77e0b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:03:53.748422 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:53.743499 4927 patch_prober.go:29] interesting pod/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-mlkdz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" start-of-body= Oct 31 00:03:53.748422 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:53.743607 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-mlkdz" podUID="d9bfa829-54c3-4c65-8596-dca8560ec433" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" Oct 31 00:03:53.869374 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:53.846565 4927 patch_prober.go:29] interesting pod/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-t4ms2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" start-of-body= Oct 31 00:03:53.869374 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:53.846630 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-t4ms2" podUID="1822bec6-cd20-4045-b362-d83766f3337b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" Oct 31 00:03:53.876318 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:53.876277 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-27e82b/distinct-w2-27e82b-kserve-75f475fcd9-mccsz" podUID="4ba8ced4-e93f-46ac-87c5-3d0b7fadc02d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:03:53.919772 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:53.918897 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:03:54.153372 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:54.126530 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-986d37/premium-w3-986d37-kserve-5c7c7757fb-r99j7" podUID="f4e38df7-355f-4b47-a0b5-7e0b907f3326" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:03:54.192368 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:54.191698 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:03:54.224166 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:54.224113 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-35a187/distinct-2-w6-35a187-kserve-7ff4b6dc55-944g6" podUID="429ee092-42d8-475e-b0c6-b5a9930766fc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.113:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:03:54.284411 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:54.266058 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fe01fa/distinct-2-w5-fe01fa-kserve-79b99966d8-dkhzd" podUID="d5c1c9e8-ce47-4868-860d-b8413ff74ab5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.111:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 00:03:54.337356 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:54.297682 4927 patch_prober.go:29] interesting pod/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-pc4b9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" start-of-body= Oct 31 00:03:54.338069 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:54.338027 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-pc4b9" podUID="83ebf014-df38-445b-b3df-d2b9e8af0595" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" Oct 31 00:03:54.338220 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:54.301451 4927 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-986d37/premium-w3-986d37-kserve-5c7c7757fb-r99j7" podUID="f4e38df7-355f-4b47-a0b5-7e0b907f3326" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:03:54.338315 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:54.301232 4927 patch_prober.go:29] interesting pod/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-hs4pf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body= Oct 31 00:03:54.338445 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:54.338425 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-hs4pf" podUID="9625c102-c118-48b2-9c66-4373c5d839eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" Oct 31 00:03:54.396356 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:54.381667 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:03:54.409637 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:54.409597 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-e0751b65-7d7cd79459-76xvm" event={"ID":"ecce302f-aa28-48f3-86b1-374399cd6540","Type":"ContainerStarted","Data":"9e50bd7996162a381067877dcad4c1ab24d0bef97743d8e9f13eed169b6abcb6"} Oct 31 00:03:54.494981 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:54.493458 4927 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-35a187/distinct-2-w6-35a187-kserve-7ff4b6dc55-944g6" podUID="429ee092-42d8-475e-b0c6-b5a9930766fc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.113:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:03:54.558434 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:54.537536 4927 patch_prober.go:29] interesting pod/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-mlkdz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" start-of-body= Oct 31 00:03:54.558434 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:54.537599 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-mlkdz" podUID="d9bfa829-54c3-4c65-8596-dca8560ec433" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" Oct 31 00:03:54.693353 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:54.674090 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:03:54.700867 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:54.700731 4927 patch_prober.go:29] interesting pod/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-t4ms2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" start-of-body= Oct 31 00:03:54.700867 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:54.700796 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-t4ms2" podUID="1822bec6-cd20-4045-b362-d83766f3337b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" Oct 31 00:03:54.770814 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:54.738224 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-46b586/simulator-w0-46b586-kserve-67588b8dbd-z8xmn" podUID="9a25e252-223d-41f2-abf7-632067c77e0b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:03:54.770814 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:54.740715 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-e0751b65-7d7cd79459-76xvm"] Oct 31 00:03:54.855622 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:54.855234 4927 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-fe01fa/distinct-2-w5-fe01fa-kserve-79b99966d8-dkhzd" podUID="d5c1c9e8-ce47-4868-860d-b8413ff74ab5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.111:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:03:55.085357 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:55.081422 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-27e82b/distinct-w2-27e82b-kserve-75f475fcd9-mccsz" podUID="4ba8ced4-e93f-46ac-87c5-3d0b7fadc02d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/ready\": context deadline exceeded" Oct 31 00:03:55.085357 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:55.082843 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-27e82b/distinct-w2-27e82b-kserve-75f475fcd9-mccsz" Oct 31 00:03:55.138836 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:55.138018 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-986d37/premium-w3-986d37-kserve-5c7c7757fb-r99j7" podUID="f4e38df7-355f-4b47-a0b5-7e0b907f3326" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:03:55.232728 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:55.232670 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-35a187/distinct-2-w6-35a187-kserve-7ff4b6dc55-944g6" podUID="429ee092-42d8-475e-b0c6-b5a9930766fc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.113:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:03:55.286442 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:55.266906 4927 patch_prober.go:29] interesting pod/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-pc4b9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" start-of-body= Oct 31 00:03:55.345771 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:55.345729 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-pc4b9" podUID="83ebf014-df38-445b-b3df-d2b9e8af0595" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" Oct 31 00:03:55.346019 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:55.271306 4927 patch_prober.go:29] interesting pod/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-hs4pf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body= Oct 31 00:03:55.346128 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:55.346108 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-hs4pf" podUID="9625c102-c118-48b2-9c66-4373c5d839eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" Oct 31 00:03:55.346217 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:55.271374 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fe01fa/distinct-2-w5-fe01fa-kserve-79b99966d8-dkhzd" podUID="d5c1c9e8-ce47-4868-860d-b8413ff74ab5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.111:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:03:55.700362 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:55.699490 4927 patch_prober.go:29] interesting pod/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-mlkdz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" start-of-body= Oct 31 00:03:55.700362 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:55.699557 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-mlkdz" podUID="d9bfa829-54c3-4c65-8596-dca8560ec433" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" Oct 31 00:03:55.708320 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:55.708285 4927 patch_prober.go:29] interesting pod/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-t4ms2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" start-of-body= Oct 31 00:03:55.708616 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:55.708586 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-t4ms2" podUID="1822bec6-cd20-4045-b362-d83766f3337b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" Oct 31 00:03:55.755452 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:55.739816 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-46b586/simulator-w0-46b586-kserve-67588b8dbd-z8xmn" podUID="9a25e252-223d-41f2-abf7-632067c77e0b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded" Oct 31 00:03:55.884731 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:55.876312 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-e0751b65-7d7cd79459-76xvm" event={"ID":"ecce302f-aa28-48f3-86b1-374399cd6540","Type":"ContainerStarted","Data":"188a1e9b3dd78fe1bb546ad232bf8e0c359561dc0b114ef937b2023182f1eba6"} Oct 31 00:03:55.884983 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:55.884963 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-e0751b65-7d7cd79459-76xvm" Oct 31 00:03:56.011920 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:56.009812 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:03:56.084750 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:56.083908 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-27e82b/distinct-w2-27e82b-kserve-75f475fcd9-mccsz" podUID="4ba8ced4-e93f-46ac-87c5-3d0b7fadc02d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:03:56.086607 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:56.086562 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5c3374/premium-w1-5c3374-kserve-6c56d4dcd-llwcl" podUID="dc813f02-e077-4251-8c1e-fea0986b0f36" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:03:56.149386 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:56.147613 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-986d37/premium-w3-986d37-kserve-5c7c7757fb-r99j7" podUID="f4e38df7-355f-4b47-a0b5-7e0b907f3326" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded" Oct 31 00:03:56.290406 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:56.286137 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-35a187/distinct-2-w6-35a187-kserve-7ff4b6dc55-944g6" podUID="429ee092-42d8-475e-b0c6-b5a9930766fc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.113:8000/ready\": context deadline exceeded" Oct 31 00:03:56.347200 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:56.347124 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fe01fa/distinct-2-w5-fe01fa-kserve-79b99966d8-dkhzd" podUID="d5c1c9e8-ce47-4868-860d-b8413ff74ab5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.111:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:03:56.368376 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:56.367774 4927 patch_prober.go:29] interesting pod/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-hs4pf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body= Oct 31 00:03:56.368376 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:56.367848 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-hs4pf" podUID="9625c102-c118-48b2-9c66-4373c5d839eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" Oct 31 00:03:56.387415 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:56.379069 4927 patch_prober.go:29] interesting pod/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-pc4b9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" start-of-body= Oct 31 00:03:56.387415 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:56.379152 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-pc4b9" podUID="83ebf014-df38-445b-b3df-d2b9e8af0595" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" Oct 31 00:03:56.389140 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:56.389097 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:03:56.561819 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:56.561704 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-e0751b65-7d7cd79459-76xvm" podStartSLOduration=8.561664227 podStartE2EDuration="8.561664227s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:03:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:03:56.068174525 +0000 UTC m=+1349.104505347" watchObservedRunningTime="2026-10-31 00:03:56.561664227 +0000 UTC m=+1349.597995050" Oct 31 00:03:56.773769 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:56.773716 4927 patch_prober.go:29] interesting pod/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-mlkdz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" start-of-body= Oct 31 00:03:56.774409 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:56.774372 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-mlkdz" podUID="d9bfa829-54c3-4c65-8596-dca8560ec433" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" Oct 31 00:03:56.841286 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:56.841196 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:03:56.884412 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:56.881038 4927 patch_prober.go:29] interesting pod/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-t4ms2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" start-of-body= Oct 31 00:03:56.884412 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:56.881115 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-t4ms2" podUID="1822bec6-cd20-4045-b362-d83766f3337b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" Oct 31 00:03:56.899391 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:56.889885 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-46b586/simulator-w0-46b586-kserve-67588b8dbd-z8xmn" podUID="9a25e252-223d-41f2-abf7-632067c77e0b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:03:57.113525 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:57.113417 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5c3374/premium-w1-5c3374-kserve-6c56d4dcd-llwcl" podUID="dc813f02-e077-4251-8c1e-fea0986b0f36" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:03:57.114980 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:57.114949 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-27e82b/distinct-w2-27e82b-kserve-75f475fcd9-mccsz" podUID="4ba8ced4-e93f-46ac-87c5-3d0b7fadc02d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/ready\": context deadline exceeded" Oct 31 00:03:57.165721 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:57.165671 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-986d37/premium-w3-986d37-kserve-5c7c7757fb-r99j7" podUID="f4e38df7-355f-4b47-a0b5-7e0b907f3326" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded" Oct 31 00:03:57.206356 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:57.183354 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5c3374/premium-w1-5c3374-kserve-6c56d4dcd-llwcl" Oct 31 00:03:57.207174 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:57.207150 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-35a187/distinct-2-w6-35a187-kserve-7ff4b6dc55-944g6" Oct 31 00:03:57.281495 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:57.281455 4927 patch_prober.go:29] interesting pod/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-pc4b9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" start-of-body= Oct 31 00:03:57.281754 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:57.281716 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-pc4b9" podUID="83ebf014-df38-445b-b3df-d2b9e8af0595" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" Oct 31 00:03:57.297480 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:57.297448 4927 patch_prober.go:29] interesting pod/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-hs4pf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body= Oct 31 00:03:57.297694 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:57.297670 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-hs4pf" podUID="9625c102-c118-48b2-9c66-4373c5d839eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" Oct 31 00:03:57.354059 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:57.354018 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fe01fa/distinct-2-w5-fe01fa-kserve-79b99966d8-dkhzd" podUID="d5c1c9e8-ce47-4868-860d-b8413ff74ab5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.111:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:03:57.385473 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:57.378299 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-46b586/simulator-w0-46b586-kserve-67588b8dbd-z8xmn" Oct 31 00:03:57.396917 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:57.396889 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5c3374/premium-w1-5c3374-kserve-6c56d4dcd-llwcl" Oct 31 00:03:57.557616 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:57.557568 4927 patch_prober.go:29] interesting pod/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-mlkdz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" start-of-body= Oct 31 00:03:57.558125 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:57.558081 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-mlkdz" podUID="d9bfa829-54c3-4c65-8596-dca8560ec433" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" Oct 31 00:03:57.576836 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:57.576657 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-fe01fa/distinct-2-w5-fe01fa-kserve-79b99966d8-dkhzd" Oct 31 00:03:57.577944 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:57.577923 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-986d37/premium-w3-986d37-kserve-5c7c7757fb-r99j7" Oct 31 00:03:57.586718 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:57.586697 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-27e82b/distinct-w2-27e82b-kserve-75f475fcd9-mccsz" Oct 31 00:03:57.717358 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:57.696152 4927 patch_prober.go:29] interesting pod/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-t4ms2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" start-of-body= Oct 31 00:03:57.884366 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:57.873402 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-t4ms2" podUID="1822bec6-cd20-4045-b362-d83766f3337b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" Oct 31 00:03:58.265312 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:58.265127 4927 patch_prober.go:29] interesting pod/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-pc4b9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" start-of-body= Oct 31 00:03:58.265312 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:58.265191 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-pc4b9" podUID="83ebf014-df38-445b-b3df-d2b9e8af0595" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" Oct 31 00:03:58.536424 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:58.536174 4927 patch_prober.go:29] interesting pod/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-mlkdz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" start-of-body= Oct 31 00:03:58.536424 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:58.536235 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-mlkdz" podUID="d9bfa829-54c3-4c65-8596-dca8560ec433" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" Oct 31 00:03:58.684871 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:58.684693 4927 patch_prober.go:29] interesting pod/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-t4ms2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" start-of-body= Oct 31 00:03:58.684871 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:58.684751 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-t4ms2" podUID="1822bec6-cd20-4045-b362-d83766f3337b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" Oct 31 00:03:58.796297 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:58.796199 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-hs4pf" Oct 31 00:03:58.797409 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:58.797386 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-hs4pf" Oct 31 00:03:58.798255 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:58.798233 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-hs4pf" Oct 31 00:03:59.463046 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:59.462987 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-pc4b9" Oct 31 00:03:59.465386 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:59.464680 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-pc4b9" Oct 31 00:03:59.467501 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:59.467482 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-pc4b9" Oct 31 00:03:59.693910 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:59.693875 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-mlkdz" Oct 31 00:03:59.694324 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:59.694298 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-mlkdz" Oct 31 00:03:59.695242 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:59.695222 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-mlkdz" Oct 31 00:03:59.802455 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:59.802248 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-t4ms2" Oct 31 00:03:59.803205 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:59.803173 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-t4ms2" Oct 31 00:03:59.806991 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:59.806627 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-t4ms2" Oct 31 00:03:59.955656 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:03:59.955623 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:04:00.121111 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:00.121013 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:04:01.020909 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:01.020827 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:04:02.330459 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:02.326245 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:04:02.405359 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:02.404881 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5c3374/premium-w1-5c3374-kserve-6c56d4dcd-llwcl" podUID="dc813f02-e077-4251-8c1e-fea0986b0f36" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:04:02.461351 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:02.460491 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-46b586/simulator-w0-46b586-kserve-67588b8dbd-z8xmn" podUID="9a25e252-223d-41f2-abf7-632067c77e0b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:04:02.728475 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:02.697592 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:04:02.946359 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:02.889941 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:04:03.138441 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:03.114247 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-27e82b/distinct-w2-27e82b-kserve-75f475fcd9-mccsz" podUID="4ba8ced4-e93f-46ac-87c5-3d0b7fadc02d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:04:03.173357 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:03.169137 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-986d37/premium-w3-986d37-kserve-5c7c7757fb-r99j7" podUID="f4e38df7-355f-4b47-a0b5-7e0b907f3326" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded" Oct 31 00:04:03.226357 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:03.222270 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-35a187/distinct-2-w6-35a187-kserve-7ff4b6dc55-944g6" podUID="429ee092-42d8-475e-b0c6-b5a9930766fc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.113:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:04:03.428443 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:03.422745 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5c3374/premium-w1-5c3374-kserve-6c56d4dcd-llwcl" podUID="dc813f02-e077-4251-8c1e-fea0986b0f36" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:04:04.122393 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:04.117704 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-27e82b/distinct-w2-27e82b-kserve-75f475fcd9-mccsz" podUID="4ba8ced4-e93f-46ac-87c5-3d0b7fadc02d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/ready\": context deadline exceeded" Oct 31 00:04:04.196044 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:04.195996 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-986d37/premium-w3-986d37-kserve-5c7c7757fb-r99j7" podUID="f4e38df7-355f-4b47-a0b5-7e0b907f3326" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded" Oct 31 00:04:04.257864 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:04:04.216739 4927 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.102s" Oct 31 00:04:04.274636 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:04.274599 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-d32a7cc0-76fd97d5d6-62rcv"] Oct 31 00:04:04.336612 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:04.294426 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fe01fa/distinct-2-w5-fe01fa-kserve-79b99966d8-dkhzd" podUID="d5c1c9e8-ce47-4868-860d-b8413ff74ab5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.111:8000/ready\": context deadline exceeded" Oct 31 00:04:04.336612 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:04.317785 4927 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-986d37/premium-w3-986d37-kserve-5c7c7757fb-r99j7" podUID="f4e38df7-355f-4b47-a0b5-7e0b907f3326" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:04:04.459284 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:04.459240 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5c3374/premium-w1-5c3374-kserve-6c56d4dcd-llwcl" Oct 31 00:04:04.479815 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:04.479782 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-d32a7cc0-76fd97d5d6-62rcv" Oct 31 00:04:04.481220 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:04.480084 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-d32a7cc0-76fd97d5d6-62rcv"] Oct 31 00:04:04.482320 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:04.482301 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-27e82b/distinct-w2-27e82b-kserve-75f475fcd9-mccsz" Oct 31 00:04:04.482768 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:04.482750 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-986d37/premium-w3-986d37-kserve-5c7c7757fb-r99j7" Oct 31 00:04:04.483128 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:04.483112 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5c3374/premium-w1-5c3374-kserve-6c56d4dcd-llwcl" Oct 31 00:04:04.492149 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:04.492122 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-d32a7cc0-6b7768cb67-ntrqr"] Oct 31 00:04:04.963293 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:04.517493 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-46b586/simulator-w0-46b586-kserve-67588b8dbd-z8xmn" podUID="9a25e252-223d-41f2-abf7-632067c77e0b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded" Oct 31 00:04:04.989594 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:04.545443 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-d32a7cc0\"" Oct 31 00:04:04.989594 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:04.545588 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-d32a7cc0-dockercfg-p9c28\"" Oct 31 00:04:04.989594 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:04.546002 4927 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-fe01fa/distinct-2-w5-fe01fa-kserve-79b99966d8-dkhzd" podUID="d5c1c9e8-ce47-4868-860d-b8413ff74ab5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.111:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:04:05.071368 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:05.035644 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-d32a7cc0-6b7768cb67-ntrqr" Oct 31 00:04:05.071368 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:05.052086 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-d32a7cc0-6b7768cb67-ntrqr"] Oct 31 00:04:05.128748 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:05.127295 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-27e82b/distinct-w2-27e82b-kserve-75f475fcd9-mccsz" podUID="4ba8ced4-e93f-46ac-87c5-3d0b7fadc02d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:04:05.163034 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:05.150822 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8t8kn\" (UniqueName: \"kubernetes.io/projected/d0961830-61e4-4528-a704-82a109b453fb-kube-api-access-8t8kn\") pod \"payload-processing-e2e-migrate-d32a7cc0-76fd97d5d6-62rcv\" (UID: \"d0961830-61e4-4528-a704-82a109b453fb\") " pod="openshift-ingress/payload-processing-e2e-migrate-d32a7cc0-76fd97d5d6-62rcv" Oct 31 00:04:05.163034 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:05.151103 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sb6f7\" (UniqueName: \"kubernetes.io/projected/ea4dd2c2-8b72-4ec5-b43d-5bc1fe96f391-kube-api-access-sb6f7\") pod \"payload-pre-processing-e2e-migrate-d32a7cc0-6b7768cb67-ntrqr\" (UID: \"ea4dd2c2-8b72-4ec5-b43d-5bc1fe96f391\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-d32a7cc0-6b7768cb67-ntrqr" Oct 31 00:04:05.163034 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:05.151230 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0961830-61e4-4528-a704-82a109b453fb-plugins-config-volume\") pod \"payload-processing-e2e-migrate-d32a7cc0-76fd97d5d6-62rcv\" (UID: \"d0961830-61e4-4528-a704-82a109b453fb\") " pod="openshift-ingress/payload-processing-e2e-migrate-d32a7cc0-76fd97d5d6-62rcv" Oct 31 00:04:05.163034 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:05.151260 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea4dd2c2-8b72-4ec5-b43d-5bc1fe96f391-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-d32a7cc0-6b7768cb67-ntrqr\" (UID: \"ea4dd2c2-8b72-4ec5-b43d-5bc1fe96f391\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-d32a7cc0-6b7768cb67-ntrqr" Oct 31 00:04:05.197692 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:05.196470 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-986d37/premium-w3-986d37-kserve-5c7c7757fb-r99j7" podUID="f4e38df7-355f-4b47-a0b5-7e0b907f3326" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:04:05.378408 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:05.259519 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-35a187/distinct-2-w6-35a187-kserve-7ff4b6dc55-944g6" podUID="429ee092-42d8-475e-b0c6-b5a9930766fc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.113:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:04:05.378408 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:05.276029 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8t8kn\" (UniqueName: \"kubernetes.io/projected/d0961830-61e4-4528-a704-82a109b453fb-kube-api-access-8t8kn\") pod \"payload-processing-e2e-migrate-d32a7cc0-76fd97d5d6-62rcv\" (UID: \"d0961830-61e4-4528-a704-82a109b453fb\") " pod="openshift-ingress/payload-processing-e2e-migrate-d32a7cc0-76fd97d5d6-62rcv" Oct 31 00:04:05.378408 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:05.276209 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sb6f7\" (UniqueName: \"kubernetes.io/projected/ea4dd2c2-8b72-4ec5-b43d-5bc1fe96f391-kube-api-access-sb6f7\") pod \"payload-pre-processing-e2e-migrate-d32a7cc0-6b7768cb67-ntrqr\" (UID: \"ea4dd2c2-8b72-4ec5-b43d-5bc1fe96f391\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-d32a7cc0-6b7768cb67-ntrqr" Oct 31 00:04:05.378408 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:05.276286 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0961830-61e4-4528-a704-82a109b453fb-plugins-config-volume\") pod \"payload-processing-e2e-migrate-d32a7cc0-76fd97d5d6-62rcv\" (UID: \"d0961830-61e4-4528-a704-82a109b453fb\") " pod="openshift-ingress/payload-processing-e2e-migrate-d32a7cc0-76fd97d5d6-62rcv" Oct 31 00:04:05.378408 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:05.276350 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea4dd2c2-8b72-4ec5-b43d-5bc1fe96f391-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-d32a7cc0-6b7768cb67-ntrqr\" (UID: \"ea4dd2c2-8b72-4ec5-b43d-5bc1fe96f391\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-d32a7cc0-6b7768cb67-ntrqr" Oct 31 00:04:05.378408 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:05.316289 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0961830-61e4-4528-a704-82a109b453fb-plugins-config-volume\") pod \"payload-processing-e2e-migrate-d32a7cc0-76fd97d5d6-62rcv\" (UID: \"d0961830-61e4-4528-a704-82a109b453fb\") " pod="openshift-ingress/payload-processing-e2e-migrate-d32a7cc0-76fd97d5d6-62rcv" Oct 31 00:04:05.378408 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:05.369023 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea4dd2c2-8b72-4ec5-b43d-5bc1fe96f391-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-d32a7cc0-6b7768cb67-ntrqr\" (UID: \"ea4dd2c2-8b72-4ec5-b43d-5bc1fe96f391\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-d32a7cc0-6b7768cb67-ntrqr" Oct 31 00:04:05.451790 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:05.410666 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5c3374/premium-w1-5c3374-kserve-6c56d4dcd-llwcl" podUID="dc813f02-e077-4251-8c1e-fea0986b0f36" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 00:04:05.730134 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:05.711306 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8t8kn\" (UniqueName: \"kubernetes.io/projected/d0961830-61e4-4528-a704-82a109b453fb-kube-api-access-8t8kn\") pod \"payload-processing-e2e-migrate-d32a7cc0-76fd97d5d6-62rcv\" (UID: \"d0961830-61e4-4528-a704-82a109b453fb\") " pod="openshift-ingress/payload-processing-e2e-migrate-d32a7cc0-76fd97d5d6-62rcv" Oct 31 00:04:05.798358 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:05.790375 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sb6f7\" (UniqueName: \"kubernetes.io/projected/ea4dd2c2-8b72-4ec5-b43d-5bc1fe96f391-kube-api-access-sb6f7\") pod \"payload-pre-processing-e2e-migrate-d32a7cc0-6b7768cb67-ntrqr\" (UID: \"ea4dd2c2-8b72-4ec5-b43d-5bc1fe96f391\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-d32a7cc0-6b7768cb67-ntrqr" Oct 31 00:04:05.886202 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:05.885912 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-d32a7cc0-76fd97d5d6-62rcv" Oct 31 00:04:06.006548 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:06.006495 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-46b586/simulator-w0-46b586-kserve-67588b8dbd-z8xmn" podUID="9a25e252-223d-41f2-abf7-632067c77e0b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:04:06.007390 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:06.007363 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-46b586/simulator-w0-46b586-kserve-67588b8dbd-z8xmn" Oct 31 00:04:06.121362 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:06.080774 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-d32a7cc0-6b7768cb67-ntrqr" Oct 31 00:04:06.171278 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:06.137767 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-27e82b/distinct-w2-27e82b-kserve-75f475fcd9-mccsz" podUID="4ba8ced4-e93f-46ac-87c5-3d0b7fadc02d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:04:06.207600 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:06.207537 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-986d37/premium-w3-986d37-kserve-5c7c7757fb-r99j7" podUID="f4e38df7-355f-4b47-a0b5-7e0b907f3326" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded" Oct 31 00:04:06.260028 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:06.259930 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-35a187/distinct-2-w6-35a187-kserve-7ff4b6dc55-944g6" podUID="429ee092-42d8-475e-b0c6-b5a9930766fc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.113:8000/ready\": context deadline exceeded" Oct 31 00:04:06.260638 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:06.260611 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-35a187/distinct-2-w6-35a187-kserve-7ff4b6dc55-944g6" Oct 31 00:04:06.316623 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:06.315462 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fe01fa/distinct-2-w5-fe01fa-kserve-79b99966d8-dkhzd" podUID="d5c1c9e8-ce47-4868-860d-b8413ff74ab5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.111:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:04:06.417854 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:06.417800 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5c3374/premium-w1-5c3374-kserve-6c56d4dcd-llwcl" podUID="dc813f02-e077-4251-8c1e-fea0986b0f36" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded" Oct 31 00:04:06.457433 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:06.457394 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5c3374/premium-w1-5c3374-kserve-6c56d4dcd-llwcl" Oct 31 00:04:06.815391 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:06.815164 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:04:07.098489 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:07.035533 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-e0751b65-7d7cd79459-76xvm" Oct 31 00:04:07.098489 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:07.052946 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-46b586/simulator-w0-46b586-kserve-67588b8dbd-z8xmn" podUID="9a25e252-223d-41f2-abf7-632067c77e0b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded" Oct 31 00:04:07.157694 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:07.157637 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-27e82b/distinct-w2-27e82b-kserve-75f475fcd9-mccsz" podUID="4ba8ced4-e93f-46ac-87c5-3d0b7fadc02d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:04:07.263778 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:07.256435 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-986d37/premium-w3-986d37-kserve-5c7c7757fb-r99j7" podUID="f4e38df7-355f-4b47-a0b5-7e0b907f3326" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 00:04:07.273360 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:07.266106 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-35a187/distinct-2-w6-35a187-kserve-7ff4b6dc55-944g6" podUID="429ee092-42d8-475e-b0c6-b5a9930766fc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.113:8000/ready\": context deadline exceeded" Oct 31 00:04:07.347358 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:07.330509 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fe01fa/distinct-2-w5-fe01fa-kserve-79b99966d8-dkhzd" podUID="d5c1c9e8-ce47-4868-860d-b8413ff74ab5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.111:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:04:07.427361 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:07.423765 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5c3374/premium-w1-5c3374-kserve-6c56d4dcd-llwcl" podUID="dc813f02-e077-4251-8c1e-fea0986b0f36" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded" Oct 31 00:04:07.669357 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:07.655657 4927 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 31 00:04:07.809351 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:07.809291 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-fe01fa/distinct-2-w5-fe01fa-kserve-79b99966d8-dkhzd" Oct 31 00:04:07.837828 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:07.837798 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-fe01fa/distinct-2-w5-fe01fa-kserve-79b99966d8-dkhzd" Oct 31 00:04:08.100636 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:08.100543 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-46b586/simulator-w0-46b586-kserve-67588b8dbd-z8xmn" podUID="9a25e252-223d-41f2-abf7-632067c77e0b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded" Oct 31 00:04:08.184736 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:08.184693 4927 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-46b586/simulator-w0-46b586-kserve-67588b8dbd-z8xmn" podUID="9a25e252-223d-41f2-abf7-632067c77e0b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:04:08.193512 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:08.184976 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-27e82b/distinct-w2-27e82b-kserve-75f475fcd9-mccsz" podUID="4ba8ced4-e93f-46ac-87c5-3d0b7fadc02d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:04:08.194494 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:08.185389 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-986d37/premium-w3-986d37-kserve-5c7c7757fb-r99j7" Oct 31 00:04:08.218411 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:08.217644 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-35a187/distinct-2-w6-35a187-kserve-7ff4b6dc55-944g6" Oct 31 00:04:08.242996 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:08.242964 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5c3374/premium-w1-5c3374-kserve-6c56d4dcd-llwcl" Oct 31 00:04:08.300372 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:08.285411 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-46b586/simulator-w0-46b586-kserve-67588b8dbd-z8xmn" Oct 31 00:04:08.353778 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:08.353692 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-27e82b/distinct-w2-27e82b-kserve-75f475fcd9-mccsz" Oct 31 00:04:08.798349 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:08.798292 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-d32a7cc0-76fd97d5d6-62rcv"] Oct 31 00:04:08.889379 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:08.888819 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-d32a7cc0-6b7768cb67-ntrqr"] Oct 31 00:04:09.648934 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:09.648894 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-d32a7cc0-6b7768cb67-ntrqr" event={"ID":"ea4dd2c2-8b72-4ec5-b43d-5bc1fe96f391","Type":"ContainerStarted","Data":"c3414683b862bd58304224fd39aa282d41e9f177168b9f828f096bf69e01163b"} Oct 31 00:04:09.649166 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:09.648982 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-d32a7cc0-6b7768cb67-ntrqr" event={"ID":"ea4dd2c2-8b72-4ec5-b43d-5bc1fe96f391","Type":"ContainerStarted","Data":"394e776fec5d1e36d4349f1d2238ed5fd5b5b43613c64dd67cbc885b7452903f"} Oct 31 00:04:09.649243 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:09.649164 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-d32a7cc0-6b7768cb67-ntrqr" Oct 31 00:04:09.651006 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:09.650973 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-d32a7cc0-76fd97d5d6-62rcv" event={"ID":"d0961830-61e4-4528-a704-82a109b453fb","Type":"ContainerStarted","Data":"49a24120ab046f00ad3aa39f7eb40170389c9d593df058a3552798bf916642c1"} Oct 31 00:04:09.651157 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:09.651017 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-d32a7cc0-76fd97d5d6-62rcv" event={"ID":"d0961830-61e4-4528-a704-82a109b453fb","Type":"ContainerStarted","Data":"b6867d288486ccf8eb336934acb3a7335d2e8016dd2041d036adcb8ea381c46e"} Oct 31 00:04:09.651157 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:09.651142 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-d32a7cc0-76fd97d5d6-62rcv" Oct 31 00:04:09.772033 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:09.771972 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-d32a7cc0-6b7768cb67-ntrqr" podStartSLOduration=6.771956779 podStartE2EDuration="6.771956779s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:04:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:04:09.767840105 +0000 UTC m=+1362.804170936" watchObservedRunningTime="2026-10-31 00:04:09.771956779 +0000 UTC m=+1362.808287611" Oct 31 00:04:09.929353 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:09.912815 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-d32a7cc0-76fd97d5d6-62rcv" podStartSLOduration=6.912791927 podStartE2EDuration="6.912791927s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:04:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:04:09.890817115 +0000 UTC m=+1362.927147961" watchObservedRunningTime="2026-10-31 00:04:09.912791927 +0000 UTC m=+1362.949122750" Oct 31 00:04:11.163774 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:11.163735 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:04:11.390738 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:11.390132 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:04:11.545801 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:11.545711 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:04:15.935407 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:15.935368 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:04:16.148368 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:16.146258 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:04:16.323061 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:16.322839 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:04:16.715127 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:16.715085 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-zvnnz"] Oct 31 00:04:16.720579 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:16.720551 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-zvnnz" Oct 31 00:04:16.795156 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:16.795117 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ba0721f-a4d2-49f6-b161-6d6f0f8ae4c2-credential-socket\") pod \"maas-default-gateway-openshift-default-8f6f87f59-zvnnz\" (UID: \"6ba0721f-a4d2-49f6-b161-6d6f0f8ae4c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-zvnnz" Oct 31 00:04:16.795343 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:16.795218 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ba0721f-a4d2-49f6-b161-6d6f0f8ae4c2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8f6f87f59-zvnnz\" (UID: \"6ba0721f-a4d2-49f6-b161-6d6f0f8ae4c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-zvnnz" Oct 31 00:04:16.795426 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:16.795400 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ba0721f-a4d2-49f6-b161-6d6f0f8ae4c2-istio-data\") pod \"maas-default-gateway-openshift-default-8f6f87f59-zvnnz\" (UID: \"6ba0721f-a4d2-49f6-b161-6d6f0f8ae4c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-zvnnz" Oct 31 00:04:16.795486 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:16.795454 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ba0721f-a4d2-49f6-b161-6d6f0f8ae4c2-workload-certs\") pod \"maas-default-gateway-openshift-default-8f6f87f59-zvnnz\" (UID: \"6ba0721f-a4d2-49f6-b161-6d6f0f8ae4c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-zvnnz" Oct 31 00:04:16.795602 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:16.795580 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ba0721f-a4d2-49f6-b161-6d6f0f8ae4c2-istio-envoy\") pod \"maas-default-gateway-openshift-default-8f6f87f59-zvnnz\" (UID: \"6ba0721f-a4d2-49f6-b161-6d6f0f8ae4c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-zvnnz" Oct 31 00:04:16.795699 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:16.795682 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ba0721f-a4d2-49f6-b161-6d6f0f8ae4c2-workload-socket\") pod \"maas-default-gateway-openshift-default-8f6f87f59-zvnnz\" (UID: \"6ba0721f-a4d2-49f6-b161-6d6f0f8ae4c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-zvnnz" Oct 31 00:04:16.795755 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:16.795719 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ba0721f-a4d2-49f6-b161-6d6f0f8ae4c2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8f6f87f59-zvnnz\" (UID: \"6ba0721f-a4d2-49f6-b161-6d6f0f8ae4c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-zvnnz" Oct 31 00:04:16.795909 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:16.795894 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ba0721f-a4d2-49f6-b161-6d6f0f8ae4c2-istio-token\") pod \"maas-default-gateway-openshift-default-8f6f87f59-zvnnz\" (UID: \"6ba0721f-a4d2-49f6-b161-6d6f0f8ae4c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-zvnnz" Oct 31 00:04:16.796046 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:16.796022 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55vnq\" (UniqueName: \"kubernetes.io/projected/6ba0721f-a4d2-49f6-b161-6d6f0f8ae4c2-kube-api-access-55vnq\") pod \"maas-default-gateway-openshift-default-8f6f87f59-zvnnz\" (UID: \"6ba0721f-a4d2-49f6-b161-6d6f0f8ae4c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-zvnnz" Oct 31 00:04:16.806218 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:16.806188 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-zvnnz"] Oct 31 00:04:16.897558 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:16.897516 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55vnq\" (UniqueName: \"kubernetes.io/projected/6ba0721f-a4d2-49f6-b161-6d6f0f8ae4c2-kube-api-access-55vnq\") pod \"maas-default-gateway-openshift-default-8f6f87f59-zvnnz\" (UID: \"6ba0721f-a4d2-49f6-b161-6d6f0f8ae4c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-zvnnz" Oct 31 00:04:16.897720 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:16.897580 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ba0721f-a4d2-49f6-b161-6d6f0f8ae4c2-credential-socket\") pod \"maas-default-gateway-openshift-default-8f6f87f59-zvnnz\" (UID: \"6ba0721f-a4d2-49f6-b161-6d6f0f8ae4c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-zvnnz" Oct 31 00:04:16.897720 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:16.897604 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ba0721f-a4d2-49f6-b161-6d6f0f8ae4c2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8f6f87f59-zvnnz\" (UID: \"6ba0721f-a4d2-49f6-b161-6d6f0f8ae4c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-zvnnz" Oct 31 00:04:16.897720 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:16.897695 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ba0721f-a4d2-49f6-b161-6d6f0f8ae4c2-istio-data\") pod \"maas-default-gateway-openshift-default-8f6f87f59-zvnnz\" (UID: \"6ba0721f-a4d2-49f6-b161-6d6f0f8ae4c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-zvnnz" Oct 31 00:04:16.897842 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:16.897727 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ba0721f-a4d2-49f6-b161-6d6f0f8ae4c2-workload-certs\") pod \"maas-default-gateway-openshift-default-8f6f87f59-zvnnz\" (UID: \"6ba0721f-a4d2-49f6-b161-6d6f0f8ae4c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-zvnnz" Oct 31 00:04:16.898047 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:16.898015 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ba0721f-a4d2-49f6-b161-6d6f0f8ae4c2-istio-envoy\") pod \"maas-default-gateway-openshift-default-8f6f87f59-zvnnz\" (UID: \"6ba0721f-a4d2-49f6-b161-6d6f0f8ae4c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-zvnnz" Oct 31 00:04:16.898150 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:16.898044 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ba0721f-a4d2-49f6-b161-6d6f0f8ae4c2-workload-certs\") pod \"maas-default-gateway-openshift-default-8f6f87f59-zvnnz\" (UID: \"6ba0721f-a4d2-49f6-b161-6d6f0f8ae4c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-zvnnz" Oct 31 00:04:16.898150 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:16.897941 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ba0721f-a4d2-49f6-b161-6d6f0f8ae4c2-credential-socket\") pod \"maas-default-gateway-openshift-default-8f6f87f59-zvnnz\" (UID: \"6ba0721f-a4d2-49f6-b161-6d6f0f8ae4c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-zvnnz" Oct 31 00:04:16.898150 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:16.898074 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ba0721f-a4d2-49f6-b161-6d6f0f8ae4c2-istio-data\") pod \"maas-default-gateway-openshift-default-8f6f87f59-zvnnz\" (UID: \"6ba0721f-a4d2-49f6-b161-6d6f0f8ae4c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-zvnnz" Oct 31 00:04:16.898150 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:16.898126 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ba0721f-a4d2-49f6-b161-6d6f0f8ae4c2-workload-socket\") pod \"maas-default-gateway-openshift-default-8f6f87f59-zvnnz\" (UID: \"6ba0721f-a4d2-49f6-b161-6d6f0f8ae4c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-zvnnz" Oct 31 00:04:16.898300 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:16.898147 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ba0721f-a4d2-49f6-b161-6d6f0f8ae4c2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8f6f87f59-zvnnz\" (UID: \"6ba0721f-a4d2-49f6-b161-6d6f0f8ae4c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-zvnnz" Oct 31 00:04:16.898372 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:16.898311 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ba0721f-a4d2-49f6-b161-6d6f0f8ae4c2-istio-token\") pod \"maas-default-gateway-openshift-default-8f6f87f59-zvnnz\" (UID: \"6ba0721f-a4d2-49f6-b161-6d6f0f8ae4c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-zvnnz" Oct 31 00:04:16.898438 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:16.898419 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ba0721f-a4d2-49f6-b161-6d6f0f8ae4c2-workload-socket\") pod \"maas-default-gateway-openshift-default-8f6f87f59-zvnnz\" (UID: \"6ba0721f-a4d2-49f6-b161-6d6f0f8ae4c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-zvnnz" Oct 31 00:04:16.898495 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:16.898463 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ba0721f-a4d2-49f6-b161-6d6f0f8ae4c2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8f6f87f59-zvnnz\" (UID: \"6ba0721f-a4d2-49f6-b161-6d6f0f8ae4c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-zvnnz" Oct 31 00:04:16.900186 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:16.900163 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ba0721f-a4d2-49f6-b161-6d6f0f8ae4c2-istio-envoy\") pod \"maas-default-gateway-openshift-default-8f6f87f59-zvnnz\" (UID: \"6ba0721f-a4d2-49f6-b161-6d6f0f8ae4c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-zvnnz" Oct 31 00:04:16.900655 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:16.900633 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ba0721f-a4d2-49f6-b161-6d6f0f8ae4c2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8f6f87f59-zvnnz\" (UID: \"6ba0721f-a4d2-49f6-b161-6d6f0f8ae4c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-zvnnz" Oct 31 00:04:16.959794 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:16.959754 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ba0721f-a4d2-49f6-b161-6d6f0f8ae4c2-istio-token\") pod \"maas-default-gateway-openshift-default-8f6f87f59-zvnnz\" (UID: \"6ba0721f-a4d2-49f6-b161-6d6f0f8ae4c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-zvnnz" Oct 31 00:04:16.960189 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:16.959874 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-55vnq\" (UniqueName: \"kubernetes.io/projected/6ba0721f-a4d2-49f6-b161-6d6f0f8ae4c2-kube-api-access-55vnq\") pod \"maas-default-gateway-openshift-default-8f6f87f59-zvnnz\" (UID: \"6ba0721f-a4d2-49f6-b161-6d6f0f8ae4c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-zvnnz" Oct 31 00:04:17.040180 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:17.040083 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-zvnnz" Oct 31 00:04:17.504371 ip-10-0-141-180 kubenswrapper[4927]: W1031 00:04:17.504308 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ba0721f_a4d2_49f6_b161_6d6f0f8ae4c2.slice/crio-dad7abb14198704cf22325290071d73dc8b4e57f05687463f2dc38060448473a WatchSource:0}: Error finding container dad7abb14198704cf22325290071d73dc8b4e57f05687463f2dc38060448473a: Status 404 returned error can't find the container with id dad7abb14198704cf22325290071d73dc8b4e57f05687463f2dc38060448473a Oct 31 00:04:17.506866 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:17.506827 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:04:17.506985 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:17.506912 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:04:17.506985 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:17.506959 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:04:17.507132 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:17.507109 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-zvnnz"] Oct 31 00:04:17.741783 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:17.741742 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-zvnnz" event={"ID":"6ba0721f-a4d2-49f6-b161-6d6f0f8ae4c2","Type":"ContainerStarted","Data":"8257ddc463e6fa3295e8c23e45ff1f3698e14ecabea72d76fe68ae6b3477a1d6"} Oct 31 00:04:17.741963 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:17.741814 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-zvnnz" event={"ID":"6ba0721f-a4d2-49f6-b161-6d6f0f8ae4c2","Type":"ContainerStarted","Data":"dad7abb14198704cf22325290071d73dc8b4e57f05687463f2dc38060448473a"} Oct 31 00:04:17.888957 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:17.888860 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-zvnnz" podStartSLOduration=1.8888440100000001 podStartE2EDuration="1.88884401s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:04:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:04:17.881725851 +0000 UTC m=+1370.918056677" watchObservedRunningTime="2026-10-31 00:04:17.88884401 +0000 UTC m=+1370.925174884" Oct 31 00:04:18.041276 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:18.041242 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-zvnnz" Oct 31 00:04:18.043031 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:18.043000 4927 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-zvnnz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body= Oct 31 00:04:18.043174 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:18.043057 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-zvnnz" podUID="6ba0721f-a4d2-49f6-b161-6d6f0f8ae4c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" Oct 31 00:04:19.043909 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:19.043870 4927 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-zvnnz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body= Oct 31 00:04:19.044625 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:19.044574 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-zvnnz" podUID="6ba0721f-a4d2-49f6-b161-6d6f0f8ae4c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" Oct 31 00:04:19.683448 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:04:19.683413 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sqv69" containerName="extractor" Oct 31 00:04:20.041551 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:20.041510 4927 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-zvnnz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body= Oct 31 00:04:20.041861 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:20.041808 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-zvnnz" podUID="6ba0721f-a4d2-49f6-b161-6d6f0f8ae4c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" Oct 31 00:04:20.664910 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:20.664881 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-d32a7cc0-76fd97d5d6-62rcv" Oct 31 00:04:20.665347 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:20.664993 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-d32a7cc0-6b7768cb67-ntrqr" Oct 31 00:04:20.738924 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:20.738759 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:04:21.054600 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:21.053865 4927 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-zvnnz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body= Oct 31 00:04:21.054600 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:21.053933 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-zvnnz" podUID="6ba0721f-a4d2-49f6-b161-6d6f0f8ae4c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" Oct 31 00:04:22.040734 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:22.040701 4927 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-zvnnz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body= Oct 31 00:04:22.041128 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:22.040755 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-zvnnz" podUID="6ba0721f-a4d2-49f6-b161-6d6f0f8ae4c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" Oct 31 00:04:23.041215 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:23.041183 4927 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-zvnnz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body= Oct 31 00:04:23.041625 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:23.041238 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-zvnnz" podUID="6ba0721f-a4d2-49f6-b161-6d6f0f8ae4c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" Oct 31 00:04:24.041176 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:24.041135 4927 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-zvnnz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body= Oct 31 00:04:24.041390 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:24.041198 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-zvnnz" podUID="6ba0721f-a4d2-49f6-b161-6d6f0f8ae4c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" Oct 31 00:04:25.041538 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:25.041494 4927 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-zvnnz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body= Oct 31 00:04:25.041908 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:25.041576 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-zvnnz" podUID="6ba0721f-a4d2-49f6-b161-6d6f0f8ae4c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" Oct 31 00:04:25.405803 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:25.405710 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:04:25.610544 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:25.610510 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:04:26.589509 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:26.589478 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-zvnnz" Oct 31 00:04:26.589964 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:26.589939 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-zvnnz" Oct 31 00:04:26.590744 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:26.590713 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-zvnnz" Oct 31 00:04:28.853381 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:28.853344 4927 scope.go:122] "RemoveContainer" containerID="814de666ae9679c71902461fe5fc2dbfd66068bea2574d8c8731cd408dc111f5" Oct 31 00:04:28.880495 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:28.880463 4927 scope.go:122] "RemoveContainer" containerID="5cd99cb82d13da4c5d9598682dff7706c9c170d579d3d39f0344434b0b9d383d" Oct 31 00:04:28.895137 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:28.895105 4927 scope.go:122] "RemoveContainer" containerID="edb199ea0829c5898e4eef8b946c683f4e0673fe9a0b509d4287dfb35f56d9a7" Oct 31 00:04:28.905877 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:28.905852 4927 scope.go:122] "RemoveContainer" containerID="eb893e33419dee156c59d8629ad9f208247a92d5f7462f9fd06899b9ebae03b7" Oct 31 00:04:29.888106 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:29.888076 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:04:31.849305 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:31.849270 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-8fws9"] Oct 31 00:04:31.856122 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:31.856099 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-8fws9" Oct 31 00:04:31.868812 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:31.868789 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-dpcqs"] Oct 31 00:04:31.871139 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:31.871112 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/daac49a1-fcf0-4772-b555-5f43b01f8c3f-istio-data\") pod \"maas-default-gateway-openshift-default-8f6f87f59-8fws9\" (UID: \"daac49a1-fcf0-4772-b555-5f43b01f8c3f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-8fws9" Oct 31 00:04:31.871242 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:31.871199 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/daac49a1-fcf0-4772-b555-5f43b01f8c3f-credential-socket\") pod \"maas-default-gateway-openshift-default-8f6f87f59-8fws9\" (UID: \"daac49a1-fcf0-4772-b555-5f43b01f8c3f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-8fws9" Oct 31 00:04:31.871242 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:31.871225 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wbxjq\" (UniqueName: \"kubernetes.io/projected/daac49a1-fcf0-4772-b555-5f43b01f8c3f-kube-api-access-wbxjq\") pod \"maas-default-gateway-openshift-default-8f6f87f59-8fws9\" (UID: \"daac49a1-fcf0-4772-b555-5f43b01f8c3f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-8fws9" Oct 31 00:04:31.871388 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:31.871371 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/daac49a1-fcf0-4772-b555-5f43b01f8c3f-istio-token\") pod \"maas-default-gateway-openshift-default-8f6f87f59-8fws9\" (UID: \"daac49a1-fcf0-4772-b555-5f43b01f8c3f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-8fws9" Oct 31 00:04:31.871459 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:31.871440 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/daac49a1-fcf0-4772-b555-5f43b01f8c3f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8f6f87f59-8fws9\" (UID: \"daac49a1-fcf0-4772-b555-5f43b01f8c3f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-8fws9" Oct 31 00:04:31.871520 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:31.871507 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/daac49a1-fcf0-4772-b555-5f43b01f8c3f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8f6f87f59-8fws9\" (UID: \"daac49a1-fcf0-4772-b555-5f43b01f8c3f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-8fws9" Oct 31 00:04:31.871626 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:31.871610 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/daac49a1-fcf0-4772-b555-5f43b01f8c3f-istio-envoy\") pod \"maas-default-gateway-openshift-default-8f6f87f59-8fws9\" (UID: \"daac49a1-fcf0-4772-b555-5f43b01f8c3f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-8fws9" Oct 31 00:04:31.871690 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:31.871673 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/daac49a1-fcf0-4772-b555-5f43b01f8c3f-workload-socket\") pod \"maas-default-gateway-openshift-default-8f6f87f59-8fws9\" (UID: \"daac49a1-fcf0-4772-b555-5f43b01f8c3f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-8fws9" Oct 31 00:04:31.871762 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:31.871710 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/daac49a1-fcf0-4772-b555-5f43b01f8c3f-workload-certs\") pod \"maas-default-gateway-openshift-default-8f6f87f59-8fws9\" (UID: \"daac49a1-fcf0-4772-b555-5f43b01f8c3f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-8fws9" Oct 31 00:04:31.874341 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:31.874312 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-dpcqs" Oct 31 00:04:31.974564 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:31.974526 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/daac49a1-fcf0-4772-b555-5f43b01f8c3f-istio-data\") pod \"maas-default-gateway-openshift-default-8f6f87f59-8fws9\" (UID: \"daac49a1-fcf0-4772-b555-5f43b01f8c3f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-8fws9" Oct 31 00:04:31.982836 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:31.977107 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/daac49a1-fcf0-4772-b555-5f43b01f8c3f-istio-data\") pod \"maas-default-gateway-openshift-default-8f6f87f59-8fws9\" (UID: \"daac49a1-fcf0-4772-b555-5f43b01f8c3f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-8fws9" Oct 31 00:04:31.983656 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:31.983148 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/daac49a1-fcf0-4772-b555-5f43b01f8c3f-credential-socket\") pod \"maas-default-gateway-openshift-default-8f6f87f59-8fws9\" (UID: \"daac49a1-fcf0-4772-b555-5f43b01f8c3f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-8fws9" Oct 31 00:04:31.983779 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:31.983686 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbxjq\" (UniqueName: \"kubernetes.io/projected/daac49a1-fcf0-4772-b555-5f43b01f8c3f-kube-api-access-wbxjq\") pod \"maas-default-gateway-openshift-default-8f6f87f59-8fws9\" (UID: \"daac49a1-fcf0-4772-b555-5f43b01f8c3f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-8fws9" Oct 31 00:04:31.983779 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:31.983763 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/daac49a1-fcf0-4772-b555-5f43b01f8c3f-istio-token\") pod \"maas-default-gateway-openshift-default-8f6f87f59-8fws9\" (UID: \"daac49a1-fcf0-4772-b555-5f43b01f8c3f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-8fws9" Oct 31 00:04:31.983914 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:31.983554 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/daac49a1-fcf0-4772-b555-5f43b01f8c3f-credential-socket\") pod \"maas-default-gateway-openshift-default-8f6f87f59-8fws9\" (UID: \"daac49a1-fcf0-4772-b555-5f43b01f8c3f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-8fws9" Oct 31 00:04:31.988849 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:31.983818 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8328651d-33b5-4160-8f99-9d6de87ba69f-credential-socket\") pod \"maas-default-gateway-openshift-default-8f6f87f59-dpcqs\" (UID: \"8328651d-33b5-4160-8f99-9d6de87ba69f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-dpcqs" Oct 31 00:04:31.988849 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:31.986160 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8328651d-33b5-4160-8f99-9d6de87ba69f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8f6f87f59-dpcqs\" (UID: \"8328651d-33b5-4160-8f99-9d6de87ba69f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-dpcqs" Oct 31 00:04:31.988849 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:31.986226 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/daac49a1-fcf0-4772-b555-5f43b01f8c3f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8f6f87f59-8fws9\" (UID: \"daac49a1-fcf0-4772-b555-5f43b01f8c3f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-8fws9" Oct 31 00:04:31.988849 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:31.986305 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/daac49a1-fcf0-4772-b555-5f43b01f8c3f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8f6f87f59-8fws9\" (UID: \"daac49a1-fcf0-4772-b555-5f43b01f8c3f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-8fws9" Oct 31 00:04:31.988849 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:31.986370 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8328651d-33b5-4160-8f99-9d6de87ba69f-workload-certs\") pod \"maas-default-gateway-openshift-default-8f6f87f59-dpcqs\" (UID: \"8328651d-33b5-4160-8f99-9d6de87ba69f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-dpcqs" Oct 31 00:04:31.988849 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:31.986466 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8328651d-33b5-4160-8f99-9d6de87ba69f-workload-socket\") pod \"maas-default-gateway-openshift-default-8f6f87f59-dpcqs\" (UID: \"8328651d-33b5-4160-8f99-9d6de87ba69f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-dpcqs" Oct 31 00:04:31.988849 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:31.986519 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8328651d-33b5-4160-8f99-9d6de87ba69f-istio-envoy\") pod \"maas-default-gateway-openshift-default-8f6f87f59-dpcqs\" (UID: \"8328651d-33b5-4160-8f99-9d6de87ba69f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-dpcqs" Oct 31 00:04:31.988849 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:31.986553 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8328651d-33b5-4160-8f99-9d6de87ba69f-istio-token\") pod \"maas-default-gateway-openshift-default-8f6f87f59-dpcqs\" (UID: \"8328651d-33b5-4160-8f99-9d6de87ba69f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-dpcqs" Oct 31 00:04:31.988849 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:31.986628 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/daac49a1-fcf0-4772-b555-5f43b01f8c3f-istio-envoy\") pod \"maas-default-gateway-openshift-default-8f6f87f59-8fws9\" (UID: \"daac49a1-fcf0-4772-b555-5f43b01f8c3f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-8fws9" Oct 31 00:04:31.988849 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:31.987212 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/daac49a1-fcf0-4772-b555-5f43b01f8c3f-workload-socket\") pod \"maas-default-gateway-openshift-default-8f6f87f59-8fws9\" (UID: \"daac49a1-fcf0-4772-b555-5f43b01f8c3f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-8fws9" Oct 31 00:04:31.988849 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:31.987298 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/daac49a1-fcf0-4772-b555-5f43b01f8c3f-workload-certs\") pod \"maas-default-gateway-openshift-default-8f6f87f59-8fws9\" (UID: \"daac49a1-fcf0-4772-b555-5f43b01f8c3f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-8fws9" Oct 31 00:04:31.988849 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:31.987368 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4zhlt\" (UniqueName: \"kubernetes.io/projected/8328651d-33b5-4160-8f99-9d6de87ba69f-kube-api-access-4zhlt\") pod \"maas-default-gateway-openshift-default-8f6f87f59-dpcqs\" (UID: \"8328651d-33b5-4160-8f99-9d6de87ba69f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-dpcqs" Oct 31 00:04:31.988849 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:31.987413 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8328651d-33b5-4160-8f99-9d6de87ba69f-istio-data\") pod \"maas-default-gateway-openshift-default-8f6f87f59-dpcqs\" (UID: \"8328651d-33b5-4160-8f99-9d6de87ba69f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-dpcqs" Oct 31 00:04:31.988849 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:31.987449 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8328651d-33b5-4160-8f99-9d6de87ba69f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8f6f87f59-dpcqs\" (UID: \"8328651d-33b5-4160-8f99-9d6de87ba69f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-dpcqs" Oct 31 00:04:31.995272 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:31.995060 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/daac49a1-fcf0-4772-b555-5f43b01f8c3f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8f6f87f59-8fws9\" (UID: \"daac49a1-fcf0-4772-b555-5f43b01f8c3f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-8fws9" Oct 31 00:04:32.000518 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:31.996848 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/daac49a1-fcf0-4772-b555-5f43b01f8c3f-workload-socket\") pod \"maas-default-gateway-openshift-default-8f6f87f59-8fws9\" (UID: \"daac49a1-fcf0-4772-b555-5f43b01f8c3f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-8fws9" Oct 31 00:04:32.000518 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:31.998786 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/daac49a1-fcf0-4772-b555-5f43b01f8c3f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8f6f87f59-8fws9\" (UID: \"daac49a1-fcf0-4772-b555-5f43b01f8c3f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-8fws9" Oct 31 00:04:32.000518 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:31.999239 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/daac49a1-fcf0-4772-b555-5f43b01f8c3f-workload-certs\") pod \"maas-default-gateway-openshift-default-8f6f87f59-8fws9\" (UID: \"daac49a1-fcf0-4772-b555-5f43b01f8c3f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-8fws9" Oct 31 00:04:32.023770 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:32.015511 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-8fws9"] Oct 31 00:04:32.031495 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:32.031459 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/daac49a1-fcf0-4772-b555-5f43b01f8c3f-istio-envoy\") pod \"maas-default-gateway-openshift-default-8f6f87f59-8fws9\" (UID: \"daac49a1-fcf0-4772-b555-5f43b01f8c3f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-8fws9" Oct 31 00:04:32.042841 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:32.042806 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-dpcqs"] Oct 31 00:04:32.060662 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:32.060626 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wbxjq\" (UniqueName: \"kubernetes.io/projected/daac49a1-fcf0-4772-b555-5f43b01f8c3f-kube-api-access-wbxjq\") pod \"maas-default-gateway-openshift-default-8f6f87f59-8fws9\" (UID: \"daac49a1-fcf0-4772-b555-5f43b01f8c3f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-8fws9" Oct 31 00:04:32.060846 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:32.060824 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/daac49a1-fcf0-4772-b555-5f43b01f8c3f-istio-token\") pod \"maas-default-gateway-openshift-default-8f6f87f59-8fws9\" (UID: \"daac49a1-fcf0-4772-b555-5f43b01f8c3f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-8fws9" Oct 31 00:04:32.095898 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:32.095868 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8328651d-33b5-4160-8f99-9d6de87ba69f-credential-socket\") pod \"maas-default-gateway-openshift-default-8f6f87f59-dpcqs\" (UID: \"8328651d-33b5-4160-8f99-9d6de87ba69f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-dpcqs" Oct 31 00:04:32.096054 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:32.095917 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8328651d-33b5-4160-8f99-9d6de87ba69f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8f6f87f59-dpcqs\" (UID: \"8328651d-33b5-4160-8f99-9d6de87ba69f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-dpcqs" Oct 31 00:04:32.096054 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:32.095955 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8328651d-33b5-4160-8f99-9d6de87ba69f-workload-certs\") pod \"maas-default-gateway-openshift-default-8f6f87f59-dpcqs\" (UID: \"8328651d-33b5-4160-8f99-9d6de87ba69f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-dpcqs" Oct 31 00:04:32.096054 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:32.095995 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8328651d-33b5-4160-8f99-9d6de87ba69f-workload-socket\") pod \"maas-default-gateway-openshift-default-8f6f87f59-dpcqs\" (UID: \"8328651d-33b5-4160-8f99-9d6de87ba69f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-dpcqs" Oct 31 00:04:32.096054 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:32.096017 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8328651d-33b5-4160-8f99-9d6de87ba69f-istio-envoy\") pod \"maas-default-gateway-openshift-default-8f6f87f59-dpcqs\" (UID: \"8328651d-33b5-4160-8f99-9d6de87ba69f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-dpcqs" Oct 31 00:04:32.096054 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:32.096035 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8328651d-33b5-4160-8f99-9d6de87ba69f-istio-token\") pod \"maas-default-gateway-openshift-default-8f6f87f59-dpcqs\" (UID: \"8328651d-33b5-4160-8f99-9d6de87ba69f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-dpcqs" Oct 31 00:04:32.096397 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:32.096146 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zhlt\" (UniqueName: \"kubernetes.io/projected/8328651d-33b5-4160-8f99-9d6de87ba69f-kube-api-access-4zhlt\") pod \"maas-default-gateway-openshift-default-8f6f87f59-dpcqs\" (UID: \"8328651d-33b5-4160-8f99-9d6de87ba69f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-dpcqs" Oct 31 00:04:32.096397 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:32.096177 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8328651d-33b5-4160-8f99-9d6de87ba69f-istio-data\") pod \"maas-default-gateway-openshift-default-8f6f87f59-dpcqs\" (UID: \"8328651d-33b5-4160-8f99-9d6de87ba69f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-dpcqs" Oct 31 00:04:32.096397 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:32.096204 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8328651d-33b5-4160-8f99-9d6de87ba69f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8f6f87f59-dpcqs\" (UID: \"8328651d-33b5-4160-8f99-9d6de87ba69f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-dpcqs" Oct 31 00:04:32.096397 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:32.096379 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8328651d-33b5-4160-8f99-9d6de87ba69f-workload-certs\") pod \"maas-default-gateway-openshift-default-8f6f87f59-dpcqs\" (UID: \"8328651d-33b5-4160-8f99-9d6de87ba69f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-dpcqs" Oct 31 00:04:32.096600 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:32.096456 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8328651d-33b5-4160-8f99-9d6de87ba69f-workload-socket\") pod \"maas-default-gateway-openshift-default-8f6f87f59-dpcqs\" (UID: \"8328651d-33b5-4160-8f99-9d6de87ba69f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-dpcqs" Oct 31 00:04:32.096646 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:32.096621 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8328651d-33b5-4160-8f99-9d6de87ba69f-credential-socket\") pod \"maas-default-gateway-openshift-default-8f6f87f59-dpcqs\" (UID: \"8328651d-33b5-4160-8f99-9d6de87ba69f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-dpcqs" Oct 31 00:04:32.096875 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:32.096852 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8328651d-33b5-4160-8f99-9d6de87ba69f-istio-data\") pod \"maas-default-gateway-openshift-default-8f6f87f59-dpcqs\" (UID: \"8328651d-33b5-4160-8f99-9d6de87ba69f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-dpcqs" Oct 31 00:04:32.096943 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:32.096909 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8328651d-33b5-4160-8f99-9d6de87ba69f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8f6f87f59-dpcqs\" (UID: \"8328651d-33b5-4160-8f99-9d6de87ba69f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-dpcqs" Oct 31 00:04:32.098620 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:32.098596 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8328651d-33b5-4160-8f99-9d6de87ba69f-istio-envoy\") pod \"maas-default-gateway-openshift-default-8f6f87f59-dpcqs\" (UID: \"8328651d-33b5-4160-8f99-9d6de87ba69f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-dpcqs" Oct 31 00:04:32.098983 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:32.098952 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8328651d-33b5-4160-8f99-9d6de87ba69f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8f6f87f59-dpcqs\" (UID: \"8328651d-33b5-4160-8f99-9d6de87ba69f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-dpcqs" Oct 31 00:04:32.158544 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:32.158511 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8328651d-33b5-4160-8f99-9d6de87ba69f-istio-token\") pod \"maas-default-gateway-openshift-default-8f6f87f59-dpcqs\" (UID: \"8328651d-33b5-4160-8f99-9d6de87ba69f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-dpcqs" Oct 31 00:04:32.158701 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:32.158571 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zhlt\" (UniqueName: \"kubernetes.io/projected/8328651d-33b5-4160-8f99-9d6de87ba69f-kube-api-access-4zhlt\") pod \"maas-default-gateway-openshift-default-8f6f87f59-dpcqs\" (UID: \"8328651d-33b5-4160-8f99-9d6de87ba69f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-dpcqs" Oct 31 00:04:32.173761 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:32.173728 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-8fws9" Oct 31 00:04:32.191320 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:32.191298 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-dpcqs" Oct 31 00:04:32.616701 ip-10-0-141-180 kubenswrapper[4927]: W1031 00:04:32.616667 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddaac49a1_fcf0_4772_b555_5f43b01f8c3f.slice/crio-7094a9598ef2afef35c2335160d4eee0bd375e4ac40f7322204bce0ec654f0df WatchSource:0}: Error finding container 7094a9598ef2afef35c2335160d4eee0bd375e4ac40f7322204bce0ec654f0df: Status 404 returned error can't find the container with id 7094a9598ef2afef35c2335160d4eee0bd375e4ac40f7322204bce0ec654f0df Oct 31 00:04:32.617594 ip-10-0-141-180 kubenswrapper[4927]: W1031 00:04:32.617570 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8328651d_33b5_4160_8f99_9d6de87ba69f.slice/crio-ebffea387d264ae2137ca37fe1e4702608376ff1b3a84e25f9ae6059cc853d1c WatchSource:0}: Error finding container ebffea387d264ae2137ca37fe1e4702608376ff1b3a84e25f9ae6059cc853d1c: Status 404 returned error can't find the container with id ebffea387d264ae2137ca37fe1e4702608376ff1b3a84e25f9ae6059cc853d1c Oct 31 00:04:32.619072 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:32.619047 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-8fws9"] Oct 31 00:04:32.619560 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:32.619527 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:04:32.619615 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:32.619594 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:04:32.619670 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:32.619623 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:04:32.620403 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:32.620370 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:04:32.620462 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:32.620447 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:04:32.620503 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:32.620488 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:04:32.626190 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:32.626168 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-dpcqs"] Oct 31 00:04:33.180503 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:33.180325 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-dpcqs" event={"ID":"8328651d-33b5-4160-8f99-9d6de87ba69f","Type":"ContainerStarted","Data":"08fc703beb7ed599fb280382f6d36d5ac0040a3441246ecb40098cb8c7e9cbe2"} Oct 31 00:04:33.181202 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:33.181177 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-dpcqs" event={"ID":"8328651d-33b5-4160-8f99-9d6de87ba69f","Type":"ContainerStarted","Data":"ebffea387d264ae2137ca37fe1e4702608376ff1b3a84e25f9ae6059cc853d1c"} Oct 31 00:04:33.183369 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:33.183311 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-8fws9" event={"ID":"daac49a1-fcf0-4772-b555-5f43b01f8c3f","Type":"ContainerStarted","Data":"cbb778d1425557c3b18f70209e5b26a2f2daaf067005fd468bb053b35672a38b"} Oct 31 00:04:33.183369 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:33.183368 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-8fws9" event={"ID":"daac49a1-fcf0-4772-b555-5f43b01f8c3f","Type":"ContainerStarted","Data":"7094a9598ef2afef35c2335160d4eee0bd375e4ac40f7322204bce0ec654f0df"} Oct 31 00:04:33.191542 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:33.191506 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-dpcqs" Oct 31 00:04:33.193205 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:33.193175 4927 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-dpcqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" start-of-body= Oct 31 00:04:33.193322 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:33.193224 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-dpcqs" podUID="8328651d-33b5-4160-8f99-9d6de87ba69f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" Oct 31 00:04:33.312023 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:33.311966 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-dpcqs" podStartSLOduration=2.311945891 podStartE2EDuration="2.311945891s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:04:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:04:33.30464614 +0000 UTC m=+1386.340976974" watchObservedRunningTime="2026-10-31 00:04:33.311945891 +0000 UTC m=+1386.348276715" Oct 31 00:04:33.457858 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:33.457728 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-8fws9" podStartSLOduration=2.457706988 podStartE2EDuration="2.457706988s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:04:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:04:33.450307153 +0000 UTC m=+1386.486637977" watchObservedRunningTime="2026-10-31 00:04:33.457706988 +0000 UTC m=+1386.494037811" Oct 31 00:04:34.174449 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:34.174414 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-8fws9" Oct 31 00:04:34.175179 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:34.175140 4927 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-8fws9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body= Oct 31 00:04:34.175226 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:34.175201 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-8fws9" podUID="daac49a1-fcf0-4772-b555-5f43b01f8c3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" Oct 31 00:04:34.191759 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:34.191726 4927 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-dpcqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" start-of-body= Oct 31 00:04:34.192138 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:34.191781 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-dpcqs" podUID="8328651d-33b5-4160-8f99-9d6de87ba69f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" Oct 31 00:04:34.470879 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:34.469055 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:04:34.673354 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:34.665062 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:04:34.842977 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:34.842881 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:04:35.174839 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:35.174790 4927 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-8fws9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body= Oct 31 00:04:35.175046 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:35.174854 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-8fws9" podUID="daac49a1-fcf0-4772-b555-5f43b01f8c3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" Oct 31 00:04:35.191945 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:35.191910 4927 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-dpcqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" start-of-body= Oct 31 00:04:35.192590 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:35.192533 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-dpcqs" podUID="8328651d-33b5-4160-8f99-9d6de87ba69f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" Oct 31 00:04:36.174387 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:36.174277 4927 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-8fws9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body= Oct 31 00:04:36.174387 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:36.174370 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-8fws9" podUID="daac49a1-fcf0-4772-b555-5f43b01f8c3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" Oct 31 00:04:36.192321 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:36.192289 4927 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-dpcqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" start-of-body= Oct 31 00:04:36.192725 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:36.192380 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-dpcqs" podUID="8328651d-33b5-4160-8f99-9d6de87ba69f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" Oct 31 00:04:37.174134 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:37.174058 4927 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-8fws9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body= Oct 31 00:04:37.174361 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:37.174314 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-8fws9" podUID="daac49a1-fcf0-4772-b555-5f43b01f8c3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" Oct 31 00:04:37.192679 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:37.192639 4927 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-dpcqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" start-of-body= Oct 31 00:04:37.193071 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:37.192704 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-dpcqs" podUID="8328651d-33b5-4160-8f99-9d6de87ba69f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" Oct 31 00:04:38.174877 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:38.174549 4927 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-8fws9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body= Oct 31 00:04:38.174877 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:38.174608 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-8fws9" podUID="daac49a1-fcf0-4772-b555-5f43b01f8c3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" Oct 31 00:04:38.192375 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:38.192313 4927 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-dpcqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" start-of-body= Oct 31 00:04:38.192518 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:38.192383 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-dpcqs" podUID="8328651d-33b5-4160-8f99-9d6de87ba69f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" Oct 31 00:04:39.175134 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:39.175095 4927 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-8fws9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body= Oct 31 00:04:39.175599 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:39.175190 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-8fws9" podUID="daac49a1-fcf0-4772-b555-5f43b01f8c3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" Oct 31 00:04:39.191907 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:39.191878 4927 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-dpcqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" start-of-body= Oct 31 00:04:39.192070 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:39.191923 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-dpcqs" podUID="8328651d-33b5-4160-8f99-9d6de87ba69f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" Oct 31 00:04:39.609153 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:39.609047 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:04:40.174169 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:40.174127 4927 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-8fws9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body= Oct 31 00:04:40.174402 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:40.174198 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-8fws9" podUID="daac49a1-fcf0-4772-b555-5f43b01f8c3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" Oct 31 00:04:40.192574 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:40.192537 4927 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-dpcqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" start-of-body= Oct 31 00:04:40.192950 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:40.192602 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-dpcqs" podUID="8328651d-33b5-4160-8f99-9d6de87ba69f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" Oct 31 00:04:41.175113 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:41.175075 4927 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-8fws9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body= Oct 31 00:04:41.175464 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:41.175425 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-8fws9" podUID="daac49a1-fcf0-4772-b555-5f43b01f8c3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" Oct 31 00:04:41.192701 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:41.192666 4927 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-dpcqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" start-of-body= Oct 31 00:04:41.193182 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:41.192743 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-dpcqs" podUID="8328651d-33b5-4160-8f99-9d6de87ba69f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" Oct 31 00:04:42.459931 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:42.459893 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-dpcqs" Oct 31 00:04:42.460623 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:42.460598 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-dpcqs" Oct 31 00:04:42.461614 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:42.461589 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-dpcqs" Oct 31 00:04:42.583193 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:42.583139 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-8fws9" Oct 31 00:04:42.583727 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:42.583700 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-8fws9" Oct 31 00:04:42.584982 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:42.584959 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-8fws9" Oct 31 00:04:44.328005 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:44.327959 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:04:48.887007 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:48.886974 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:04:49.679732 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:04:49.679693 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sqv69" containerName="extractor" Oct 31 00:04:54.134357 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:54.084444 4927 patch_prober.go:29] interesting pod/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-d44pq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:04:54.134357 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:54.084541 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-d44pq" podUID="f08dd946-14b1-4546-8fae-01564f3c6908" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:04:54.134357 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:54.131433 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-27e82b/distinct-w2-27e82b-kserve-75f475fcd9-mccsz" podUID="4ba8ced4-e93f-46ac-87c5-3d0b7fadc02d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:04:54.221520 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:54.221463 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-986d37/premium-w3-986d37-kserve-5c7c7757fb-r99j7" podUID="f4e38df7-355f-4b47-a0b5-7e0b907f3326" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:04:54.281364 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:54.258536 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5c3374/premium-w1-5c3374-kserve-6c56d4dcd-llwcl" podUID="dc813f02-e077-4251-8c1e-fea0986b0f36" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded" Oct 31 00:04:54.287376 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:54.281926 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-35a187/distinct-2-w6-35a187-kserve-7ff4b6dc55-944g6" podUID="429ee092-42d8-475e-b0c6-b5a9930766fc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.113:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:04:54.326504 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:54.326455 4927 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-986d37/premium-w3-986d37-kserve-5c7c7757fb-r99j7" podUID="f4e38df7-355f-4b47-a0b5-7e0b907f3326" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/health\": context deadline exceeded" Oct 31 00:04:54.328503 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:54.328219 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-46b586/simulator-w0-46b586-kserve-67588b8dbd-z8xmn" podUID="9a25e252-223d-41f2-abf7-632067c77e0b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:04:54.424837 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:54.424791 4927 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-35a187/distinct-2-w6-35a187-kserve-7ff4b6dc55-944g6" podUID="429ee092-42d8-475e-b0c6-b5a9930766fc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.113:8000/health\": context deadline exceeded" Oct 31 00:04:54.497364 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:54.489665 4927 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-fe01fa/distinct-2-w5-fe01fa-kserve-79b99966d8-dkhzd" podUID="d5c1c9e8-ce47-4868-860d-b8413ff74ab5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.111:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:04:55.133353 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:55.132421 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-27e82b/distinct-w2-27e82b-kserve-75f475fcd9-mccsz" podUID="4ba8ced4-e93f-46ac-87c5-3d0b7fadc02d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:04:55.173474 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:55.164476 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-27e82b/distinct-w2-27e82b-kserve-75f475fcd9-mccsz" Oct 31 00:04:55.227355 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:55.213843 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-27e82b/distinct-w2-27e82b-kserve-75f475fcd9-mccsz" Oct 31 00:04:55.241547 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:55.241505 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-986d37/premium-w3-986d37-kserve-5c7c7757fb-r99j7" podUID="f4e38df7-355f-4b47-a0b5-7e0b907f3326" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:04:55.241838 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:55.241816 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-986d37/premium-w3-986d37-kserve-5c7c7757fb-r99j7" Oct 31 00:04:55.265501 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:55.265466 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:04:55.491477 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:55.488295 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:04:55.635358 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:55.633934 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-986d37/premium-w3-986d37-kserve-5c7c7757fb-r99j7" Oct 31 00:04:56.764390 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:56.764352 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-8e7d0098-openshift-default-7c65b994b7-tc65h"] Oct 31 00:04:56.771555 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:56.771529 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-8e7d0098-openshift-default-7c65b994b7-tc65h" Oct 31 00:04:56.791851 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:56.790605 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d357695b-ec8c-435b-aeea-35e7409bfdd5-istio-token\") pod \"e2e-ait-8e7d0098-openshift-default-7c65b994b7-tc65h\" (UID: \"d357695b-ec8c-435b-aeea-35e7409bfdd5\") " pod="openshift-ingress/e2e-ait-8e7d0098-openshift-default-7c65b994b7-tc65h" Oct 31 00:04:56.791851 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:56.790676 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d357695b-ec8c-435b-aeea-35e7409bfdd5-credential-socket\") pod \"e2e-ait-8e7d0098-openshift-default-7c65b994b7-tc65h\" (UID: \"d357695b-ec8c-435b-aeea-35e7409bfdd5\") " pod="openshift-ingress/e2e-ait-8e7d0098-openshift-default-7c65b994b7-tc65h" Oct 31 00:04:56.791851 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:56.790827 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d357695b-ec8c-435b-aeea-35e7409bfdd5-workload-socket\") pod \"e2e-ait-8e7d0098-openshift-default-7c65b994b7-tc65h\" (UID: \"d357695b-ec8c-435b-aeea-35e7409bfdd5\") " pod="openshift-ingress/e2e-ait-8e7d0098-openshift-default-7c65b994b7-tc65h" Oct 31 00:04:56.791851 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:56.790888 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d357695b-ec8c-435b-aeea-35e7409bfdd5-istio-envoy\") pod \"e2e-ait-8e7d0098-openshift-default-7c65b994b7-tc65h\" (UID: \"d357695b-ec8c-435b-aeea-35e7409bfdd5\") " pod="openshift-ingress/e2e-ait-8e7d0098-openshift-default-7c65b994b7-tc65h" Oct 31 00:04:56.791851 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:56.790991 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d357695b-ec8c-435b-aeea-35e7409bfdd5-istio-data\") pod \"e2e-ait-8e7d0098-openshift-default-7c65b994b7-tc65h\" (UID: \"d357695b-ec8c-435b-aeea-35e7409bfdd5\") " pod="openshift-ingress/e2e-ait-8e7d0098-openshift-default-7c65b994b7-tc65h" Oct 31 00:04:56.791851 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:56.791036 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d357695b-ec8c-435b-aeea-35e7409bfdd5-istio-podinfo\") pod \"e2e-ait-8e7d0098-openshift-default-7c65b994b7-tc65h\" (UID: \"d357695b-ec8c-435b-aeea-35e7409bfdd5\") " pod="openshift-ingress/e2e-ait-8e7d0098-openshift-default-7c65b994b7-tc65h" Oct 31 00:04:56.791851 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:56.791063 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6j9g8\" (UniqueName: \"kubernetes.io/projected/d357695b-ec8c-435b-aeea-35e7409bfdd5-kube-api-access-6j9g8\") pod \"e2e-ait-8e7d0098-openshift-default-7c65b994b7-tc65h\" (UID: \"d357695b-ec8c-435b-aeea-35e7409bfdd5\") " pod="openshift-ingress/e2e-ait-8e7d0098-openshift-default-7c65b994b7-tc65h" Oct 31 00:04:56.791851 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:56.791136 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d357695b-ec8c-435b-aeea-35e7409bfdd5-workload-certs\") pod \"e2e-ait-8e7d0098-openshift-default-7c65b994b7-tc65h\" (UID: \"d357695b-ec8c-435b-aeea-35e7409bfdd5\") " pod="openshift-ingress/e2e-ait-8e7d0098-openshift-default-7c65b994b7-tc65h" Oct 31 00:04:56.791851 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:56.791190 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d357695b-ec8c-435b-aeea-35e7409bfdd5-istiod-ca-cert\") pod \"e2e-ait-8e7d0098-openshift-default-7c65b994b7-tc65h\" (UID: \"d357695b-ec8c-435b-aeea-35e7409bfdd5\") " pod="openshift-ingress/e2e-ait-8e7d0098-openshift-default-7c65b994b7-tc65h" Oct 31 00:04:56.839357 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:56.829680 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-8e7d0098-openshift-default-dockercfg-5jg95\"" Oct 31 00:04:56.906247 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:56.906207 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d357695b-ec8c-435b-aeea-35e7409bfdd5-istio-data\") pod \"e2e-ait-8e7d0098-openshift-default-7c65b994b7-tc65h\" (UID: \"d357695b-ec8c-435b-aeea-35e7409bfdd5\") " pod="openshift-ingress/e2e-ait-8e7d0098-openshift-default-7c65b994b7-tc65h" Oct 31 00:04:56.907029 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:56.907003 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d357695b-ec8c-435b-aeea-35e7409bfdd5-istio-podinfo\") pod \"e2e-ait-8e7d0098-openshift-default-7c65b994b7-tc65h\" (UID: \"d357695b-ec8c-435b-aeea-35e7409bfdd5\") " pod="openshift-ingress/e2e-ait-8e7d0098-openshift-default-7c65b994b7-tc65h" Oct 31 00:04:56.922254 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:56.922221 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6j9g8\" (UniqueName: \"kubernetes.io/projected/d357695b-ec8c-435b-aeea-35e7409bfdd5-kube-api-access-6j9g8\") pod \"e2e-ait-8e7d0098-openshift-default-7c65b994b7-tc65h\" (UID: \"d357695b-ec8c-435b-aeea-35e7409bfdd5\") " pod="openshift-ingress/e2e-ait-8e7d0098-openshift-default-7c65b994b7-tc65h" Oct 31 00:04:56.922663 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:56.922641 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d357695b-ec8c-435b-aeea-35e7409bfdd5-workload-certs\") pod \"e2e-ait-8e7d0098-openshift-default-7c65b994b7-tc65h\" (UID: \"d357695b-ec8c-435b-aeea-35e7409bfdd5\") " pod="openshift-ingress/e2e-ait-8e7d0098-openshift-default-7c65b994b7-tc65h" Oct 31 00:04:56.922824 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:56.922807 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d357695b-ec8c-435b-aeea-35e7409bfdd5-istiod-ca-cert\") pod \"e2e-ait-8e7d0098-openshift-default-7c65b994b7-tc65h\" (UID: \"d357695b-ec8c-435b-aeea-35e7409bfdd5\") " pod="openshift-ingress/e2e-ait-8e7d0098-openshift-default-7c65b994b7-tc65h" Oct 31 00:04:56.923065 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:56.923050 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d357695b-ec8c-435b-aeea-35e7409bfdd5-istio-token\") pod \"e2e-ait-8e7d0098-openshift-default-7c65b994b7-tc65h\" (UID: \"d357695b-ec8c-435b-aeea-35e7409bfdd5\") " pod="openshift-ingress/e2e-ait-8e7d0098-openshift-default-7c65b994b7-tc65h" Oct 31 00:04:56.923181 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:56.923168 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d357695b-ec8c-435b-aeea-35e7409bfdd5-credential-socket\") pod \"e2e-ait-8e7d0098-openshift-default-7c65b994b7-tc65h\" (UID: \"d357695b-ec8c-435b-aeea-35e7409bfdd5\") " pod="openshift-ingress/e2e-ait-8e7d0098-openshift-default-7c65b994b7-tc65h" Oct 31 00:04:56.923478 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:56.923456 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d357695b-ec8c-435b-aeea-35e7409bfdd5-workload-socket\") pod \"e2e-ait-8e7d0098-openshift-default-7c65b994b7-tc65h\" (UID: \"d357695b-ec8c-435b-aeea-35e7409bfdd5\") " pod="openshift-ingress/e2e-ait-8e7d0098-openshift-default-7c65b994b7-tc65h" Oct 31 00:04:56.957721 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:56.957668 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d357695b-ec8c-435b-aeea-35e7409bfdd5-istio-envoy\") pod \"e2e-ait-8e7d0098-openshift-default-7c65b994b7-tc65h\" (UID: \"d357695b-ec8c-435b-aeea-35e7409bfdd5\") " pod="openshift-ingress/e2e-ait-8e7d0098-openshift-default-7c65b994b7-tc65h" Oct 31 00:04:56.958180 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:56.906925 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d357695b-ec8c-435b-aeea-35e7409bfdd5-istio-data\") pod \"e2e-ait-8e7d0098-openshift-default-7c65b994b7-tc65h\" (UID: \"d357695b-ec8c-435b-aeea-35e7409bfdd5\") " pod="openshift-ingress/e2e-ait-8e7d0098-openshift-default-7c65b994b7-tc65h" Oct 31 00:04:56.958367 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:56.928186 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d357695b-ec8c-435b-aeea-35e7409bfdd5-credential-socket\") pod \"e2e-ait-8e7d0098-openshift-default-7c65b994b7-tc65h\" (UID: \"d357695b-ec8c-435b-aeea-35e7409bfdd5\") " pod="openshift-ingress/e2e-ait-8e7d0098-openshift-default-7c65b994b7-tc65h" Oct 31 00:04:56.958474 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:56.934714 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d357695b-ec8c-435b-aeea-35e7409bfdd5-workload-socket\") pod \"e2e-ait-8e7d0098-openshift-default-7c65b994b7-tc65h\" (UID: \"d357695b-ec8c-435b-aeea-35e7409bfdd5\") " pod="openshift-ingress/e2e-ait-8e7d0098-openshift-default-7c65b994b7-tc65h" Oct 31 00:04:56.958571 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:56.935091 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d357695b-ec8c-435b-aeea-35e7409bfdd5-istiod-ca-cert\") pod \"e2e-ait-8e7d0098-openshift-default-7c65b994b7-tc65h\" (UID: \"d357695b-ec8c-435b-aeea-35e7409bfdd5\") " pod="openshift-ingress/e2e-ait-8e7d0098-openshift-default-7c65b994b7-tc65h" Oct 31 00:04:56.958685 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:56.926857 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d357695b-ec8c-435b-aeea-35e7409bfdd5-workload-certs\") pod \"e2e-ait-8e7d0098-openshift-default-7c65b994b7-tc65h\" (UID: \"d357695b-ec8c-435b-aeea-35e7409bfdd5\") " pod="openshift-ingress/e2e-ait-8e7d0098-openshift-default-7c65b994b7-tc65h" Oct 31 00:04:57.014811 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:57.014722 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d357695b-ec8c-435b-aeea-35e7409bfdd5-istio-envoy\") pod \"e2e-ait-8e7d0098-openshift-default-7c65b994b7-tc65h\" (UID: \"d357695b-ec8c-435b-aeea-35e7409bfdd5\") " pod="openshift-ingress/e2e-ait-8e7d0098-openshift-default-7c65b994b7-tc65h" Oct 31 00:04:57.270216 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:57.082918 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d357695b-ec8c-435b-aeea-35e7409bfdd5-istio-podinfo\") pod \"e2e-ait-8e7d0098-openshift-default-7c65b994b7-tc65h\" (UID: \"d357695b-ec8c-435b-aeea-35e7409bfdd5\") " pod="openshift-ingress/e2e-ait-8e7d0098-openshift-default-7c65b994b7-tc65h" Oct 31 00:04:57.270216 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:57.206089 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-8e7d0098-openshift-default-7c65b994b7-tc65h"] Oct 31 00:04:57.270216 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:57.265940 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6j9g8\" (UniqueName: \"kubernetes.io/projected/d357695b-ec8c-435b-aeea-35e7409bfdd5-kube-api-access-6j9g8\") pod \"e2e-ait-8e7d0098-openshift-default-7c65b994b7-tc65h\" (UID: \"d357695b-ec8c-435b-aeea-35e7409bfdd5\") " pod="openshift-ingress/e2e-ait-8e7d0098-openshift-default-7c65b994b7-tc65h" Oct 31 00:04:57.273480 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:57.273451 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d357695b-ec8c-435b-aeea-35e7409bfdd5-istio-token\") pod \"e2e-ait-8e7d0098-openshift-default-7c65b994b7-tc65h\" (UID: \"d357695b-ec8c-435b-aeea-35e7409bfdd5\") " pod="openshift-ingress/e2e-ait-8e7d0098-openshift-default-7c65b994b7-tc65h" Oct 31 00:04:57.398024 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:57.397986 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-8e7d0098-openshift-default-7c65b994b7-tc65h" Oct 31 00:04:57.882076 ip-10-0-141-180 kubenswrapper[4927]: W1031 00:04:57.882037 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd357695b_ec8c_435b_aeea_35e7409bfdd5.slice/crio-4cddccd1fc7515bef4fdfb8f591efc653d0beba2ded7682fdf1356d3be8ae674 WatchSource:0}: Error finding container 4cddccd1fc7515bef4fdfb8f591efc653d0beba2ded7682fdf1356d3be8ae674: Status 404 returned error can't find the container with id 4cddccd1fc7515bef4fdfb8f591efc653d0beba2ded7682fdf1356d3be8ae674 Oct 31 00:04:57.885425 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:57.885384 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:04:57.885512 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:57.885434 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-8e7d0098-openshift-default-7c65b994b7-tc65h"] Oct 31 00:04:57.885512 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:57.885485 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:04:57.885579 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:57.885523 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:04:58.288049 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:58.288004 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8e7d0098-openshift-default-7c65b994b7-tc65h" event={"ID":"d357695b-ec8c-435b-aeea-35e7409bfdd5","Type":"ContainerStarted","Data":"6c4e481c3c0e8f8e4d6055719ee6fac0544e963ac2f1ff1fd355da3788b367bc"} Oct 31 00:04:58.288212 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:58.288068 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8e7d0098-openshift-default-7c65b994b7-tc65h" event={"ID":"d357695b-ec8c-435b-aeea-35e7409bfdd5","Type":"ContainerStarted","Data":"4cddccd1fc7515bef4fdfb8f591efc653d0beba2ded7682fdf1356d3be8ae674"} Oct 31 00:04:58.399067 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:58.399029 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-8e7d0098-openshift-default-7c65b994b7-tc65h" Oct 31 00:04:58.400228 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:58.400197 4927 patch_prober.go:29] interesting pod/e2e-ait-8e7d0098-openshift-default-7c65b994b7-tc65h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body= Oct 31 00:04:58.400402 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:58.400245 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-8e7d0098-openshift-default-7c65b994b7-tc65h" podUID="d357695b-ec8c-435b-aeea-35e7409bfdd5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" Oct 31 00:04:58.448786 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:58.448704 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-8e7d0098-openshift-default-7c65b994b7-tc65h" podStartSLOduration=2.4486827079999998 podStartE2EDuration="2.448682708s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:04:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:04:58.427045385 +0000 UTC m=+1411.463376208" watchObservedRunningTime="2026-10-31 00:04:58.448682708 +0000 UTC m=+1411.485013531" Oct 31 00:04:58.949415 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:58.949374 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:04:59.398422 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:59.398303 4927 patch_prober.go:29] interesting pod/e2e-ait-8e7d0098-openshift-default-7c65b994b7-tc65h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body= Oct 31 00:04:59.398646 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:04:59.398619 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-8e7d0098-openshift-default-7c65b994b7-tc65h" podUID="d357695b-ec8c-435b-aeea-35e7409bfdd5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" Oct 31 00:05:00.398754 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:05:00.398722 4927 patch_prober.go:29] interesting pod/e2e-ait-8e7d0098-openshift-default-7c65b994b7-tc65h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body= Oct 31 00:05:00.399145 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:05:00.398790 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-8e7d0098-openshift-default-7c65b994b7-tc65h" podUID="d357695b-ec8c-435b-aeea-35e7409bfdd5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" Oct 31 00:05:01.400547 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:05:01.398316 4927 patch_prober.go:29] interesting pod/e2e-ait-8e7d0098-openshift-default-7c65b994b7-tc65h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body= Oct 31 00:05:01.400547 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:05:01.398396 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-8e7d0098-openshift-default-7c65b994b7-tc65h" podUID="d357695b-ec8c-435b-aeea-35e7409bfdd5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" Oct 31 00:05:02.399047 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:05:02.399009 4927 patch_prober.go:29] interesting pod/e2e-ait-8e7d0098-openshift-default-7c65b994b7-tc65h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body= Oct 31 00:05:02.399230 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:05:02.399079 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-8e7d0098-openshift-default-7c65b994b7-tc65h" podUID="d357695b-ec8c-435b-aeea-35e7409bfdd5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" Oct 31 00:05:03.398974 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:05:03.398931 4927 patch_prober.go:29] interesting pod/e2e-ait-8e7d0098-openshift-default-7c65b994b7-tc65h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body= Oct 31 00:05:03.399507 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:05:03.399006 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-8e7d0098-openshift-default-7c65b994b7-tc65h" podUID="d357695b-ec8c-435b-aeea-35e7409bfdd5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" Oct 31 00:05:03.706808 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:05:03.706763 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:05:04.039406 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:05:04.037411 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:05:04.173742 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:05:04.173707 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f77eb00c-hprr6"] Oct 31 00:05:04.199213 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:05:04.199183 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f77eb00c-hprr6" Oct 31 00:05:04.263665 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:05:04.262949 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f77eb00c-hprr6"] Oct 31 00:05:04.282432 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:05:04.280244 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:05:04.282432 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:05:04.280425 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4fdzs\"" Oct 31 00:05:04.297713 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:05:04.297624 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/541d0cfd-a005-480d-84e7-a345701a23a1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f77eb00c-hprr6\" (UID: \"541d0cfd-a005-480d-84e7-a345701a23a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f77eb00c-hprr6" Oct 31 00:05:04.297713 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:05:04.297686 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vspth\" (UniqueName: \"kubernetes.io/projected/541d0cfd-a005-480d-84e7-a345701a23a1-kube-api-access-vspth\") pod \"maas-api-revoke-sub-f77eb00c-hprr6\" (UID: \"541d0cfd-a005-480d-84e7-a345701a23a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f77eb00c-hprr6" Oct 31 00:05:04.297941 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:05:04.297783 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/541d0cfd-a005-480d-84e7-a345701a23a1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f77eb00c-hprr6\" (UID: \"541d0cfd-a005-480d-84e7-a345701a23a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f77eb00c-hprr6" Oct 31 00:05:04.398731 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:05:04.398692 4927 patch_prober.go:29] interesting pod/e2e-ait-8e7d0098-openshift-default-7c65b994b7-tc65h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body= Oct 31 00:05:04.398944 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:05:04.398762 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-8e7d0098-openshift-default-7c65b994b7-tc65h" podUID="d357695b-ec8c-435b-aeea-35e7409bfdd5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" Oct 31 00:05:04.399396 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:05:04.399367 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/541d0cfd-a005-480d-84e7-a345701a23a1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f77eb00c-hprr6\" (UID: \"541d0cfd-a005-480d-84e7-a345701a23a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f77eb00c-hprr6" Oct 31 00:05:04.399784 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:05:04.399415 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vspth\" (UniqueName: \"kubernetes.io/projected/541d0cfd-a005-480d-84e7-a345701a23a1-kube-api-access-vspth\") pod \"maas-api-revoke-sub-f77eb00c-hprr6\" (UID: \"541d0cfd-a005-480d-84e7-a345701a23a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f77eb00c-hprr6" Oct 31 00:05:04.399784 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:05:04.399443 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/541d0cfd-a005-480d-84e7-a345701a23a1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f77eb00c-hprr6\" (UID: \"541d0cfd-a005-480d-84e7-a345701a23a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f77eb00c-hprr6" Oct 31 00:05:04.400197 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:05:04.400171 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/541d0cfd-a005-480d-84e7-a345701a23a1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f77eb00c-hprr6\" (UID: \"541d0cfd-a005-480d-84e7-a345701a23a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f77eb00c-hprr6" Oct 31 00:05:04.401883 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:05:04.401861 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/541d0cfd-a005-480d-84e7-a345701a23a1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f77eb00c-hprr6\" (UID: \"541d0cfd-a005-480d-84e7-a345701a23a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f77eb00c-hprr6" Oct 31 00:05:04.465439 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:05:04.465404 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vspth\" (UniqueName: \"kubernetes.io/projected/541d0cfd-a005-480d-84e7-a345701a23a1-kube-api-access-vspth\") pod \"maas-api-revoke-sub-f77eb00c-hprr6\" (UID: \"541d0cfd-a005-480d-84e7-a345701a23a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f77eb00c-hprr6" Oct 31 00:05:04.578534 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:05:04.578453 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f77eb00c-hprr6" Oct 31 00:05:05.134827 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:05:05.133819 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f77eb00c-hprr6"] Oct 31 00:05:05.403570 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:05:05.403502 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-8e7d0098-openshift-default-7c65b994b7-tc65h" Oct 31 00:05:05.403978 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:05:05.403960 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-8e7d0098-openshift-default-7c65b994b7-tc65h" Oct 31 00:05:05.404860 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:05:05.404833 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-8e7d0098-openshift-default-7c65b994b7-tc65h" Oct 31 00:05:05.470617 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:05:05.470576 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f77eb00c-hprr6" event={"ID":"541d0cfd-a005-480d-84e7-a345701a23a1","Type":"ContainerStarted","Data":"b90bb060d8f960423e54bc5f4f0198069fd40b026d233d7509d78b09933050cf"} Oct 31 00:05:06.483598 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:05:06.483566 4927 generic.go:356] "Generic (PLEG): container finished" podID="541d0cfd-a005-480d-84e7-a345701a23a1" containerID="e81e24db8cd912deac0573ac2b43848a00c994307a2cb1d92d30ffc0f9d80dd3" exitCode=0 Oct 31 00:05:06.484116 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:05:06.483642 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f77eb00c-hprr6" event={"ID":"541d0cfd-a005-480d-84e7-a345701a23a1","Type":"ContainerDied","Data":"e81e24db8cd912deac0573ac2b43848a00c994307a2cb1d92d30ffc0f9d80dd3"} Oct 31 00:05:07.494738 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:05:07.494704 4927 generic.go:356] "Generic (PLEG): container finished" podID="d310a62f-9f54-4000-b280-0581fd91192f" containerID="ae5b79a50331596a3b0abc27a755878cbe38e756b0abf2963eab10d18de553a0" exitCode=0 Oct 31 00:05:07.495312 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:05:07.494784 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0d96587a-openshift-default-bcfbd49cc-xrrp4" event={"ID":"d310a62f-9f54-4000-b280-0581fd91192f","Type":"ContainerDied","Data":"ae5b79a50331596a3b0abc27a755878cbe38e756b0abf2963eab10d18de553a0"} Oct 31 00:05:07.495829 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:05:07.495443 4927 scope.go:122] "RemoveContainer" containerID="ae5b79a50331596a3b0abc27a755878cbe38e756b0abf2963eab10d18de553a0" Oct 31 00:05:07.497180 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:05:07.497147 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:05:07.497259 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:05:07.497223 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:05:07.497297 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:05:07.497258 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:05:07.712521 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:05:07.712486 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f77eb00c-hprr6" Oct 31 00:05:07.863842 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:05:07.863735 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/541d0cfd-a005-480d-84e7-a345701a23a1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/541d0cfd-a005-480d-84e7-a345701a23a1-maas-api-openshift-ca\") pod \"541d0cfd-a005-480d-84e7-a345701a23a1\" (UID: \"541d0cfd-a005-480d-84e7-a345701a23a1\") " Oct 31 00:05:07.864050 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:05:07.863909 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/541d0cfd-a005-480d-84e7-a345701a23a1-kube-api-access-vspth\" (UniqueName: \"kubernetes.io/projected/541d0cfd-a005-480d-84e7-a345701a23a1-kube-api-access-vspth\") pod \"541d0cfd-a005-480d-84e7-a345701a23a1\" (UID: \"541d0cfd-a005-480d-84e7-a345701a23a1\") " Oct 31 00:05:07.864176 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:05:07.864146 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/541d0cfd-a005-480d-84e7-a345701a23a1-maas-api-openshift-ca" pod "541d0cfd-a005-480d-84e7-a345701a23a1" (UID: "541d0cfd-a005-480d-84e7-a345701a23a1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:05:07.864308 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:05:07.864286 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/541d0cfd-a005-480d-84e7-a345701a23a1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/541d0cfd-a005-480d-84e7-a345701a23a1-maas-api-xks-ca\") pod \"541d0cfd-a005-480d-84e7-a345701a23a1\" (UID: \"541d0cfd-a005-480d-84e7-a345701a23a1\") " Oct 31 00:05:07.865104 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:05:07.865067 4927 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/541d0cfd-a005-480d-84e7-a345701a23a1-maas-api-openshift-ca\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:05:07.866488 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:05:07.866463 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/541d0cfd-a005-480d-84e7-a345701a23a1-kube-api-access-vspth" pod "541d0cfd-a005-480d-84e7-a345701a23a1" (UID: "541d0cfd-a005-480d-84e7-a345701a23a1"). InnerVolumeSpecName "kube-api-access-vspth". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:05:07.867136 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:05:07.867109 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/541d0cfd-a005-480d-84e7-a345701a23a1-maas-api-xks-ca" pod "541d0cfd-a005-480d-84e7-a345701a23a1" (UID: "541d0cfd-a005-480d-84e7-a345701a23a1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:05:07.966737 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:05:07.966692 4927 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/541d0cfd-a005-480d-84e7-a345701a23a1-maas-api-xks-ca\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:05:07.966737 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:05:07.966725 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vspth\" (UniqueName: \"kubernetes.io/projected/541d0cfd-a005-480d-84e7-a345701a23a1-kube-api-access-vspth\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:05:08.504968 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:05:08.504936 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f77eb00c-hprr6" Oct 31 00:05:08.505381 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:05:08.504966 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f77eb00c-hprr6" event={"ID":"541d0cfd-a005-480d-84e7-a345701a23a1","Type":"ContainerDied","Data":"b90bb060d8f960423e54bc5f4f0198069fd40b026d233d7509d78b09933050cf"} Oct 31 00:05:08.505381 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:05:08.505020 4927 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b90bb060d8f960423e54bc5f4f0198069fd40b026d233d7509d78b09933050cf" Oct 31 00:05:08.511355 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:05:08.507613 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0d96587a-openshift-default-bcfbd49cc-xrrp4" event={"ID":"d310a62f-9f54-4000-b280-0581fd91192f","Type":"ContainerStarted","Data":"056e95e3b188a1fabbeb31f46aef4d2dc9c73b6713ef16e37291388071c7365d"} Oct 31 00:05:08.764418 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:05:08.762269 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:05:08.793354 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:05:08.791713 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-0d96587a-openshift-default-bcfbd49cc-xrrp4" Oct 31 00:05:08.793354 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:05:08.791838 4927 patch_prober.go:29] interesting pod/e2e-ait-0d96587a-openshift-default-bcfbd49cc-xrrp4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body= Oct 31 00:05:08.793354 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:05:08.791887 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-0d96587a-openshift-default-bcfbd49cc-xrrp4" podUID="d310a62f-9f54-4000-b280-0581fd91192f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" Oct 31 00:05:09.787018 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:05:09.786976 4927 patch_prober.go:29] interesting pod/e2e-ait-0d96587a-openshift-default-bcfbd49cc-xrrp4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body= Oct 31 00:05:09.787536 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:05:09.787076 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-0d96587a-openshift-default-bcfbd49cc-xrrp4" podUID="d310a62f-9f54-4000-b280-0581fd91192f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" Oct 31 00:05:10.787001 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:05:10.786964 4927 patch_prober.go:29] interesting pod/e2e-ait-0d96587a-openshift-default-bcfbd49cc-xrrp4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body= Oct 31 00:05:10.787176 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:05:10.787031 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-0d96587a-openshift-default-bcfbd49cc-xrrp4" podUID="d310a62f-9f54-4000-b280-0581fd91192f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" Oct 31 00:05:10.863200 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:05:10.863163 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:05:11.437929 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:05:11.437886 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:05:11.592467 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:05:11.592432 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:05:11.787000 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:05:11.786915 4927 patch_prober.go:29] interesting pod/e2e-ait-0d96587a-openshift-default-bcfbd49cc-xrrp4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body= Oct 31 00:05:11.787000 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:05:11.786973 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-0d96587a-openshift-default-bcfbd49cc-xrrp4" podUID="d310a62f-9f54-4000-b280-0581fd91192f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" Oct 31 00:05:12.789380 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:05:12.789250 4927 patch_prober.go:29] interesting pod/e2e-ait-0d96587a-openshift-default-bcfbd49cc-xrrp4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body= Oct 31 00:05:12.789380 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:05:12.789314 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-0d96587a-openshift-default-bcfbd49cc-xrrp4" podUID="d310a62f-9f54-4000-b280-0581fd91192f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" Oct 31 00:05:13.500968 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:05:13.500930 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:05:13.787096 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:05:13.786986 4927 patch_prober.go:29] interesting pod/e2e-ait-0d96587a-openshift-default-bcfbd49cc-xrrp4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body= Oct 31 00:05:13.787096 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:05:13.787066 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-0d96587a-openshift-default-bcfbd49cc-xrrp4" podUID="d310a62f-9f54-4000-b280-0581fd91192f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" Oct 31 00:05:14.405181 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:05:14.405144 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-0d96587a-openshift-default-bcfbd49cc-xrrp4" Oct 31 00:05:14.792735 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:05:14.792698 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-0d96587a-openshift-default-bcfbd49cc-xrrp4" Oct 31 00:05:14.794068 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:05:14.794041 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-0d96587a-openshift-default-bcfbd49cc-xrrp4" Oct 31 00:05:14.973696 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:05:14.973652 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:05:18.362228 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:05:18.362181 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8e7d0098-mjkns"] Oct 31 00:05:18.364389 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:05:18.364284 4927 cpu_manager.go:414] "containerMap: removing container" podUID="541d0cfd-a005-480d-84e7-a345701a23a1" containerName="revoke-keys" Oct 31 00:05:18.364555 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:05:18.364537 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="541d0cfd-a005-480d-84e7-a345701a23a1" containerName="revoke-keys" Oct 31 00:05:18.365352 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:05:18.365313 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="541d0cfd-a005-480d-84e7-a345701a23a1" containerName="revoke-keys" Oct 31 00:05:18.375772 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:05:18.375738 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8e7d0098-mjkns" Oct 31 00:05:18.402607 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:05:18.402576 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8e7d0098-mjkns"] Oct 31 00:05:18.434805 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:05:18.434770 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4fdzs\"" Oct 31 00:05:18.434998 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:05:18.434870 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:05:18.501433 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:05:18.500837 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-576rd\" (UniqueName: \"kubernetes.io/projected/27a0d6f4-fbe8-4bdd-a8cb-7a413d3939ce-kube-api-access-576rd\") pod \"maas-api-revoke-keys-e2e-ait-8e7d0098-mjkns\" (UID: \"27a0d6f4-fbe8-4bdd-a8cb-7a413d3939ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8e7d0098-mjkns" Oct 31 00:05:18.501433 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:05:18.500998 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27a0d6f4-fbe8-4bdd-a8cb-7a413d3939ce-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-8e7d0098-mjkns\" (UID: \"27a0d6f4-fbe8-4bdd-a8cb-7a413d3939ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8e7d0098-mjkns" Oct 31 00:05:18.501433 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:05:18.501286 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27a0d6f4-fbe8-4bdd-a8cb-7a413d3939ce-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-8e7d0098-mjkns\" (UID: \"27a0d6f4-fbe8-4bdd-a8cb-7a413d3939ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8e7d0098-mjkns" Oct 31 00:05:18.602811 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:05:18.602769 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27a0d6f4-fbe8-4bdd-a8cb-7a413d3939ce-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-8e7d0098-mjkns\" (UID: \"27a0d6f4-fbe8-4bdd-a8cb-7a413d3939ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8e7d0098-mjkns" Oct 31 00:05:18.603741 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:05:18.602858 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-576rd\" (UniqueName: \"kubernetes.io/projected/27a0d6f4-fbe8-4bdd-a8cb-7a413d3939ce-kube-api-access-576rd\") pod \"maas-api-revoke-keys-e2e-ait-8e7d0098-mjkns\" (UID: \"27a0d6f4-fbe8-4bdd-a8cb-7a413d3939ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8e7d0098-mjkns" Oct 31 00:05:18.603741 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:05:18.602972 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27a0d6f4-fbe8-4bdd-a8cb-7a413d3939ce-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-8e7d0098-mjkns\" (UID: \"27a0d6f4-fbe8-4bdd-a8cb-7a413d3939ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8e7d0098-mjkns" Oct 31 00:05:18.603741 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:05:18.603571 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27a0d6f4-fbe8-4bdd-a8cb-7a413d3939ce-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-8e7d0098-mjkns\" (UID: \"27a0d6f4-fbe8-4bdd-a8cb-7a413d3939ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8e7d0098-mjkns" Oct 31 00:05:18.605290 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:05:18.605264 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27a0d6f4-fbe8-4bdd-a8cb-7a413d3939ce-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-8e7d0098-mjkns\" (UID: \"27a0d6f4-fbe8-4bdd-a8cb-7a413d3939ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8e7d0098-mjkns" Oct 31 00:05:18.666040 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:05:18.666003 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-576rd\" (UniqueName: \"kubernetes.io/projected/27a0d6f4-fbe8-4bdd-a8cb-7a413d3939ce-kube-api-access-576rd\") pod \"maas-api-revoke-keys-e2e-ait-8e7d0098-mjkns\" (UID: \"27a0d6f4-fbe8-4bdd-a8cb-7a413d3939ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8e7d0098-mjkns" Oct 31 00:05:18.715935 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:05:18.715901 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8e7d0098-mjkns" Oct 31 00:05:18.975173 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:05:18.975017 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:05:19.149497 ip-10-0-141-180 kubenswrapper[4927]: W1031 00:05:19.149467 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27a0d6f4_fbe8_4bdd_a8cb_7a413d3939ce.slice/crio-aaf3a000d57478623e9b78ceaef0d9e35420105dd7c3ae84aa4e8fd5757bc101 WatchSource:0}: Error finding container aaf3a000d57478623e9b78ceaef0d9e35420105dd7c3ae84aa4e8fd5757bc101: Status 404 returned error can't find the container with id aaf3a000d57478623e9b78ceaef0d9e35420105dd7c3ae84aa4e8fd5757bc101 Oct 31 00:05:19.151385 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:05:19.151355 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8e7d0098-mjkns"] Oct 31 00:05:19.680604 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:05:19.680568 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sqv69" containerName="extractor" Oct 31 00:05:19.725317 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:05:19.725275 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8e7d0098-mjkns" event={"ID":"27a0d6f4-fbe8-4bdd-a8cb-7a413d3939ce","Type":"ContainerStarted","Data":"4db49853d5571d577dd419949769b96ec9ce49fb1cc21f1eb223367eb3987675"} Oct 31 00:05:19.725317 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:05:19.725347 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8e7d0098-mjkns" event={"ID":"27a0d6f4-fbe8-4bdd-a8cb-7a413d3939ce","Type":"ContainerStarted","Data":"aaf3a000d57478623e9b78ceaef0d9e35420105dd7c3ae84aa4e8fd5757bc101"} Oct 31 00:05:19.852406 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:05:19.852328 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8e7d0098-mjkns" podStartSLOduration=1.852312856 podStartE2EDuration="1.852312856s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:05:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:05:19.850174939 +0000 UTC m=+1432.886505761" watchObservedRunningTime="2026-10-31 00:05:19.852312856 +0000 UTC m=+1432.888643677" Oct 31 00:05:20.737001 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:05:20.736966 4927 generic.go:356] "Generic (PLEG): container finished" podID="27a0d6f4-fbe8-4bdd-a8cb-7a413d3939ce" containerID="4db49853d5571d577dd419949769b96ec9ce49fb1cc21f1eb223367eb3987675" exitCode=0 Oct 31 00:05:20.737421 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:05:20.737048 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8e7d0098-mjkns" event={"ID":"27a0d6f4-fbe8-4bdd-a8cb-7a413d3939ce","Type":"ContainerDied","Data":"4db49853d5571d577dd419949769b96ec9ce49fb1cc21f1eb223367eb3987675"} Oct 31 00:05:21.845217 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:05:21.845164 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:05:21.924629 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:05:21.924601 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8e7d0098-mjkns" Oct 31 00:05:21.962719 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:05:21.962684 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/27a0d6f4-fbe8-4bdd-a8cb-7a413d3939ce-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27a0d6f4-fbe8-4bdd-a8cb-7a413d3939ce-maas-api-xks-ca\") pod \"27a0d6f4-fbe8-4bdd-a8cb-7a413d3939ce\" (UID: \"27a0d6f4-fbe8-4bdd-a8cb-7a413d3939ce\") " Oct 31 00:05:21.962904 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:05:21.962883 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27a0d6f4-fbe8-4bdd-a8cb-7a413d3939ce-kube-api-access-576rd\" (UniqueName: \"kubernetes.io/projected/27a0d6f4-fbe8-4bdd-a8cb-7a413d3939ce-kube-api-access-576rd\") pod \"27a0d6f4-fbe8-4bdd-a8cb-7a413d3939ce\" (UID: \"27a0d6f4-fbe8-4bdd-a8cb-7a413d3939ce\") " Oct 31 00:05:21.962986 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:05:21.962955 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/27a0d6f4-fbe8-4bdd-a8cb-7a413d3939ce-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27a0d6f4-fbe8-4bdd-a8cb-7a413d3939ce-maas-api-openshift-ca\") pod \"27a0d6f4-fbe8-4bdd-a8cb-7a413d3939ce\" (UID: \"27a0d6f4-fbe8-4bdd-a8cb-7a413d3939ce\") " Oct 31 00:05:21.963321 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:05:21.963288 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27a0d6f4-fbe8-4bdd-a8cb-7a413d3939ce-maas-api-openshift-ca" pod "27a0d6f4-fbe8-4bdd-a8cb-7a413d3939ce" (UID: "27a0d6f4-fbe8-4bdd-a8cb-7a413d3939ce"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:05:21.963710 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:05:21.963687 4927 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27a0d6f4-fbe8-4bdd-a8cb-7a413d3939ce-maas-api-openshift-ca\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:05:21.964817 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:05:21.964792 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/27a0d6f4-fbe8-4bdd-a8cb-7a413d3939ce-maas-api-xks-ca" pod "27a0d6f4-fbe8-4bdd-a8cb-7a413d3939ce" (UID: "27a0d6f4-fbe8-4bdd-a8cb-7a413d3939ce"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:05:21.964927 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:05:21.964912 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27a0d6f4-fbe8-4bdd-a8cb-7a413d3939ce-kube-api-access-576rd" pod "27a0d6f4-fbe8-4bdd-a8cb-7a413d3939ce" (UID: "27a0d6f4-fbe8-4bdd-a8cb-7a413d3939ce"). InnerVolumeSpecName "kube-api-access-576rd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:05:22.065146 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:05:22.065054 4927 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27a0d6f4-fbe8-4bdd-a8cb-7a413d3939ce-maas-api-xks-ca\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:05:22.065146 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:05:22.065085 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-576rd\" (UniqueName: \"kubernetes.io/projected/27a0d6f4-fbe8-4bdd-a8cb-7a413d3939ce-kube-api-access-576rd\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:05:22.758870 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:05:22.758838 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8e7d0098-mjkns" Oct 31 00:05:22.759029 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:05:22.758873 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8e7d0098-mjkns" event={"ID":"27a0d6f4-fbe8-4bdd-a8cb-7a413d3939ce","Type":"ContainerDied","Data":"aaf3a000d57478623e9b78ceaef0d9e35420105dd7c3ae84aa4e8fd5757bc101"} Oct 31 00:05:22.759029 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:05:22.758948 4927 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aaf3a000d57478623e9b78ceaef0d9e35420105dd7c3ae84aa4e8fd5757bc101" Oct 31 00:05:24.087296 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:05:24.087258 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:05:25.444901 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:05:25.444859 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0d96587a-fdvr4"] Oct 31 00:05:25.446151 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:05:25.446126 4927 cpu_manager.go:414] "containerMap: removing container" podUID="27a0d6f4-fbe8-4bdd-a8cb-7a413d3939ce" containerName="revoke-keys" Oct 31 00:05:25.446250 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:05:25.446154 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27a0d6f4-fbe8-4bdd-a8cb-7a413d3939ce" containerName="revoke-keys" Oct 31 00:05:25.446451 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:05:25.446435 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="27a0d6f4-fbe8-4bdd-a8cb-7a413d3939ce" containerName="revoke-keys" Oct 31 00:05:25.451519 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:05:25.451496 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0d96587a-fdvr4" Oct 31 00:05:25.505542 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:05:25.505496 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4fdzs\"" Oct 31 00:05:25.505860 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:05:25.505837 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0d96587a-fdvr4"] Oct 31 00:05:25.548448 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:05:25.548419 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:05:25.549111 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:05:25.549070 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a32ae26c-e389-4784-979e-f6b5e87e56dc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-0d96587a-fdvr4\" (UID: \"a32ae26c-e389-4784-979e-f6b5e87e56dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0d96587a-fdvr4" Oct 31 00:05:25.549416 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:05:25.549395 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a32ae26c-e389-4784-979e-f6b5e87e56dc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-0d96587a-fdvr4\" (UID: \"a32ae26c-e389-4784-979e-f6b5e87e56dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0d96587a-fdvr4" Oct 31 00:05:25.549497 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:05:25.549426 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jgm9r\" (UniqueName: \"kubernetes.io/projected/a32ae26c-e389-4784-979e-f6b5e87e56dc-kube-api-access-jgm9r\") pod \"maas-api-revoke-keys-e2e-ait-0d96587a-fdvr4\" (UID: \"a32ae26c-e389-4784-979e-f6b5e87e56dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0d96587a-fdvr4" Oct 31 00:05:25.651851 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:05:25.651811 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a32ae26c-e389-4784-979e-f6b5e87e56dc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-0d96587a-fdvr4\" (UID: \"a32ae26c-e389-4784-979e-f6b5e87e56dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0d96587a-fdvr4" Oct 31 00:05:25.652051 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:05:25.651863 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgm9r\" (UniqueName: \"kubernetes.io/projected/a32ae26c-e389-4784-979e-f6b5e87e56dc-kube-api-access-jgm9r\") pod \"maas-api-revoke-keys-e2e-ait-0d96587a-fdvr4\" (UID: \"a32ae26c-e389-4784-979e-f6b5e87e56dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0d96587a-fdvr4" Oct 31 00:05:25.652051 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:05:25.652025 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a32ae26c-e389-4784-979e-f6b5e87e56dc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-0d96587a-fdvr4\" (UID: \"a32ae26c-e389-4784-979e-f6b5e87e56dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0d96587a-fdvr4" Oct 31 00:05:25.652575 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:05:25.652547 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a32ae26c-e389-4784-979e-f6b5e87e56dc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-0d96587a-fdvr4\" (UID: \"a32ae26c-e389-4784-979e-f6b5e87e56dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0d96587a-fdvr4" Oct 31 00:05:25.654499 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:05:25.654466 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a32ae26c-e389-4784-979e-f6b5e87e56dc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-0d96587a-fdvr4\" (UID: \"a32ae26c-e389-4784-979e-f6b5e87e56dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0d96587a-fdvr4" Oct 31 00:05:25.715106 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:05:25.715028 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jgm9r\" (UniqueName: \"kubernetes.io/projected/a32ae26c-e389-4784-979e-f6b5e87e56dc-kube-api-access-jgm9r\") pod \"maas-api-revoke-keys-e2e-ait-0d96587a-fdvr4\" (UID: \"a32ae26c-e389-4784-979e-f6b5e87e56dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0d96587a-fdvr4" Oct 31 00:05:25.767284 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:05:25.767252 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0d96587a-fdvr4" Oct 31 00:05:25.773287 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:05:25.773251 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:05:26.122947 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:05:26.122843 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:05:26.167871 ip-10-0-141-180 kubenswrapper[4927]: W1031 00:05:26.167836 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda32ae26c_e389_4784_979e_f6b5e87e56dc.slice/crio-93161dba62b6f68e2133591288c41af080f5053991708a6c12a863e8aa1935e0 WatchSource:0}: Error finding container 93161dba62b6f68e2133591288c41af080f5053991708a6c12a863e8aa1935e0: Status 404 returned error can't find the container with id 93161dba62b6f68e2133591288c41af080f5053991708a6c12a863e8aa1935e0 Oct 31 00:05:26.171639 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:05:26.171601 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0d96587a-fdvr4"] Oct 31 00:05:26.306112 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:05:26.306073 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:05:26.799010 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:05:26.798903 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0d96587a-fdvr4" event={"ID":"a32ae26c-e389-4784-979e-f6b5e87e56dc","Type":"ContainerStarted","Data":"61d254c0a0eb124fd07bfad8fe7dfc6b785f25c5411d961cdf3f23e322e6b7db"} Oct 31 00:05:26.799010 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:05:26.798970 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0d96587a-fdvr4" event={"ID":"a32ae26c-e389-4784-979e-f6b5e87e56dc","Type":"ContainerStarted","Data":"93161dba62b6f68e2133591288c41af080f5053991708a6c12a863e8aa1935e0"} Oct 31 00:05:26.923328 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:05:26.923274 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0d96587a-fdvr4" podStartSLOduration=1.92326069 podStartE2EDuration="1.92326069s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:05:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:05:26.921230617 +0000 UTC m=+1439.957561439" watchObservedRunningTime="2026-10-31 00:05:26.92326069 +0000 UTC m=+1439.959591517" Oct 31 00:05:27.810495 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:05:27.810459 4927 generic.go:356] "Generic (PLEG): container finished" podID="a32ae26c-e389-4784-979e-f6b5e87e56dc" containerID="61d254c0a0eb124fd07bfad8fe7dfc6b785f25c5411d961cdf3f23e322e6b7db" exitCode=0 Oct 31 00:05:27.811010 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:05:27.810547 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0d96587a-fdvr4" event={"ID":"a32ae26c-e389-4784-979e-f6b5e87e56dc","Type":"ContainerDied","Data":"61d254c0a0eb124fd07bfad8fe7dfc6b785f25c5411d961cdf3f23e322e6b7db"} Oct 31 00:05:28.999897 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:05:28.999865 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:05:29.022887 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:05:29.022861 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0d96587a-fdvr4" Oct 31 00:05:29.157194 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:05:29.155441 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a32ae26c-e389-4784-979e-f6b5e87e56dc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a32ae26c-e389-4784-979e-f6b5e87e56dc-maas-api-openshift-ca\") pod \"a32ae26c-e389-4784-979e-f6b5e87e56dc\" (UID: \"a32ae26c-e389-4784-979e-f6b5e87e56dc\") " Oct 31 00:05:29.157194 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:05:29.155769 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a32ae26c-e389-4784-979e-f6b5e87e56dc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a32ae26c-e389-4784-979e-f6b5e87e56dc-maas-api-xks-ca\") pod \"a32ae26c-e389-4784-979e-f6b5e87e56dc\" (UID: \"a32ae26c-e389-4784-979e-f6b5e87e56dc\") " Oct 31 00:05:29.157194 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:05:29.155806 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a32ae26c-e389-4784-979e-f6b5e87e56dc-kube-api-access-jgm9r\" (UniqueName: \"kubernetes.io/projected/a32ae26c-e389-4784-979e-f6b5e87e56dc-kube-api-access-jgm9r\") pod \"a32ae26c-e389-4784-979e-f6b5e87e56dc\" (UID: \"a32ae26c-e389-4784-979e-f6b5e87e56dc\") " Oct 31 00:05:29.159434 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:05:29.157932 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a32ae26c-e389-4784-979e-f6b5e87e56dc-maas-api-openshift-ca" pod "a32ae26c-e389-4784-979e-f6b5e87e56dc" (UID: "a32ae26c-e389-4784-979e-f6b5e87e56dc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:05:29.164216 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:05:29.164168 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a32ae26c-e389-4784-979e-f6b5e87e56dc-maas-api-xks-ca" pod "a32ae26c-e389-4784-979e-f6b5e87e56dc" (UID: "a32ae26c-e389-4784-979e-f6b5e87e56dc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:05:29.179604 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:05:29.179533 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a32ae26c-e389-4784-979e-f6b5e87e56dc-kube-api-access-jgm9r" pod "a32ae26c-e389-4784-979e-f6b5e87e56dc" (UID: "a32ae26c-e389-4784-979e-f6b5e87e56dc"). InnerVolumeSpecName "kube-api-access-jgm9r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:05:29.266506 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:05:29.263076 4927 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a32ae26c-e389-4784-979e-f6b5e87e56dc-maas-api-openshift-ca\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:05:29.266506 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:05:29.263117 4927 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a32ae26c-e389-4784-979e-f6b5e87e56dc-maas-api-xks-ca\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:05:29.266506 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:05:29.263133 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jgm9r\" (UniqueName: \"kubernetes.io/projected/a32ae26c-e389-4784-979e-f6b5e87e56dc-kube-api-access-jgm9r\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:05:29.730271 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:05:29.730232 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-8e7d0098-8f775757b-4mt6m"] Oct 31 00:05:29.837455 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:05:29.837414 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-8e7d0098-6794f6ff67-429f5"] Oct 31 00:05:29.850976 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:05:29.850932 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0d96587a-fdvr4" event={"ID":"a32ae26c-e389-4784-979e-f6b5e87e56dc","Type":"ContainerDied","Data":"93161dba62b6f68e2133591288c41af080f5053991708a6c12a863e8aa1935e0"} Oct 31 00:05:29.851171 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:05:29.851001 4927 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="93161dba62b6f68e2133591288c41af080f5053991708a6c12a863e8aa1935e0" Oct 31 00:05:29.851171 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:05:29.851082 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0d96587a-fdvr4" Oct 31 00:05:30.064957 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:05:30.064867 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:05:30.334850 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:05:30.334737 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:05:34.161682 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:05:34.161641 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:05:34.733476 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:05:34.733399 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-8e7d0098-8f775757b-4mt6m" podUID="daee2336-91a5-4c33-8bdf-b1c0e5538c35" containerName="payload-processing" containerID="cri-o://c80e4c0cfb6a79697cd6d5bfb8b83d4e602034d065f2423ad284d2f5a43b9a85" gracePeriod=25 Oct 31 00:05:34.838480 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:05:34.838411 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-8e7d0098-6794f6ff67-429f5" podUID="5779f4e6-fab2-4060-a6e6-b64f4756d7a5" containerName="payload-pre-processing" containerID="cri-o://d7e2809ce27ccd8fc6ea633bc9188efa8209d8d8f5e88252632b3c8c99397927" gracePeriod=25 Oct 31 00:05:35.264347 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:05:35.264269 4927 generic.go:356] "Generic (PLEG): container finished" podID="daee2336-91a5-4c33-8bdf-b1c0e5538c35" containerID="c80e4c0cfb6a79697cd6d5bfb8b83d4e602034d065f2423ad284d2f5a43b9a85" exitCode=0 Oct 31 00:05:35.264857 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:05:35.264432 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-8e7d0098-8f775757b-4mt6m" event={"ID":"daee2336-91a5-4c33-8bdf-b1c0e5538c35","Type":"ContainerDied","Data":"c80e4c0cfb6a79697cd6d5bfb8b83d4e602034d065f2423ad284d2f5a43b9a85"} Oct 31 00:05:35.283670 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:05:35.283619 4927 generic.go:356] "Generic (PLEG): container finished" podID="5779f4e6-fab2-4060-a6e6-b64f4756d7a5" containerID="d7e2809ce27ccd8fc6ea633bc9188efa8209d8d8f5e88252632b3c8c99397927" exitCode=0 Oct 31 00:05:35.284376 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:05:35.284318 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-8e7d0098-6794f6ff67-429f5" event={"ID":"5779f4e6-fab2-4060-a6e6-b64f4756d7a5","Type":"ContainerDied","Data":"d7e2809ce27ccd8fc6ea633bc9188efa8209d8d8f5e88252632b3c8c99397927"} Oct 31 00:05:35.822913 ip-10-0-141-180 kubenswrapper[4927]: W1031 00:05:35.788605 4927 logging.go:55] [core] [Channel #2890 SubChannel #2891]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.147:50052", ServerName: "10.132.0.147:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.147:50052: connect: connection refused" Oct 31 00:05:36.238355 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:05:36.231886 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-8e7d0098-8f775757b-4mt6m" Oct 31 00:05:36.438361 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:05:36.429241 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/daee2336-91a5-4c33-8bdf-b1c0e5538c35-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/daee2336-91a5-4c33-8bdf-b1c0e5538c35-plugins-config-volume\") pod \"daee2336-91a5-4c33-8bdf-b1c0e5538c35\" (UID: \"daee2336-91a5-4c33-8bdf-b1c0e5538c35\") " Oct 31 00:05:36.439158 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:05:36.439114 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/daee2336-91a5-4c33-8bdf-b1c0e5538c35-kube-api-access-wl9zg\" (UniqueName: \"kubernetes.io/projected/daee2336-91a5-4c33-8bdf-b1c0e5538c35-kube-api-access-wl9zg\") pod \"daee2336-91a5-4c33-8bdf-b1c0e5538c35\" (UID: \"daee2336-91a5-4c33-8bdf-b1c0e5538c35\") " Oct 31 00:05:36.447067 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:05:36.430757 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/daee2336-91a5-4c33-8bdf-b1c0e5538c35-plugins-config-volume" pod "daee2336-91a5-4c33-8bdf-b1c0e5538c35" (UID: "daee2336-91a5-4c33-8bdf-b1c0e5538c35"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:05:36.490674 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:05:36.490591 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-8e7d0098-8f775757b-4mt6m" Oct 31 00:05:36.490862 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:05:36.490807 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-8e7d0098-8f775757b-4mt6m" event={"ID":"daee2336-91a5-4c33-8bdf-b1c0e5538c35","Type":"ContainerDied","Data":"f79ced99e90bcecaa11a4aee39003cf3c3316e2c7c7c72500e30a621ada3d7b3"} Oct 31 00:05:36.490946 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:05:36.490881 4927 scope.go:122] "RemoveContainer" containerID="c80e4c0cfb6a79697cd6d5bfb8b83d4e602034d065f2423ad284d2f5a43b9a85" Oct 31 00:05:36.499119 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:05:36.497436 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/daee2336-91a5-4c33-8bdf-b1c0e5538c35-kube-api-access-wl9zg" pod "daee2336-91a5-4c33-8bdf-b1c0e5538c35" (UID: "daee2336-91a5-4c33-8bdf-b1c0e5538c35"). InnerVolumeSpecName "kube-api-access-wl9zg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:05:36.500096 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:05:36.500074 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-8e7d0098-6794f6ff67-429f5" Oct 31 00:05:36.544544 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:05:36.541816 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5779f4e6-fab2-4060-a6e6-b64f4756d7a5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5779f4e6-fab2-4060-a6e6-b64f4756d7a5-plugins-config-volume\") pod \"5779f4e6-fab2-4060-a6e6-b64f4756d7a5\" (UID: \"5779f4e6-fab2-4060-a6e6-b64f4756d7a5\") " Oct 31 00:05:36.544544 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:05:36.541885 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5779f4e6-fab2-4060-a6e6-b64f4756d7a5-kube-api-access-lp9tn\" (UniqueName: \"kubernetes.io/projected/5779f4e6-fab2-4060-a6e6-b64f4756d7a5-kube-api-access-lp9tn\") pod \"5779f4e6-fab2-4060-a6e6-b64f4756d7a5\" (UID: \"5779f4e6-fab2-4060-a6e6-b64f4756d7a5\") " Oct 31 00:05:36.544544 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:05:36.542548 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wl9zg\" (UniqueName: \"kubernetes.io/projected/daee2336-91a5-4c33-8bdf-b1c0e5538c35-kube-api-access-wl9zg\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:05:36.544544 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:05:36.542570 4927 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/daee2336-91a5-4c33-8bdf-b1c0e5538c35-plugins-config-volume\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:05:36.544544 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:05:36.543273 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5779f4e6-fab2-4060-a6e6-b64f4756d7a5-plugins-config-volume" pod "5779f4e6-fab2-4060-a6e6-b64f4756d7a5" (UID: "5779f4e6-fab2-4060-a6e6-b64f4756d7a5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:05:36.545470 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:05:36.545441 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5779f4e6-fab2-4060-a6e6-b64f4756d7a5-kube-api-access-lp9tn" pod "5779f4e6-fab2-4060-a6e6-b64f4756d7a5" (UID: "5779f4e6-fab2-4060-a6e6-b64f4756d7a5"). InnerVolumeSpecName "kube-api-access-lp9tn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:05:36.645802 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:05:36.645765 4927 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5779f4e6-fab2-4060-a6e6-b64f4756d7a5-plugins-config-volume\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:05:36.646079 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:05:36.646060 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lp9tn\" (UniqueName: \"kubernetes.io/projected/5779f4e6-fab2-4060-a6e6-b64f4756d7a5-kube-api-access-lp9tn\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:05:36.743524 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:05:36.743411 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-ait-8e7d0098-8f775757b-4mt6m" podUID="daee2336-91a5-4c33-8bdf-b1c0e5538c35" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.148:50052\" within 1s: context deadline exceeded" Oct 31 00:05:36.743722 ip-10-0-141-180 kubenswrapper[4927]: W1031 00:05:36.743597 4927 logging.go:55] [core] [Channel #2888 SubChannel #2889]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.148:50052", ServerName: "10.132.0.148:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.148:50052: operation was canceled" Oct 31 00:05:36.757690 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:05:36.757635 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-ait-8e7d0098-6794f6ff67-429f5" podUID="5779f4e6-fab2-4060-a6e6-b64f4756d7a5" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.147:50052\" within 1s: context deadline exceeded" Oct 31 00:05:36.812940 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:05:36.812903 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-0d96587a-58d9d8846-j62hj"] Oct 31 00:05:36.911123 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:05:36.911088 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-0d96587a-584d7b5dff-ccjj6"] Oct 31 00:05:36.922431 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:05:36.922395 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-8e7d0098-8f775757b-4mt6m"] Oct 31 00:05:36.928607 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:05:36.928584 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-8e7d0098-8f775757b-4mt6m"] Oct 31 00:05:37.091731 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:05:37.091633 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="daee2336-91a5-4c33-8bdf-b1c0e5538c35" path="/var/lib/kubelet/pods/daee2336-91a5-4c33-8bdf-b1c0e5538c35/volumes" Oct 31 00:05:37.164828 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:05:37.164797 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:05:37.418513 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:05:37.416824 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:05:37.630824 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:05:37.630766 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-8e7d0098-6794f6ff67-429f5" event={"ID":"5779f4e6-fab2-4060-a6e6-b64f4756d7a5","Type":"ContainerDied","Data":"4dc4a9688df47d0033355b06bc79f8f5a904886816347686ce9ae04f9fa14dbc"} Oct 31 00:05:37.631434 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:05:37.630907 4927 scope.go:122] "RemoveContainer" containerID="d7e2809ce27ccd8fc6ea633bc9188efa8209d8d8f5e88252632b3c8c99397927" Oct 31 00:05:37.631434 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:05:37.631095 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-8e7d0098-6794f6ff67-429f5" Oct 31 00:05:37.808173 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:05:37.808132 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-8e7d0098-6794f6ff67-429f5"] Oct 31 00:05:37.814407 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:05:37.814376 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-8e7d0098-6794f6ff67-429f5"] Oct 31 00:05:39.115427 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:05:39.110219 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5779f4e6-fab2-4060-a6e6-b64f4756d7a5" path="/var/lib/kubelet/pods/5779f4e6-fab2-4060-a6e6-b64f4756d7a5/volumes" Oct 31 00:05:39.298215 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:05:39.298177 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:05:41.813287 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:05:41.813226 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-0d96587a-58d9d8846-j62hj" podUID="0668f592-496f-412b-92c6-23b551aff729" containerName="payload-processing" containerID="cri-o://a4dd6e554576bacab85bf0e0513a9cac19874cd497b4ebed88e29c72ac90bb3f" gracePeriod=25 Oct 31 00:05:41.911575 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:05:41.911505 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-0d96587a-584d7b5dff-ccjj6" podUID="46996ba2-e5f5-4d2a-897b-7d82db8ba74f" containerName="payload-pre-processing" containerID="cri-o://9cc22c403958fcddd41f8cdf1b9e66e82f32ab9313cd99ed5e0cf4f054cd0e7a" gracePeriod=25 Oct 31 00:05:42.086690 ip-10-0-141-180 kubenswrapper[4927]: W1031 00:05:42.083881 4927 logging.go:55] [core] [Channel #2928 SubChannel #2930]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.149:50052", ServerName: "10.132.0.149:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.149:50052: connect: connection refused" Oct 31 00:05:42.146183 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:05:42.146156 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-0d96587a-58d9d8846-j62hj" Oct 31 00:05:42.236769 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:05:42.236719 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0668f592-496f-412b-92c6-23b551aff729-kube-api-access-mz695\" (UniqueName: \"kubernetes.io/projected/0668f592-496f-412b-92c6-23b551aff729-kube-api-access-mz695\") pod \"0668f592-496f-412b-92c6-23b551aff729\" (UID: \"0668f592-496f-412b-92c6-23b551aff729\") " Oct 31 00:05:42.236978 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:05:42.236782 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0668f592-496f-412b-92c6-23b551aff729-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0668f592-496f-412b-92c6-23b551aff729-plugins-config-volume\") pod \"0668f592-496f-412b-92c6-23b551aff729\" (UID: \"0668f592-496f-412b-92c6-23b551aff729\") " Oct 31 00:05:42.237217 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:05:42.237181 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0668f592-496f-412b-92c6-23b551aff729-plugins-config-volume" pod "0668f592-496f-412b-92c6-23b551aff729" (UID: "0668f592-496f-412b-92c6-23b551aff729"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:05:42.237570 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:05:42.237548 4927 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0668f592-496f-412b-92c6-23b551aff729-plugins-config-volume\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:05:42.239808 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:05:42.239779 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0668f592-496f-412b-92c6-23b551aff729-kube-api-access-mz695" pod "0668f592-496f-412b-92c6-23b551aff729" (UID: "0668f592-496f-412b-92c6-23b551aff729"). InnerVolumeSpecName "kube-api-access-mz695". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:05:42.283481 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:05:42.283452 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-0d96587a-584d7b5dff-ccjj6" Oct 31 00:05:42.339031 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:05:42.338873 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/46996ba2-e5f5-4d2a-897b-7d82db8ba74f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46996ba2-e5f5-4d2a-897b-7d82db8ba74f-plugins-config-volume\") pod \"46996ba2-e5f5-4d2a-897b-7d82db8ba74f\" (UID: \"46996ba2-e5f5-4d2a-897b-7d82db8ba74f\") " Oct 31 00:05:42.339031 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:05:42.338996 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46996ba2-e5f5-4d2a-897b-7d82db8ba74f-kube-api-access-4mdbm\" (UniqueName: \"kubernetes.io/projected/46996ba2-e5f5-4d2a-897b-7d82db8ba74f-kube-api-access-4mdbm\") pod \"46996ba2-e5f5-4d2a-897b-7d82db8ba74f\" (UID: \"46996ba2-e5f5-4d2a-897b-7d82db8ba74f\") " Oct 31 00:05:42.340253 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:05:42.339851 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mz695\" (UniqueName: \"kubernetes.io/projected/0668f592-496f-412b-92c6-23b551aff729-kube-api-access-mz695\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:05:42.340253 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:05:42.340228 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46996ba2-e5f5-4d2a-897b-7d82db8ba74f-plugins-config-volume" pod "46996ba2-e5f5-4d2a-897b-7d82db8ba74f" (UID: "46996ba2-e5f5-4d2a-897b-7d82db8ba74f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:05:42.341009 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:05:42.340981 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:05:42.343111 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:05:42.343079 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46996ba2-e5f5-4d2a-897b-7d82db8ba74f-kube-api-access-4mdbm" pod "46996ba2-e5f5-4d2a-897b-7d82db8ba74f" (UID: "46996ba2-e5f5-4d2a-897b-7d82db8ba74f"). InnerVolumeSpecName "kube-api-access-4mdbm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:05:42.447405 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:05:42.442083 4927 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46996ba2-e5f5-4d2a-897b-7d82db8ba74f-plugins-config-volume\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:05:42.447656 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:05:42.447628 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4mdbm\" (UniqueName: \"kubernetes.io/projected/46996ba2-e5f5-4d2a-897b-7d82db8ba74f-kube-api-access-4mdbm\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:05:42.693857 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:05:42.693816 4927 generic.go:356] "Generic (PLEG): container finished" podID="46996ba2-e5f5-4d2a-897b-7d82db8ba74f" containerID="9cc22c403958fcddd41f8cdf1b9e66e82f32ab9313cd99ed5e0cf4f054cd0e7a" exitCode=0 Oct 31 00:05:42.694064 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:05:42.693865 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-0d96587a-584d7b5dff-ccjj6" Oct 31 00:05:42.694064 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:05:42.693903 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-0d96587a-584d7b5dff-ccjj6" event={"ID":"46996ba2-e5f5-4d2a-897b-7d82db8ba74f","Type":"ContainerDied","Data":"9cc22c403958fcddd41f8cdf1b9e66e82f32ab9313cd99ed5e0cf4f054cd0e7a"} Oct 31 00:05:42.694064 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:05:42.693971 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-0d96587a-584d7b5dff-ccjj6" event={"ID":"46996ba2-e5f5-4d2a-897b-7d82db8ba74f","Type":"ContainerDied","Data":"f3a20c1bcac87e6971f0015d417fcd4e50966a88be6d456f22a794a424a0100d"} Oct 31 00:05:42.694064 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:05:42.694003 4927 scope.go:122] "RemoveContainer" containerID="9cc22c403958fcddd41f8cdf1b9e66e82f32ab9313cd99ed5e0cf4f054cd0e7a" Oct 31 00:05:42.695567 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:05:42.695543 4927 generic.go:356] "Generic (PLEG): container finished" podID="0668f592-496f-412b-92c6-23b551aff729" containerID="a4dd6e554576bacab85bf0e0513a9cac19874cd497b4ebed88e29c72ac90bb3f" exitCode=0 Oct 31 00:05:42.695679 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:05:42.695583 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-0d96587a-58d9d8846-j62hj" Oct 31 00:05:42.695679 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:05:42.695606 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-0d96587a-58d9d8846-j62hj" event={"ID":"0668f592-496f-412b-92c6-23b551aff729","Type":"ContainerDied","Data":"a4dd6e554576bacab85bf0e0513a9cac19874cd497b4ebed88e29c72ac90bb3f"} Oct 31 00:05:42.695679 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:05:42.695655 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-0d96587a-58d9d8846-j62hj" event={"ID":"0668f592-496f-412b-92c6-23b551aff729","Type":"ContainerDied","Data":"c49e2ec0600c34aad0df0e25037a218d4cbb33e172bac9dd72734cb23275c2f4"} Oct 31 00:05:42.708041 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:05:42.708022 4927 scope.go:122] "RemoveContainer" containerID="9cc22c403958fcddd41f8cdf1b9e66e82f32ab9313cd99ed5e0cf4f054cd0e7a" Oct 31 00:05:42.708391 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:05:42.708367 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9cc22c403958fcddd41f8cdf1b9e66e82f32ab9313cd99ed5e0cf4f054cd0e7a\": container with ID starting with 9cc22c403958fcddd41f8cdf1b9e66e82f32ab9313cd99ed5e0cf4f054cd0e7a not found: ID does not exist" containerID="9cc22c403958fcddd41f8cdf1b9e66e82f32ab9313cd99ed5e0cf4f054cd0e7a" Oct 31 00:05:42.708476 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:05:42.708401 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9cc22c403958fcddd41f8cdf1b9e66e82f32ab9313cd99ed5e0cf4f054cd0e7a"} err="failed to get container status \"9cc22c403958fcddd41f8cdf1b9e66e82f32ab9313cd99ed5e0cf4f054cd0e7a\": rpc error: code = NotFound desc = could not find container \"9cc22c403958fcddd41f8cdf1b9e66e82f32ab9313cd99ed5e0cf4f054cd0e7a\": container with ID starting with 9cc22c403958fcddd41f8cdf1b9e66e82f32ab9313cd99ed5e0cf4f054cd0e7a not found: ID does not exist" Oct 31 00:05:42.708476 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:05:42.708428 4927 scope.go:122] "RemoveContainer" containerID="a4dd6e554576bacab85bf0e0513a9cac19874cd497b4ebed88e29c72ac90bb3f" Oct 31 00:05:42.720896 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:05:42.720873 4927 scope.go:122] "RemoveContainer" containerID="a4dd6e554576bacab85bf0e0513a9cac19874cd497b4ebed88e29c72ac90bb3f" Oct 31 00:05:42.721225 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:05:42.721201 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a4dd6e554576bacab85bf0e0513a9cac19874cd497b4ebed88e29c72ac90bb3f\": container with ID starting with a4dd6e554576bacab85bf0e0513a9cac19874cd497b4ebed88e29c72ac90bb3f not found: ID does not exist" containerID="a4dd6e554576bacab85bf0e0513a9cac19874cd497b4ebed88e29c72ac90bb3f" Oct 31 00:05:42.721285 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:05:42.721235 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4dd6e554576bacab85bf0e0513a9cac19874cd497b4ebed88e29c72ac90bb3f"} err="failed to get container status \"a4dd6e554576bacab85bf0e0513a9cac19874cd497b4ebed88e29c72ac90bb3f\": rpc error: code = NotFound desc = could not find container \"a4dd6e554576bacab85bf0e0513a9cac19874cd497b4ebed88e29c72ac90bb3f\": container with ID starting with a4dd6e554576bacab85bf0e0513a9cac19874cd497b4ebed88e29c72ac90bb3f not found: ID does not exist" Oct 31 00:05:42.753872 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:05:42.753834 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:05:42.867986 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:05:42.867954 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-0d96587a-58d9d8846-j62hj"] Oct 31 00:05:42.874096 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:05:42.874072 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-0d96587a-58d9d8846-j62hj"] Oct 31 00:05:42.985805 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:05:42.985714 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-0d96587a-584d7b5dff-ccjj6"] Oct 31 00:05:42.991905 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:05:42.991871 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-0d96587a-584d7b5dff-ccjj6"] Oct 31 00:05:43.078319 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:05:43.078269 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-ait-0d96587a-58d9d8846-j62hj" podUID="0668f592-496f-412b-92c6-23b551aff729" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.150:50052\" within 1s: context deadline exceeded" Oct 31 00:05:43.078536 ip-10-0-141-180 kubenswrapper[4927]: W1031 00:05:43.078364 4927 logging.go:55] [core] [Channel #2929 SubChannel #2931]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.150:50052", ServerName: "10.132.0.150:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.150:50052: operation was canceled" Oct 31 00:05:43.078536 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:05:43.078278 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-ait-0d96587a-584d7b5dff-ccjj6" podUID="46996ba2-e5f5-4d2a-897b-7d82db8ba74f" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.149:50052\" within 1s: context deadline exceeded" Oct 31 00:05:43.088052 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:05:43.088025 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0668f592-496f-412b-92c6-23b551aff729" path="/var/lib/kubelet/pods/0668f592-496f-412b-92c6-23b551aff729/volumes" Oct 31 00:05:43.088549 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:05:43.088525 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46996ba2-e5f5-4d2a-897b-7d82db8ba74f" path="/var/lib/kubelet/pods/46996ba2-e5f5-4d2a-897b-7d82db8ba74f/volumes" Oct 31 00:05:43.308607 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:05:43.308506 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:05:43.510269 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:05:43.510240 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:05:43.916905 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:05:43.916866 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:05:44.283314 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:05:44.283234 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:05:49.037059 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:05:49.037021 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:05:49.678533 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:05:49.678499 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:05:49.680298 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:05:49.680270 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sqv69" containerName="extractor" Oct 31 00:05:50.638658 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:05:50.638619 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:05:53.513376 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:05:53.513310 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:05:53.750380 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:05:53.750312 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:05:57.241994 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:05:57.241956 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:05:58.226096 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:05:58.226050 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:06:01.059717 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:01.059677 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-46e53d8c-h6qr7"] Oct 31 00:06:01.108568 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:01.108525 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-46e53d8c-h6qr7"] Oct 31 00:06:02.519246 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:02.519207 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:06:02.719075 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:02.719033 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:06:02.927262 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:02.927224 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d32a7cc0-6h24p"] Oct 31 00:06:02.928109 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:02.928087 4927 cpu_manager.go:414] "containerMap: removing container" podUID="5779f4e6-fab2-4060-a6e6-b64f4756d7a5" containerName="payload-pre-processing" Oct 31 00:06:02.928109 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:02.928109 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5779f4e6-fab2-4060-a6e6-b64f4756d7a5" containerName="payload-pre-processing" Oct 31 00:06:02.928277 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:02.928131 4927 cpu_manager.go:414] "containerMap: removing container" podUID="daee2336-91a5-4c33-8bdf-b1c0e5538c35" containerName="payload-processing" Oct 31 00:06:02.928277 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:02.928138 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="daee2336-91a5-4c33-8bdf-b1c0e5538c35" containerName="payload-processing" Oct 31 00:06:02.928277 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:02.928172 4927 cpu_manager.go:414] "containerMap: removing container" podUID="46996ba2-e5f5-4d2a-897b-7d82db8ba74f" containerName="payload-pre-processing" Oct 31 00:06:02.928277 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:02.928178 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46996ba2-e5f5-4d2a-897b-7d82db8ba74f" containerName="payload-pre-processing" Oct 31 00:06:02.928277 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:02.928194 4927 cpu_manager.go:414] "containerMap: removing container" podUID="0668f592-496f-412b-92c6-23b551aff729" containerName="payload-processing" Oct 31 00:06:02.928277 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:02.928199 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0668f592-496f-412b-92c6-23b551aff729" containerName="payload-processing" Oct 31 00:06:02.928277 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:02.928209 4927 cpu_manager.go:414] "containerMap: removing container" podUID="a32ae26c-e389-4784-979e-f6b5e87e56dc" containerName="revoke-keys" Oct 31 00:06:02.928277 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:02.928214 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a32ae26c-e389-4784-979e-f6b5e87e56dc" containerName="revoke-keys" Oct 31 00:06:02.928702 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:02.928320 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="a32ae26c-e389-4784-979e-f6b5e87e56dc" containerName="revoke-keys" Oct 31 00:06:02.928702 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:02.928346 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="5779f4e6-fab2-4060-a6e6-b64f4756d7a5" containerName="payload-pre-processing" Oct 31 00:06:02.928702 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:02.928360 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="daee2336-91a5-4c33-8bdf-b1c0e5538c35" containerName="payload-processing" Oct 31 00:06:02.928702 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:02.928376 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="46996ba2-e5f5-4d2a-897b-7d82db8ba74f" containerName="payload-pre-processing" Oct 31 00:06:02.928702 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:02.928389 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="0668f592-496f-412b-92c6-23b551aff729" containerName="payload-processing" Oct 31 00:06:02.932804 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:02.932782 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d32a7cc0-6h24p" Oct 31 00:06:02.991012 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:02.990972 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d32a7cc0-6h24p"] Oct 31 00:06:03.007775 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:03.007734 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f032e374-6050-4177-b404-a4f3e3bc50cf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-d32a7cc0-6h24p\" (UID: \"f032e374-6050-4177-b404-a4f3e3bc50cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d32a7cc0-6h24p" Oct 31 00:06:03.007972 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:03.007890 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwl6r\" (UniqueName: \"kubernetes.io/projected/f032e374-6050-4177-b404-a4f3e3bc50cf-kube-api-access-fwl6r\") pod \"maas-api-revoke-keys-e2e-migrate-d32a7cc0-6h24p\" (UID: \"f032e374-6050-4177-b404-a4f3e3bc50cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d32a7cc0-6h24p" Oct 31 00:06:03.008080 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:03.008062 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f032e374-6050-4177-b404-a4f3e3bc50cf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-d32a7cc0-6h24p\" (UID: \"f032e374-6050-4177-b404-a4f3e3bc50cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d32a7cc0-6h24p" Oct 31 00:06:03.033210 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:03.033177 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:06:03.033415 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:03.033319 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4fdzs\"" Oct 31 00:06:03.089697 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:03.089664 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="91a1209c-7d60-4ea7-af17-ad33b985792e" path="/var/lib/kubelet/pods/91a1209c-7d60-4ea7-af17-ad33b985792e/volumes" Oct 31 00:06:03.110026 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:03.109989 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f032e374-6050-4177-b404-a4f3e3bc50cf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-d32a7cc0-6h24p\" (UID: \"f032e374-6050-4177-b404-a4f3e3bc50cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d32a7cc0-6h24p" Oct 31 00:06:03.110275 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:03.110246 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f032e374-6050-4177-b404-a4f3e3bc50cf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-d32a7cc0-6h24p\" (UID: \"f032e374-6050-4177-b404-a4f3e3bc50cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d32a7cc0-6h24p" Oct 31 00:06:03.110468 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:03.110438 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwl6r\" (UniqueName: \"kubernetes.io/projected/f032e374-6050-4177-b404-a4f3e3bc50cf-kube-api-access-fwl6r\") pod \"maas-api-revoke-keys-e2e-migrate-d32a7cc0-6h24p\" (UID: \"f032e374-6050-4177-b404-a4f3e3bc50cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d32a7cc0-6h24p" Oct 31 00:06:03.111221 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:03.111201 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f032e374-6050-4177-b404-a4f3e3bc50cf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-d32a7cc0-6h24p\" (UID: \"f032e374-6050-4177-b404-a4f3e3bc50cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d32a7cc0-6h24p" Oct 31 00:06:03.112679 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:03.112659 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f032e374-6050-4177-b404-a4f3e3bc50cf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-d32a7cc0-6h24p\" (UID: \"f032e374-6050-4177-b404-a4f3e3bc50cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d32a7cc0-6h24p" Oct 31 00:06:03.173493 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:03.173461 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwl6r\" (UniqueName: \"kubernetes.io/projected/f032e374-6050-4177-b404-a4f3e3bc50cf-kube-api-access-fwl6r\") pod \"maas-api-revoke-keys-e2e-migrate-d32a7cc0-6h24p\" (UID: \"f032e374-6050-4177-b404-a4f3e3bc50cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d32a7cc0-6h24p" Oct 31 00:06:03.225073 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:03.224959 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:06:03.257141 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:03.257106 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d32a7cc0-6h24p" Oct 31 00:06:03.807504 ip-10-0-141-180 kubenswrapper[4927]: W1031 00:06:03.807465 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf032e374_6050_4177_b404_a4f3e3bc50cf.slice/crio-babeb9eb431b3167718f5829a48c2f39b8fefb0e0ed0b70bff9e15e07ecdcb5a WatchSource:0}: Error finding container babeb9eb431b3167718f5829a48c2f39b8fefb0e0ed0b70bff9e15e07ecdcb5a: Status 404 returned error can't find the container with id babeb9eb431b3167718f5829a48c2f39b8fefb0e0ed0b70bff9e15e07ecdcb5a Oct 31 00:06:03.816932 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:03.816901 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d32a7cc0-6h24p"] Oct 31 00:06:03.978527 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:03.978491 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d32a7cc0-6h24p" event={"ID":"f032e374-6050-4177-b404-a4f3e3bc50cf","Type":"ContainerStarted","Data":"babeb9eb431b3167718f5829a48c2f39b8fefb0e0ed0b70bff9e15e07ecdcb5a"} Oct 31 00:06:04.989508 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:04.989418 4927 generic.go:356] "Generic (PLEG): container finished" podID="f032e374-6050-4177-b404-a4f3e3bc50cf" containerID="45a7cfbae4b2f81551d80e5ce3b2b56559aa77672d1ed322ad5e28483bbfa6d9" exitCode=0 Oct 31 00:06:04.989878 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:04.989503 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d32a7cc0-6h24p" event={"ID":"f032e374-6050-4177-b404-a4f3e3bc50cf","Type":"ContainerDied","Data":"45a7cfbae4b2f81551d80e5ce3b2b56559aa77672d1ed322ad5e28483bbfa6d9"} Oct 31 00:06:06.220400 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:06.220372 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d32a7cc0-6h24p" Oct 31 00:06:06.369292 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:06.369199 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f032e374-6050-4177-b404-a4f3e3bc50cf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f032e374-6050-4177-b404-a4f3e3bc50cf-maas-api-openshift-ca\") pod \"f032e374-6050-4177-b404-a4f3e3bc50cf\" (UID: \"f032e374-6050-4177-b404-a4f3e3bc50cf\") " Oct 31 00:06:06.369529 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:06.369348 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f032e374-6050-4177-b404-a4f3e3bc50cf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f032e374-6050-4177-b404-a4f3e3bc50cf-maas-api-xks-ca\") pod \"f032e374-6050-4177-b404-a4f3e3bc50cf\" (UID: \"f032e374-6050-4177-b404-a4f3e3bc50cf\") " Oct 31 00:06:06.369663 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:06.369636 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f032e374-6050-4177-b404-a4f3e3bc50cf-maas-api-openshift-ca" pod "f032e374-6050-4177-b404-a4f3e3bc50cf" (UID: "f032e374-6050-4177-b404-a4f3e3bc50cf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:06:06.369731 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:06.369649 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f032e374-6050-4177-b404-a4f3e3bc50cf-kube-api-access-fwl6r\" (UniqueName: \"kubernetes.io/projected/f032e374-6050-4177-b404-a4f3e3bc50cf-kube-api-access-fwl6r\") pod \"f032e374-6050-4177-b404-a4f3e3bc50cf\" (UID: \"f032e374-6050-4177-b404-a4f3e3bc50cf\") " Oct 31 00:06:06.370493 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:06.370462 4927 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f032e374-6050-4177-b404-a4f3e3bc50cf-maas-api-openshift-ca\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:06:06.371775 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:06.371750 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f032e374-6050-4177-b404-a4f3e3bc50cf-maas-api-xks-ca" pod "f032e374-6050-4177-b404-a4f3e3bc50cf" (UID: "f032e374-6050-4177-b404-a4f3e3bc50cf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:06:06.371870 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:06.371838 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f032e374-6050-4177-b404-a4f3e3bc50cf-kube-api-access-fwl6r" pod "f032e374-6050-4177-b404-a4f3e3bc50cf" (UID: "f032e374-6050-4177-b404-a4f3e3bc50cf"). InnerVolumeSpecName "kube-api-access-fwl6r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:06:06.471504 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:06.471475 4927 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f032e374-6050-4177-b404-a4f3e3bc50cf-maas-api-xks-ca\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:06:06.471504 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:06.471502 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fwl6r\" (UniqueName: \"kubernetes.io/projected/f032e374-6050-4177-b404-a4f3e3bc50cf-kube-api-access-fwl6r\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:06:07.014634 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:07.014555 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d32a7cc0-6h24p" Oct 31 00:06:07.014817 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:07.014557 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d32a7cc0-6h24p" event={"ID":"f032e374-6050-4177-b404-a4f3e3bc50cf","Type":"ContainerDied","Data":"babeb9eb431b3167718f5829a48c2f39b8fefb0e0ed0b70bff9e15e07ecdcb5a"} Oct 31 00:06:07.014817 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:07.014729 4927 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="babeb9eb431b3167718f5829a48c2f39b8fefb0e0ed0b70bff9e15e07ecdcb5a" Oct 31 00:06:07.411616 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:07.411581 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:06:08.553317 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:08.553277 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:06:08.899920 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:08.899787 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:06:09.197360 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:09.194576 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:06:09.745354 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:09.745050 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8e7d0098-mjkns"] Oct 31 00:06:09.797398 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:09.785122 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:06:09.844384 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:09.829176 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8e7d0098-mjkns"] Oct 31 00:06:09.844384 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:09.832424 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-8e7d0098-openshift-default-7c65b994b7-tc65h"] Oct 31 00:06:09.844384 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:09.832697 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-8e7d0098-openshift-default-7c65b994b7-tc65h" podUID="d357695b-ec8c-435b-aeea-35e7409bfdd5" containerName="istio-proxy" containerID="cri-o://6c4e481c3c0e8f8e4d6055719ee6fac0544e963ac2f1ff1fd355da3788b367bc" gracePeriod=30 Oct 31 00:06:10.231247 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:10.231105 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:06:10.395763 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:10.395723 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:06:10.556399 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:10.556228 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:06:10.665239 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:10.665208 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0d96587a-fdvr4"] Oct 31 00:06:10.671004 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:10.670974 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0d96587a-fdvr4"] Oct 31 00:06:11.019435 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:11.019394 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-09efcb6d-openshift-default-7c46fd677c-6wtzg"] Oct 31 00:06:11.021262 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:11.021231 4927 cpu_manager.go:414] "containerMap: removing container" podUID="f032e374-6050-4177-b404-a4f3e3bc50cf" containerName="revoke-keys" Oct 31 00:06:11.021515 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:11.021497 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f032e374-6050-4177-b404-a4f3e3bc50cf" containerName="revoke-keys" Oct 31 00:06:11.021932 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:11.021915 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="f032e374-6050-4177-b404-a4f3e3bc50cf" containerName="revoke-keys" Oct 31 00:06:11.082417 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:11.081926 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:06:11.375441 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:11.345592 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-09efcb6d-openshift-default-7c46fd677c-6wtzg"] Oct 31 00:06:11.375441 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:11.353462 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-09efcb6d-openshift-default-7c46fd677c-6wtzg" Oct 31 00:06:11.411014 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:11.409791 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vv2n\" (UniqueName: \"kubernetes.io/projected/d403f39c-fdf7-4a18-b6c6-60425475307d-kube-api-access-2vv2n\") pod \"e2e-derive-09efcb6d-openshift-default-7c46fd677c-6wtzg\" (UID: \"d403f39c-fdf7-4a18-b6c6-60425475307d\") " pod="openshift-ingress/e2e-derive-09efcb6d-openshift-default-7c46fd677c-6wtzg" Oct 31 00:06:11.411014 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:11.410050 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d403f39c-fdf7-4a18-b6c6-60425475307d-istiod-ca-cert\") pod \"e2e-derive-09efcb6d-openshift-default-7c46fd677c-6wtzg\" (UID: \"d403f39c-fdf7-4a18-b6c6-60425475307d\") " pod="openshift-ingress/e2e-derive-09efcb6d-openshift-default-7c46fd677c-6wtzg" Oct 31 00:06:11.411014 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:11.410123 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d403f39c-fdf7-4a18-b6c6-60425475307d-workload-certs\") pod \"e2e-derive-09efcb6d-openshift-default-7c46fd677c-6wtzg\" (UID: \"d403f39c-fdf7-4a18-b6c6-60425475307d\") " pod="openshift-ingress/e2e-derive-09efcb6d-openshift-default-7c46fd677c-6wtzg" Oct 31 00:06:11.411014 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:11.410147 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d403f39c-fdf7-4a18-b6c6-60425475307d-istio-data\") pod \"e2e-derive-09efcb6d-openshift-default-7c46fd677c-6wtzg\" (UID: \"d403f39c-fdf7-4a18-b6c6-60425475307d\") " pod="openshift-ingress/e2e-derive-09efcb6d-openshift-default-7c46fd677c-6wtzg" Oct 31 00:06:11.411014 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:11.410175 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d403f39c-fdf7-4a18-b6c6-60425475307d-workload-socket\") pod \"e2e-derive-09efcb6d-openshift-default-7c46fd677c-6wtzg\" (UID: \"d403f39c-fdf7-4a18-b6c6-60425475307d\") " pod="openshift-ingress/e2e-derive-09efcb6d-openshift-default-7c46fd677c-6wtzg" Oct 31 00:06:11.411014 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:11.410291 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d403f39c-fdf7-4a18-b6c6-60425475307d-credential-socket\") pod \"e2e-derive-09efcb6d-openshift-default-7c46fd677c-6wtzg\" (UID: \"d403f39c-fdf7-4a18-b6c6-60425475307d\") " pod="openshift-ingress/e2e-derive-09efcb6d-openshift-default-7c46fd677c-6wtzg" Oct 31 00:06:11.411014 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:11.410458 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d403f39c-fdf7-4a18-b6c6-60425475307d-istio-podinfo\") pod \"e2e-derive-09efcb6d-openshift-default-7c46fd677c-6wtzg\" (UID: \"d403f39c-fdf7-4a18-b6c6-60425475307d\") " pod="openshift-ingress/e2e-derive-09efcb6d-openshift-default-7c46fd677c-6wtzg" Oct 31 00:06:11.411014 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:11.410507 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d403f39c-fdf7-4a18-b6c6-60425475307d-istio-envoy\") pod \"e2e-derive-09efcb6d-openshift-default-7c46fd677c-6wtzg\" (UID: \"d403f39c-fdf7-4a18-b6c6-60425475307d\") " pod="openshift-ingress/e2e-derive-09efcb6d-openshift-default-7c46fd677c-6wtzg" Oct 31 00:06:11.411014 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:11.410557 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d403f39c-fdf7-4a18-b6c6-60425475307d-istio-token\") pod \"e2e-derive-09efcb6d-openshift-default-7c46fd677c-6wtzg\" (UID: \"d403f39c-fdf7-4a18-b6c6-60425475307d\") " pod="openshift-ingress/e2e-derive-09efcb6d-openshift-default-7c46fd677c-6wtzg" Oct 31 00:06:11.459470 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:11.458349 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-09efcb6d-openshift-default-dockercfg-l8xvr\"" Oct 31 00:06:11.516393 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:11.514968 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d403f39c-fdf7-4a18-b6c6-60425475307d-istiod-ca-cert\") pod \"e2e-derive-09efcb6d-openshift-default-7c46fd677c-6wtzg\" (UID: \"d403f39c-fdf7-4a18-b6c6-60425475307d\") " pod="openshift-ingress/e2e-derive-09efcb6d-openshift-default-7c46fd677c-6wtzg" Oct 31 00:06:11.516393 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:11.515052 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d403f39c-fdf7-4a18-b6c6-60425475307d-workload-certs\") pod \"e2e-derive-09efcb6d-openshift-default-7c46fd677c-6wtzg\" (UID: \"d403f39c-fdf7-4a18-b6c6-60425475307d\") " pod="openshift-ingress/e2e-derive-09efcb6d-openshift-default-7c46fd677c-6wtzg" Oct 31 00:06:11.516393 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:11.515081 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d403f39c-fdf7-4a18-b6c6-60425475307d-istio-data\") pod \"e2e-derive-09efcb6d-openshift-default-7c46fd677c-6wtzg\" (UID: \"d403f39c-fdf7-4a18-b6c6-60425475307d\") " pod="openshift-ingress/e2e-derive-09efcb6d-openshift-default-7c46fd677c-6wtzg" Oct 31 00:06:11.516393 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:11.515105 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d403f39c-fdf7-4a18-b6c6-60425475307d-workload-socket\") pod \"e2e-derive-09efcb6d-openshift-default-7c46fd677c-6wtzg\" (UID: \"d403f39c-fdf7-4a18-b6c6-60425475307d\") " pod="openshift-ingress/e2e-derive-09efcb6d-openshift-default-7c46fd677c-6wtzg" Oct 31 00:06:11.516393 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:11.515210 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d403f39c-fdf7-4a18-b6c6-60425475307d-credential-socket\") pod \"e2e-derive-09efcb6d-openshift-default-7c46fd677c-6wtzg\" (UID: \"d403f39c-fdf7-4a18-b6c6-60425475307d\") " pod="openshift-ingress/e2e-derive-09efcb6d-openshift-default-7c46fd677c-6wtzg" Oct 31 00:06:11.516393 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:11.515276 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d403f39c-fdf7-4a18-b6c6-60425475307d-istio-podinfo\") pod \"e2e-derive-09efcb6d-openshift-default-7c46fd677c-6wtzg\" (UID: \"d403f39c-fdf7-4a18-b6c6-60425475307d\") " pod="openshift-ingress/e2e-derive-09efcb6d-openshift-default-7c46fd677c-6wtzg" Oct 31 00:06:11.516393 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:11.515317 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d403f39c-fdf7-4a18-b6c6-60425475307d-istio-envoy\") pod \"e2e-derive-09efcb6d-openshift-default-7c46fd677c-6wtzg\" (UID: \"d403f39c-fdf7-4a18-b6c6-60425475307d\") " pod="openshift-ingress/e2e-derive-09efcb6d-openshift-default-7c46fd677c-6wtzg" Oct 31 00:06:11.516393 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:11.515385 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d403f39c-fdf7-4a18-b6c6-60425475307d-istio-token\") pod \"e2e-derive-09efcb6d-openshift-default-7c46fd677c-6wtzg\" (UID: \"d403f39c-fdf7-4a18-b6c6-60425475307d\") " pod="openshift-ingress/e2e-derive-09efcb6d-openshift-default-7c46fd677c-6wtzg" Oct 31 00:06:11.516393 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:11.515474 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vv2n\" (UniqueName: \"kubernetes.io/projected/d403f39c-fdf7-4a18-b6c6-60425475307d-kube-api-access-2vv2n\") pod \"e2e-derive-09efcb6d-openshift-default-7c46fd677c-6wtzg\" (UID: \"d403f39c-fdf7-4a18-b6c6-60425475307d\") " pod="openshift-ingress/e2e-derive-09efcb6d-openshift-default-7c46fd677c-6wtzg" Oct 31 00:06:11.527870 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:11.526165 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d403f39c-fdf7-4a18-b6c6-60425475307d-istiod-ca-cert\") pod \"e2e-derive-09efcb6d-openshift-default-7c46fd677c-6wtzg\" (UID: \"d403f39c-fdf7-4a18-b6c6-60425475307d\") " pod="openshift-ingress/e2e-derive-09efcb6d-openshift-default-7c46fd677c-6wtzg" Oct 31 00:06:11.527870 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:11.526845 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d403f39c-fdf7-4a18-b6c6-60425475307d-credential-socket\") pod \"e2e-derive-09efcb6d-openshift-default-7c46fd677c-6wtzg\" (UID: \"d403f39c-fdf7-4a18-b6c6-60425475307d\") " pod="openshift-ingress/e2e-derive-09efcb6d-openshift-default-7c46fd677c-6wtzg" Oct 31 00:06:11.527870 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:11.527182 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d403f39c-fdf7-4a18-b6c6-60425475307d-workload-certs\") pod \"e2e-derive-09efcb6d-openshift-default-7c46fd677c-6wtzg\" (UID: \"d403f39c-fdf7-4a18-b6c6-60425475307d\") " pod="openshift-ingress/e2e-derive-09efcb6d-openshift-default-7c46fd677c-6wtzg" Oct 31 00:06:11.527870 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:11.527446 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d403f39c-fdf7-4a18-b6c6-60425475307d-istio-data\") pod \"e2e-derive-09efcb6d-openshift-default-7c46fd677c-6wtzg\" (UID: \"d403f39c-fdf7-4a18-b6c6-60425475307d\") " pod="openshift-ingress/e2e-derive-09efcb6d-openshift-default-7c46fd677c-6wtzg" Oct 31 00:06:11.527870 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:11.527684 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d403f39c-fdf7-4a18-b6c6-60425475307d-workload-socket\") pod \"e2e-derive-09efcb6d-openshift-default-7c46fd677c-6wtzg\" (UID: \"d403f39c-fdf7-4a18-b6c6-60425475307d\") " pod="openshift-ingress/e2e-derive-09efcb6d-openshift-default-7c46fd677c-6wtzg" Oct 31 00:06:11.533096 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:11.533064 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:06:11.549409 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:11.536965 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d403f39c-fdf7-4a18-b6c6-60425475307d-istio-envoy\") pod \"e2e-derive-09efcb6d-openshift-default-7c46fd677c-6wtzg\" (UID: \"d403f39c-fdf7-4a18-b6c6-60425475307d\") " pod="openshift-ingress/e2e-derive-09efcb6d-openshift-default-7c46fd677c-6wtzg" Oct 31 00:06:11.587635 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:11.587584 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="27a0d6f4-fbe8-4bdd-a8cb-7a413d3939ce" path="/var/lib/kubelet/pods/27a0d6f4-fbe8-4bdd-a8cb-7a413d3939ce/volumes" Oct 31 00:06:11.588755 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:11.588714 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a32ae26c-e389-4784-979e-f6b5e87e56dc" path="/var/lib/kubelet/pods/a32ae26c-e389-4784-979e-f6b5e87e56dc/volumes" Oct 31 00:06:11.658688 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:11.649399 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vv2n\" (UniqueName: \"kubernetes.io/projected/d403f39c-fdf7-4a18-b6c6-60425475307d-kube-api-access-2vv2n\") pod \"e2e-derive-09efcb6d-openshift-default-7c46fd677c-6wtzg\" (UID: \"d403f39c-fdf7-4a18-b6c6-60425475307d\") " pod="openshift-ingress/e2e-derive-09efcb6d-openshift-default-7c46fd677c-6wtzg" Oct 31 00:06:11.658688 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:11.650836 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d403f39c-fdf7-4a18-b6c6-60425475307d-istio-podinfo\") pod \"e2e-derive-09efcb6d-openshift-default-7c46fd677c-6wtzg\" (UID: \"d403f39c-fdf7-4a18-b6c6-60425475307d\") " pod="openshift-ingress/e2e-derive-09efcb6d-openshift-default-7c46fd677c-6wtzg" Oct 31 00:06:11.666866 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:11.665669 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d403f39c-fdf7-4a18-b6c6-60425475307d-istio-token\") pod \"e2e-derive-09efcb6d-openshift-default-7c46fd677c-6wtzg\" (UID: \"d403f39c-fdf7-4a18-b6c6-60425475307d\") " pod="openshift-ingress/e2e-derive-09efcb6d-openshift-default-7c46fd677c-6wtzg" Oct 31 00:06:11.881310 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:11.881277 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-09efcb6d-openshift-default-7c46fd677c-6wtzg" Oct 31 00:06:12.231753 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:12.231673 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-0d96587a-openshift-default-bcfbd49cc-xrrp4"] Oct 31 00:06:12.232157 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:12.231900 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-0d96587a-openshift-default-bcfbd49cc-xrrp4" podUID="d310a62f-9f54-4000-b280-0581fd91192f" containerName="istio-proxy" containerID="cri-o://056e95e3b188a1fabbeb31f46aef4d2dc9c73b6713ef16e37291388071c7365d" gracePeriod=30 Oct 31 00:06:12.288020 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:12.287984 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:06:12.498391 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:12.492960 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:06:12.502353 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:12.499588 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:06:12.502576 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:12.502544 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:06:12.565784 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:12.565232 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-09efcb6d-openshift-default-7c46fd677c-6wtzg"] Oct 31 00:06:12.678584 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:12.678545 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-09efcb6d-openshift-default-7c46fd677c-6wtzg" event={"ID":"d403f39c-fdf7-4a18-b6c6-60425475307d","Type":"ContainerStarted","Data":"d005f23958edc1970c4d32bfd6721a0b6376a106b71858f2a2d7e622d8168655"} Oct 31 00:06:12.875211 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:12.875177 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:06:13.427588 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:13.427553 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-7c8802f8-openshift-default-74499c95cc-v5qrl"] Oct 31 00:06:13.432499 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:13.432476 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-7c8802f8-openshift-default-74499c95cc-v5qrl" Oct 31 00:06:13.529354 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:13.524238 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85533e83-d620-4bd3-8290-408d0c41164c-istio-data\") pod \"e2e-ait-7c8802f8-openshift-default-74499c95cc-v5qrl\" (UID: \"85533e83-d620-4bd3-8290-408d0c41164c\") " pod="openshift-ingress/e2e-ait-7c8802f8-openshift-default-74499c95cc-v5qrl" Oct 31 00:06:13.529354 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:13.524296 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85533e83-d620-4bd3-8290-408d0c41164c-credential-socket\") pod \"e2e-ait-7c8802f8-openshift-default-74499c95cc-v5qrl\" (UID: \"85533e83-d620-4bd3-8290-408d0c41164c\") " pod="openshift-ingress/e2e-ait-7c8802f8-openshift-default-74499c95cc-v5qrl" Oct 31 00:06:13.529354 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:13.524398 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t78vf\" (UniqueName: \"kubernetes.io/projected/85533e83-d620-4bd3-8290-408d0c41164c-kube-api-access-t78vf\") pod \"e2e-ait-7c8802f8-openshift-default-74499c95cc-v5qrl\" (UID: \"85533e83-d620-4bd3-8290-408d0c41164c\") " pod="openshift-ingress/e2e-ait-7c8802f8-openshift-default-74499c95cc-v5qrl" Oct 31 00:06:13.529354 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:13.524429 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85533e83-d620-4bd3-8290-408d0c41164c-istiod-ca-cert\") pod \"e2e-ait-7c8802f8-openshift-default-74499c95cc-v5qrl\" (UID: \"85533e83-d620-4bd3-8290-408d0c41164c\") " pod="openshift-ingress/e2e-ait-7c8802f8-openshift-default-74499c95cc-v5qrl" Oct 31 00:06:13.529354 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:13.524536 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85533e83-d620-4bd3-8290-408d0c41164c-workload-certs\") pod \"e2e-ait-7c8802f8-openshift-default-74499c95cc-v5qrl\" (UID: \"85533e83-d620-4bd3-8290-408d0c41164c\") " pod="openshift-ingress/e2e-ait-7c8802f8-openshift-default-74499c95cc-v5qrl" Oct 31 00:06:13.529354 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:13.524622 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85533e83-d620-4bd3-8290-408d0c41164c-istio-podinfo\") pod \"e2e-ait-7c8802f8-openshift-default-74499c95cc-v5qrl\" (UID: \"85533e83-d620-4bd3-8290-408d0c41164c\") " pod="openshift-ingress/e2e-ait-7c8802f8-openshift-default-74499c95cc-v5qrl" Oct 31 00:06:13.529354 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:13.524647 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85533e83-d620-4bd3-8290-408d0c41164c-istio-token\") pod \"e2e-ait-7c8802f8-openshift-default-74499c95cc-v5qrl\" (UID: \"85533e83-d620-4bd3-8290-408d0c41164c\") " pod="openshift-ingress/e2e-ait-7c8802f8-openshift-default-74499c95cc-v5qrl" Oct 31 00:06:13.529354 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:13.524724 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85533e83-d620-4bd3-8290-408d0c41164c-workload-socket\") pod \"e2e-ait-7c8802f8-openshift-default-74499c95cc-v5qrl\" (UID: \"85533e83-d620-4bd3-8290-408d0c41164c\") " pod="openshift-ingress/e2e-ait-7c8802f8-openshift-default-74499c95cc-v5qrl" Oct 31 00:06:13.529354 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:13.524769 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85533e83-d620-4bd3-8290-408d0c41164c-istio-envoy\") pod \"e2e-ait-7c8802f8-openshift-default-74499c95cc-v5qrl\" (UID: \"85533e83-d620-4bd3-8290-408d0c41164c\") " pod="openshift-ingress/e2e-ait-7c8802f8-openshift-default-74499c95cc-v5qrl" Oct 31 00:06:13.529354 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:13.525895 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-7c8802f8-openshift-default-dockercfg-2czxb\"" Oct 31 00:06:13.664183 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:13.626542 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-7c8802f8-openshift-default-74499c95cc-v5qrl"] Oct 31 00:06:13.752069 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:13.750568 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:06:13.765614 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:13.760801 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85533e83-d620-4bd3-8290-408d0c41164c-workload-certs\") pod \"e2e-ait-7c8802f8-openshift-default-74499c95cc-v5qrl\" (UID: \"85533e83-d620-4bd3-8290-408d0c41164c\") " pod="openshift-ingress/e2e-ait-7c8802f8-openshift-default-74499c95cc-v5qrl" Oct 31 00:06:13.766559 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:13.766528 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85533e83-d620-4bd3-8290-408d0c41164c-istio-podinfo\") pod \"e2e-ait-7c8802f8-openshift-default-74499c95cc-v5qrl\" (UID: \"85533e83-d620-4bd3-8290-408d0c41164c\") " pod="openshift-ingress/e2e-ait-7c8802f8-openshift-default-74499c95cc-v5qrl" Oct 31 00:06:13.766723 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:13.766704 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85533e83-d620-4bd3-8290-408d0c41164c-istio-token\") pod \"e2e-ait-7c8802f8-openshift-default-74499c95cc-v5qrl\" (UID: \"85533e83-d620-4bd3-8290-408d0c41164c\") " pod="openshift-ingress/e2e-ait-7c8802f8-openshift-default-74499c95cc-v5qrl" Oct 31 00:06:13.766971 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:13.766950 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85533e83-d620-4bd3-8290-408d0c41164c-workload-socket\") pod \"e2e-ait-7c8802f8-openshift-default-74499c95cc-v5qrl\" (UID: \"85533e83-d620-4bd3-8290-408d0c41164c\") " pod="openshift-ingress/e2e-ait-7c8802f8-openshift-default-74499c95cc-v5qrl" Oct 31 00:06:13.767135 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:13.767116 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85533e83-d620-4bd3-8290-408d0c41164c-istio-envoy\") pod \"e2e-ait-7c8802f8-openshift-default-74499c95cc-v5qrl\" (UID: \"85533e83-d620-4bd3-8290-408d0c41164c\") " pod="openshift-ingress/e2e-ait-7c8802f8-openshift-default-74499c95cc-v5qrl" Oct 31 00:06:13.767418 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:13.767399 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85533e83-d620-4bd3-8290-408d0c41164c-istio-data\") pod \"e2e-ait-7c8802f8-openshift-default-74499c95cc-v5qrl\" (UID: \"85533e83-d620-4bd3-8290-408d0c41164c\") " pod="openshift-ingress/e2e-ait-7c8802f8-openshift-default-74499c95cc-v5qrl" Oct 31 00:06:13.768542 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:13.768519 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85533e83-d620-4bd3-8290-408d0c41164c-credential-socket\") pod \"e2e-ait-7c8802f8-openshift-default-74499c95cc-v5qrl\" (UID: \"85533e83-d620-4bd3-8290-408d0c41164c\") " pod="openshift-ingress/e2e-ait-7c8802f8-openshift-default-74499c95cc-v5qrl" Oct 31 00:06:13.768920 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:13.768896 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t78vf\" (UniqueName: \"kubernetes.io/projected/85533e83-d620-4bd3-8290-408d0c41164c-kube-api-access-t78vf\") pod \"e2e-ait-7c8802f8-openshift-default-74499c95cc-v5qrl\" (UID: \"85533e83-d620-4bd3-8290-408d0c41164c\") " pod="openshift-ingress/e2e-ait-7c8802f8-openshift-default-74499c95cc-v5qrl" Oct 31 00:06:13.769097 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:13.769070 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85533e83-d620-4bd3-8290-408d0c41164c-istiod-ca-cert\") pod \"e2e-ait-7c8802f8-openshift-default-74499c95cc-v5qrl\" (UID: \"85533e83-d620-4bd3-8290-408d0c41164c\") " pod="openshift-ingress/e2e-ait-7c8802f8-openshift-default-74499c95cc-v5qrl" Oct 31 00:06:13.771100 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:13.770592 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85533e83-d620-4bd3-8290-408d0c41164c-istiod-ca-cert\") pod \"e2e-ait-7c8802f8-openshift-default-74499c95cc-v5qrl\" (UID: \"85533e83-d620-4bd3-8290-408d0c41164c\") " pod="openshift-ingress/e2e-ait-7c8802f8-openshift-default-74499c95cc-v5qrl" Oct 31 00:06:13.772107 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:13.770773 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85533e83-d620-4bd3-8290-408d0c41164c-workload-socket\") pod \"e2e-ait-7c8802f8-openshift-default-74499c95cc-v5qrl\" (UID: \"85533e83-d620-4bd3-8290-408d0c41164c\") " pod="openshift-ingress/e2e-ait-7c8802f8-openshift-default-74499c95cc-v5qrl" Oct 31 00:06:13.772242 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:13.771507 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85533e83-d620-4bd3-8290-408d0c41164c-credential-socket\") pod \"e2e-ait-7c8802f8-openshift-default-74499c95cc-v5qrl\" (UID: \"85533e83-d620-4bd3-8290-408d0c41164c\") " pod="openshift-ingress/e2e-ait-7c8802f8-openshift-default-74499c95cc-v5qrl" Oct 31 00:06:13.772368 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:13.767726 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85533e83-d620-4bd3-8290-408d0c41164c-workload-certs\") pod \"e2e-ait-7c8802f8-openshift-default-74499c95cc-v5qrl\" (UID: \"85533e83-d620-4bd3-8290-408d0c41164c\") " pod="openshift-ingress/e2e-ait-7c8802f8-openshift-default-74499c95cc-v5qrl" Oct 31 00:06:13.772488 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:13.772016 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85533e83-d620-4bd3-8290-408d0c41164c-istio-data\") pod \"e2e-ait-7c8802f8-openshift-default-74499c95cc-v5qrl\" (UID: \"85533e83-d620-4bd3-8290-408d0c41164c\") " pod="openshift-ingress/e2e-ait-7c8802f8-openshift-default-74499c95cc-v5qrl" Oct 31 00:06:13.824959 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:13.824908 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85533e83-d620-4bd3-8290-408d0c41164c-istio-podinfo\") pod \"e2e-ait-7c8802f8-openshift-default-74499c95cc-v5qrl\" (UID: \"85533e83-d620-4bd3-8290-408d0c41164c\") " pod="openshift-ingress/e2e-ait-7c8802f8-openshift-default-74499c95cc-v5qrl" Oct 31 00:06:13.831578 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:13.831549 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85533e83-d620-4bd3-8290-408d0c41164c-istio-token\") pod \"e2e-ait-7c8802f8-openshift-default-74499c95cc-v5qrl\" (UID: \"85533e83-d620-4bd3-8290-408d0c41164c\") " pod="openshift-ingress/e2e-ait-7c8802f8-openshift-default-74499c95cc-v5qrl" Oct 31 00:06:13.874408 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:13.869757 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-09efcb6d-openshift-default-7c46fd677c-6wtzg" event={"ID":"d403f39c-fdf7-4a18-b6c6-60425475307d","Type":"ContainerStarted","Data":"813b90da108fa46f8396f51017fb233e30c03277f9b21df73a24237ba27c8b95"} Oct 31 00:06:13.888694 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:13.888659 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-09efcb6d-openshift-default-7c46fd677c-6wtzg" Oct 31 00:06:13.901095 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:13.901056 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-09efcb6d-openshift-default-7c46fd677c-6wtzg" Oct 31 00:06:13.951381 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:13.951344 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:06:14.043082 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:14.009266 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t78vf\" (UniqueName: \"kubernetes.io/projected/85533e83-d620-4bd3-8290-408d0c41164c-kube-api-access-t78vf\") pod \"e2e-ait-7c8802f8-openshift-default-74499c95cc-v5qrl\" (UID: \"85533e83-d620-4bd3-8290-408d0c41164c\") " pod="openshift-ingress/e2e-ait-7c8802f8-openshift-default-74499c95cc-v5qrl" Oct 31 00:06:14.043082 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:14.009781 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85533e83-d620-4bd3-8290-408d0c41164c-istio-envoy\") pod \"e2e-ait-7c8802f8-openshift-default-74499c95cc-v5qrl\" (UID: \"85533e83-d620-4bd3-8290-408d0c41164c\") " pod="openshift-ingress/e2e-ait-7c8802f8-openshift-default-74499c95cc-v5qrl" Oct 31 00:06:14.047400 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:14.047367 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-7c8802f8-openshift-default-74499c95cc-v5qrl" Oct 31 00:06:14.310872 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:14.305058 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-09efcb6d-openshift-default-7c46fd677c-6wtzg" podStartSLOduration=4.305032801 podStartE2EDuration="4.305032801s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:06:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:06:14.011737056 +0000 UTC m=+1487.048067899" watchObservedRunningTime="2026-10-31 00:06:14.305032801 +0000 UTC m=+1487.341363625" Oct 31 00:06:14.456568 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:14.455674 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-d32a7cc0-76fd97d5d6-62rcv"] Oct 31 00:06:14.508512 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:14.508473 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-d32a7cc0-6b7768cb67-ntrqr"] Oct 31 00:06:14.712543 ip-10-0-141-180 kubenswrapper[4927]: W1031 00:06:14.708569 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85533e83_d620_4bd3_8290_408d0c41164c.slice/crio-d9c09cea150c30ffc7688aa7f177d0f9cfa9936a4f8af8604d7e360c6d087dfe WatchSource:0}: Error finding container d9c09cea150c30ffc7688aa7f177d0f9cfa9936a4f8af8604d7e360c6d087dfe: Status 404 returned error can't find the container with id d9c09cea150c30ffc7688aa7f177d0f9cfa9936a4f8af8604d7e360c6d087dfe Oct 31 00:06:14.713868 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:14.713419 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:06:14.714693 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:14.714638 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:06:14.714795 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:14.714709 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:06:14.719461 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:14.719411 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-7c8802f8-openshift-default-74499c95cc-v5qrl"] Oct 31 00:06:14.748851 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:14.748827 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:06:14.797676 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:14.797612 4927 patch_prober.go:29] interesting pod/e2e-ait-0d96587a-openshift-default-bcfbd49cc-xrrp4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 00:06:14.797867 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:14.797840 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-0d96587a-openshift-default-bcfbd49cc-xrrp4" podUID="d310a62f-9f54-4000-b280-0581fd91192f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:06:14.883891 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:14.883854 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-7c8802f8-openshift-default-74499c95cc-v5qrl" event={"ID":"85533e83-d620-4bd3-8290-408d0c41164c","Type":"ContainerStarted","Data":"6459ff1422fee57db06f23be1010e336bf13a79c0d217e76dcff62c9d8f7296c"} Oct 31 00:06:14.884079 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:14.883920 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-7c8802f8-openshift-default-74499c95cc-v5qrl" event={"ID":"85533e83-d620-4bd3-8290-408d0c41164c","Type":"ContainerStarted","Data":"d9c09cea150c30ffc7688aa7f177d0f9cfa9936a4f8af8604d7e360c6d087dfe"} Oct 31 00:06:14.884193 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:14.884171 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-09efcb6d-openshift-default-7c46fd677c-6wtzg" Oct 31 00:06:14.885142 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:14.885118 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-09efcb6d-openshift-default-7c46fd677c-6wtzg" Oct 31 00:06:15.046465 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:15.005447 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:06:15.046465 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:15.045519 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-7c8802f8-openshift-default-74499c95cc-v5qrl" podStartSLOduration=2.045497736 podStartE2EDuration="2.045497736s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:06:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:06:15.034878543 +0000 UTC m=+1488.071209380" watchObservedRunningTime="2026-10-31 00:06:15.045497736 +0000 UTC m=+1488.081828615" Oct 31 00:06:15.068356 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:15.061679 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-7c8802f8-openshift-default-74499c95cc-v5qrl" Oct 31 00:06:15.068356 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:15.063367 4927 patch_prober.go:29] interesting pod/e2e-ait-7c8802f8-openshift-default-74499c95cc-v5qrl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" start-of-body= Oct 31 00:06:15.068356 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:15.063428 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-7c8802f8-openshift-default-74499c95cc-v5qrl" podUID="85533e83-d620-4bd3-8290-408d0c41164c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" Oct 31 00:06:15.610127 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:15.610098 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-8e7d0098-openshift-default-7c65b994b7-tc65h" Oct 31 00:06:15.726950 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:15.726847 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d357695b-ec8c-435b-aeea-35e7409bfdd5-kube-api-access-6j9g8\" (UniqueName: \"kubernetes.io/projected/d357695b-ec8c-435b-aeea-35e7409bfdd5-kube-api-access-6j9g8\") pod \"d357695b-ec8c-435b-aeea-35e7409bfdd5\" (UID: \"d357695b-ec8c-435b-aeea-35e7409bfdd5\") " Oct 31 00:06:15.727147 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:15.726977 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d357695b-ec8c-435b-aeea-35e7409bfdd5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d357695b-ec8c-435b-aeea-35e7409bfdd5-credential-socket\") pod \"d357695b-ec8c-435b-aeea-35e7409bfdd5\" (UID: \"d357695b-ec8c-435b-aeea-35e7409bfdd5\") " Oct 31 00:06:15.727147 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:15.727071 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d357695b-ec8c-435b-aeea-35e7409bfdd5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d357695b-ec8c-435b-aeea-35e7409bfdd5-workload-socket\") pod \"d357695b-ec8c-435b-aeea-35e7409bfdd5\" (UID: \"d357695b-ec8c-435b-aeea-35e7409bfdd5\") " Oct 31 00:06:15.727147 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:15.727106 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d357695b-ec8c-435b-aeea-35e7409bfdd5-istio-token\" (UniqueName: \"kubernetes.io/projected/d357695b-ec8c-435b-aeea-35e7409bfdd5-istio-token\") pod \"d357695b-ec8c-435b-aeea-35e7409bfdd5\" (UID: \"d357695b-ec8c-435b-aeea-35e7409bfdd5\") " Oct 31 00:06:15.727379 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:15.727187 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d357695b-ec8c-435b-aeea-35e7409bfdd5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d357695b-ec8c-435b-aeea-35e7409bfdd5-istio-data\") pod \"d357695b-ec8c-435b-aeea-35e7409bfdd5\" (UID: \"d357695b-ec8c-435b-aeea-35e7409bfdd5\") " Oct 31 00:06:15.727379 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:15.727308 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d357695b-ec8c-435b-aeea-35e7409bfdd5-credential-socket" pod "d357695b-ec8c-435b-aeea-35e7409bfdd5" (UID: "d357695b-ec8c-435b-aeea-35e7409bfdd5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:06:15.727502 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:15.727395 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d357695b-ec8c-435b-aeea-35e7409bfdd5-workload-socket" pod "d357695b-ec8c-435b-aeea-35e7409bfdd5" (UID: "d357695b-ec8c-435b-aeea-35e7409bfdd5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:06:15.727502 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:15.727411 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d357695b-ec8c-435b-aeea-35e7409bfdd5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d357695b-ec8c-435b-aeea-35e7409bfdd5-istio-podinfo\") pod \"d357695b-ec8c-435b-aeea-35e7409bfdd5\" (UID: \"d357695b-ec8c-435b-aeea-35e7409bfdd5\") " Oct 31 00:06:15.727502 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:15.727474 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d357695b-ec8c-435b-aeea-35e7409bfdd5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d357695b-ec8c-435b-aeea-35e7409bfdd5-istiod-ca-cert\") pod \"d357695b-ec8c-435b-aeea-35e7409bfdd5\" (UID: \"d357695b-ec8c-435b-aeea-35e7409bfdd5\") " Oct 31 00:06:15.727655 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:15.727501 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d357695b-ec8c-435b-aeea-35e7409bfdd5-istio-data" pod "d357695b-ec8c-435b-aeea-35e7409bfdd5" (UID: "d357695b-ec8c-435b-aeea-35e7409bfdd5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:06:15.727655 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:15.727561 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d357695b-ec8c-435b-aeea-35e7409bfdd5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d357695b-ec8c-435b-aeea-35e7409bfdd5-workload-certs\") pod \"d357695b-ec8c-435b-aeea-35e7409bfdd5\" (UID: \"d357695b-ec8c-435b-aeea-35e7409bfdd5\") " Oct 31 00:06:15.727655 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:15.727621 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d357695b-ec8c-435b-aeea-35e7409bfdd5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d357695b-ec8c-435b-aeea-35e7409bfdd5-istio-envoy\") pod \"d357695b-ec8c-435b-aeea-35e7409bfdd5\" (UID: \"d357695b-ec8c-435b-aeea-35e7409bfdd5\") " Oct 31 00:06:15.727841 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:15.727820 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d357695b-ec8c-435b-aeea-35e7409bfdd5-workload-certs" pod "d357695b-ec8c-435b-aeea-35e7409bfdd5" (UID: "d357695b-ec8c-435b-aeea-35e7409bfdd5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:06:15.727878 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:15.727858 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d357695b-ec8c-435b-aeea-35e7409bfdd5-istiod-ca-cert" pod "d357695b-ec8c-435b-aeea-35e7409bfdd5" (UID: "d357695b-ec8c-435b-aeea-35e7409bfdd5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:06:15.728551 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:15.728528 4927 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d357695b-ec8c-435b-aeea-35e7409bfdd5-workload-certs\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:06:15.728671 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:15.728554 4927 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d357695b-ec8c-435b-aeea-35e7409bfdd5-credential-socket\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:06:15.728671 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:15.728571 4927 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d357695b-ec8c-435b-aeea-35e7409bfdd5-workload-socket\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:06:15.728671 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:15.728585 4927 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d357695b-ec8c-435b-aeea-35e7409bfdd5-istio-data\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:06:15.728671 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:15.728600 4927 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d357695b-ec8c-435b-aeea-35e7409bfdd5-istiod-ca-cert\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:06:15.729569 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:15.729533 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d357695b-ec8c-435b-aeea-35e7409bfdd5-kube-api-access-6j9g8" pod "d357695b-ec8c-435b-aeea-35e7409bfdd5" (UID: "d357695b-ec8c-435b-aeea-35e7409bfdd5"). InnerVolumeSpecName "kube-api-access-6j9g8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:06:15.730040 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:15.730015 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d357695b-ec8c-435b-aeea-35e7409bfdd5-istio-envoy" pod "d357695b-ec8c-435b-aeea-35e7409bfdd5" (UID: "d357695b-ec8c-435b-aeea-35e7409bfdd5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:06:15.730143 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:15.730116 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d357695b-ec8c-435b-aeea-35e7409bfdd5-istio-podinfo" pod "d357695b-ec8c-435b-aeea-35e7409bfdd5" (UID: "d357695b-ec8c-435b-aeea-35e7409bfdd5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:06:15.730224 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:15.730201 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d357695b-ec8c-435b-aeea-35e7409bfdd5-istio-token" pod "d357695b-ec8c-435b-aeea-35e7409bfdd5" (UID: "d357695b-ec8c-435b-aeea-35e7409bfdd5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:06:15.986425 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:15.967298 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6j9g8\" (UniqueName: \"kubernetes.io/projected/d357695b-ec8c-435b-aeea-35e7409bfdd5-kube-api-access-6j9g8\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:06:15.990004 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:15.986664 4927 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d357695b-ec8c-435b-aeea-35e7409bfdd5-istio-token\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:06:15.990004 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:15.986708 4927 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d357695b-ec8c-435b-aeea-35e7409bfdd5-istio-podinfo\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:06:15.990004 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:15.986721 4927 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d357695b-ec8c-435b-aeea-35e7409bfdd5-istio-envoy\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:06:16.017319 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:16.017282 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:06:16.096838 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:16.096784 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-7c8802f8-openshift-default-74499c95cc-v5qrl" Oct 31 00:06:16.189596 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:16.189548 4927 generic.go:356] "Generic (PLEG): container finished" podID="d357695b-ec8c-435b-aeea-35e7409bfdd5" containerID="6c4e481c3c0e8f8e4d6055719ee6fac0544e963ac2f1ff1fd355da3788b367bc" exitCode=0 Oct 31 00:06:16.193786 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:16.193764 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-8e7d0098-openshift-default-7c65b994b7-tc65h" Oct 31 00:06:16.215368 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:16.193433 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8e7d0098-openshift-default-7c65b994b7-tc65h" event={"ID":"d357695b-ec8c-435b-aeea-35e7409bfdd5","Type":"ContainerDied","Data":"6c4e481c3c0e8f8e4d6055719ee6fac0544e963ac2f1ff1fd355da3788b367bc"} Oct 31 00:06:16.215368 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:16.206148 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8e7d0098-openshift-default-7c65b994b7-tc65h" event={"ID":"d357695b-ec8c-435b-aeea-35e7409bfdd5","Type":"ContainerDied","Data":"4cddccd1fc7515bef4fdfb8f591efc653d0beba2ded7682fdf1356d3be8ae674"} Oct 31 00:06:16.215368 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:16.206182 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-7c8802f8-openshift-default-74499c95cc-v5qrl" Oct 31 00:06:16.215368 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:16.206366 4927 scope.go:122] "RemoveContainer" containerID="6c4e481c3c0e8f8e4d6055719ee6fac0544e963ac2f1ff1fd355da3788b367bc" Oct 31 00:06:16.255102 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:16.251111 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-7c8802f8-openshift-default-74499c95cc-v5qrl" Oct 31 00:06:16.346125 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:16.346090 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:06:16.401366 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:16.398679 4927 scope.go:122] "RemoveContainer" containerID="6c4e481c3c0e8f8e4d6055719ee6fac0544e963ac2f1ff1fd355da3788b367bc" Oct 31 00:06:16.406361 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:06:16.402485 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c4e481c3c0e8f8e4d6055719ee6fac0544e963ac2f1ff1fd355da3788b367bc\": container with ID starting with 6c4e481c3c0e8f8e4d6055719ee6fac0544e963ac2f1ff1fd355da3788b367bc not found: ID does not exist" containerID="6c4e481c3c0e8f8e4d6055719ee6fac0544e963ac2f1ff1fd355da3788b367bc" Oct 31 00:06:16.406361 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:16.402532 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c4e481c3c0e8f8e4d6055719ee6fac0544e963ac2f1ff1fd355da3788b367bc"} err="failed to get container status \"6c4e481c3c0e8f8e4d6055719ee6fac0544e963ac2f1ff1fd355da3788b367bc\": rpc error: code = NotFound desc = could not find container \"6c4e481c3c0e8f8e4d6055719ee6fac0544e963ac2f1ff1fd355da3788b367bc\": container with ID starting with 6c4e481c3c0e8f8e4d6055719ee6fac0544e963ac2f1ff1fd355da3788b367bc not found: ID does not exist" Oct 31 00:06:16.561402 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:16.553322 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:06:16.640540 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:16.640497 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-8e7d0098-openshift-default-7c65b994b7-tc65h"] Oct 31 00:06:16.647040 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:16.647009 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-8e7d0098-openshift-default-7c65b994b7-tc65h"] Oct 31 00:06:17.088248 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:17.087917 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:06:17.092735 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:17.092704 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d357695b-ec8c-435b-aeea-35e7409bfdd5" path="/var/lib/kubelet/pods/d357695b-ec8c-435b-aeea-35e7409bfdd5/volumes" Oct 31 00:06:17.372461 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:17.372413 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:06:17.549918 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:17.549891 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-0d96587a-openshift-default-bcfbd49cc-xrrp4" Oct 31 00:06:17.574473 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:17.574441 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d310a62f-9f54-4000-b280-0581fd91192f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d310a62f-9f54-4000-b280-0581fd91192f-workload-certs\") pod \"d310a62f-9f54-4000-b280-0581fd91192f\" (UID: \"d310a62f-9f54-4000-b280-0581fd91192f\") " Oct 31 00:06:17.574607 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:17.574488 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d310a62f-9f54-4000-b280-0581fd91192f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d310a62f-9f54-4000-b280-0581fd91192f-istio-podinfo\") pod \"d310a62f-9f54-4000-b280-0581fd91192f\" (UID: \"d310a62f-9f54-4000-b280-0581fd91192f\") " Oct 31 00:06:17.574666 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:17.574649 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d310a62f-9f54-4000-b280-0581fd91192f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d310a62f-9f54-4000-b280-0581fd91192f-istio-envoy\") pod \"d310a62f-9f54-4000-b280-0581fd91192f\" (UID: \"d310a62f-9f54-4000-b280-0581fd91192f\") " Oct 31 00:06:17.574718 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:17.574685 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d310a62f-9f54-4000-b280-0581fd91192f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d310a62f-9f54-4000-b280-0581fd91192f-istio-data\") pod \"d310a62f-9f54-4000-b280-0581fd91192f\" (UID: \"d310a62f-9f54-4000-b280-0581fd91192f\") " Oct 31 00:06:17.574770 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:17.574715 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d310a62f-9f54-4000-b280-0581fd91192f-kube-api-access-dmdsb\" (UniqueName: \"kubernetes.io/projected/d310a62f-9f54-4000-b280-0581fd91192f-kube-api-access-dmdsb\") pod \"d310a62f-9f54-4000-b280-0581fd91192f\" (UID: \"d310a62f-9f54-4000-b280-0581fd91192f\") " Oct 31 00:06:17.574770 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:17.574726 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d310a62f-9f54-4000-b280-0581fd91192f-workload-certs" pod "d310a62f-9f54-4000-b280-0581fd91192f" (UID: "d310a62f-9f54-4000-b280-0581fd91192f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:06:17.574770 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:17.574750 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d310a62f-9f54-4000-b280-0581fd91192f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d310a62f-9f54-4000-b280-0581fd91192f-istiod-ca-cert\") pod \"d310a62f-9f54-4000-b280-0581fd91192f\" (UID: \"d310a62f-9f54-4000-b280-0581fd91192f\") " Oct 31 00:06:17.575055 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:17.574973 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d310a62f-9f54-4000-b280-0581fd91192f-istio-data" pod "d310a62f-9f54-4000-b280-0581fd91192f" (UID: "d310a62f-9f54-4000-b280-0581fd91192f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:06:17.575166 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:17.575072 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d310a62f-9f54-4000-b280-0581fd91192f-istio-token\" (UniqueName: \"kubernetes.io/projected/d310a62f-9f54-4000-b280-0581fd91192f-istio-token\") pod \"d310a62f-9f54-4000-b280-0581fd91192f\" (UID: \"d310a62f-9f54-4000-b280-0581fd91192f\") " Oct 31 00:06:17.575166 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:17.575140 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d310a62f-9f54-4000-b280-0581fd91192f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d310a62f-9f54-4000-b280-0581fd91192f-credential-socket\") pod \"d310a62f-9f54-4000-b280-0581fd91192f\" (UID: \"d310a62f-9f54-4000-b280-0581fd91192f\") " Oct 31 00:06:17.575273 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:17.575260 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d310a62f-9f54-4000-b280-0581fd91192f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d310a62f-9f54-4000-b280-0581fd91192f-workload-socket\") pod \"d310a62f-9f54-4000-b280-0581fd91192f\" (UID: \"d310a62f-9f54-4000-b280-0581fd91192f\") " Oct 31 00:06:17.575413 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:17.575383 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d310a62f-9f54-4000-b280-0581fd91192f-istiod-ca-cert" pod "d310a62f-9f54-4000-b280-0581fd91192f" (UID: "d310a62f-9f54-4000-b280-0581fd91192f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:06:17.576459 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:17.576112 4927 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d310a62f-9f54-4000-b280-0581fd91192f-workload-certs\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:06:17.576572 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:17.576474 4927 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d310a62f-9f54-4000-b280-0581fd91192f-istio-data\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:06:17.576572 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:17.576523 4927 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d310a62f-9f54-4000-b280-0581fd91192f-istiod-ca-cert\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:06:17.576789 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:17.576639 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d310a62f-9f54-4000-b280-0581fd91192f-credential-socket" pod "d310a62f-9f54-4000-b280-0581fd91192f" (UID: "d310a62f-9f54-4000-b280-0581fd91192f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:06:17.576789 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:17.576659 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d310a62f-9f54-4000-b280-0581fd91192f-workload-socket" pod "d310a62f-9f54-4000-b280-0581fd91192f" (UID: "d310a62f-9f54-4000-b280-0581fd91192f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:06:17.577296 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:17.577272 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d310a62f-9f54-4000-b280-0581fd91192f-istio-podinfo" pod "d310a62f-9f54-4000-b280-0581fd91192f" (UID: "d310a62f-9f54-4000-b280-0581fd91192f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:06:17.577448 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:17.577420 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d310a62f-9f54-4000-b280-0581fd91192f-istio-envoy" pod "d310a62f-9f54-4000-b280-0581fd91192f" (UID: "d310a62f-9f54-4000-b280-0581fd91192f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:06:17.577525 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:17.577498 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d310a62f-9f54-4000-b280-0581fd91192f-kube-api-access-dmdsb" pod "d310a62f-9f54-4000-b280-0581fd91192f" (UID: "d310a62f-9f54-4000-b280-0581fd91192f"). InnerVolumeSpecName "kube-api-access-dmdsb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:06:17.578098 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:17.578074 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d310a62f-9f54-4000-b280-0581fd91192f-istio-token" pod "d310a62f-9f54-4000-b280-0581fd91192f" (UID: "d310a62f-9f54-4000-b280-0581fd91192f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:06:17.594021 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:17.593992 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:06:17.677738 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:17.677703 4927 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d310a62f-9f54-4000-b280-0581fd91192f-istio-envoy\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:06:17.677738 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:17.677736 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dmdsb\" (UniqueName: \"kubernetes.io/projected/d310a62f-9f54-4000-b280-0581fd91192f-kube-api-access-dmdsb\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:06:17.678170 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:17.677752 4927 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d310a62f-9f54-4000-b280-0581fd91192f-istio-token\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:06:17.678170 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:17.677764 4927 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d310a62f-9f54-4000-b280-0581fd91192f-credential-socket\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:06:17.678170 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:17.677776 4927 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d310a62f-9f54-4000-b280-0581fd91192f-workload-socket\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:06:17.678170 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:17.677788 4927 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d310a62f-9f54-4000-b280-0581fd91192f-istio-podinfo\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:06:18.230899 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:18.230859 4927 generic.go:356] "Generic (PLEG): container finished" podID="d310a62f-9f54-4000-b280-0581fd91192f" containerID="056e95e3b188a1fabbeb31f46aef4d2dc9c73b6713ef16e37291388071c7365d" exitCode=0 Oct 31 00:06:18.231093 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:18.230908 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-0d96587a-openshift-default-bcfbd49cc-xrrp4" Oct 31 00:06:18.231093 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:18.230939 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0d96587a-openshift-default-bcfbd49cc-xrrp4" event={"ID":"d310a62f-9f54-4000-b280-0581fd91192f","Type":"ContainerDied","Data":"056e95e3b188a1fabbeb31f46aef4d2dc9c73b6713ef16e37291388071c7365d"} Oct 31 00:06:18.231093 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:18.230989 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0d96587a-openshift-default-bcfbd49cc-xrrp4" event={"ID":"d310a62f-9f54-4000-b280-0581fd91192f","Type":"ContainerDied","Data":"d331d7d5f45c04477f2bf720d3faa58524ddad8fff465202e2042c77af99fc8e"} Oct 31 00:06:18.231093 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:18.231009 4927 scope.go:122] "RemoveContainer" containerID="056e95e3b188a1fabbeb31f46aef4d2dc9c73b6713ef16e37291388071c7365d" Oct 31 00:06:18.243650 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:18.243634 4927 scope.go:122] "RemoveContainer" containerID="ae5b79a50331596a3b0abc27a755878cbe38e756b0abf2963eab10d18de553a0" Oct 31 00:06:18.254068 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:18.254048 4927 scope.go:122] "RemoveContainer" containerID="056e95e3b188a1fabbeb31f46aef4d2dc9c73b6713ef16e37291388071c7365d" Oct 31 00:06:18.254376 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:06:18.254302 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"056e95e3b188a1fabbeb31f46aef4d2dc9c73b6713ef16e37291388071c7365d\": container with ID starting with 056e95e3b188a1fabbeb31f46aef4d2dc9c73b6713ef16e37291388071c7365d not found: ID does not exist" containerID="056e95e3b188a1fabbeb31f46aef4d2dc9c73b6713ef16e37291388071c7365d" Oct 31 00:06:18.254376 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:18.254345 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"056e95e3b188a1fabbeb31f46aef4d2dc9c73b6713ef16e37291388071c7365d"} err="failed to get container status \"056e95e3b188a1fabbeb31f46aef4d2dc9c73b6713ef16e37291388071c7365d\": rpc error: code = NotFound desc = could not find container \"056e95e3b188a1fabbeb31f46aef4d2dc9c73b6713ef16e37291388071c7365d\": container with ID starting with 056e95e3b188a1fabbeb31f46aef4d2dc9c73b6713ef16e37291388071c7365d not found: ID does not exist" Oct 31 00:06:18.254376 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:18.254370 4927 scope.go:122] "RemoveContainer" containerID="ae5b79a50331596a3b0abc27a755878cbe38e756b0abf2963eab10d18de553a0" Oct 31 00:06:18.254648 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:06:18.254631 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae5b79a50331596a3b0abc27a755878cbe38e756b0abf2963eab10d18de553a0\": container with ID starting with ae5b79a50331596a3b0abc27a755878cbe38e756b0abf2963eab10d18de553a0 not found: ID does not exist" containerID="ae5b79a50331596a3b0abc27a755878cbe38e756b0abf2963eab10d18de553a0" Oct 31 00:06:18.254690 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:18.254653 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae5b79a50331596a3b0abc27a755878cbe38e756b0abf2963eab10d18de553a0"} err="failed to get container status \"ae5b79a50331596a3b0abc27a755878cbe38e756b0abf2963eab10d18de553a0\": rpc error: code = NotFound desc = could not find container \"ae5b79a50331596a3b0abc27a755878cbe38e756b0abf2963eab10d18de553a0\": container with ID starting with ae5b79a50331596a3b0abc27a755878cbe38e756b0abf2963eab10d18de553a0 not found: ID does not exist" Oct 31 00:06:18.408057 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:18.408025 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-0d96587a-openshift-default-bcfbd49cc-xrrp4"] Oct 31 00:06:18.459042 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:18.459004 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-0d96587a-openshift-default-bcfbd49cc-xrrp4"] Oct 31 00:06:19.088920 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:19.088883 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d310a62f-9f54-4000-b280-0581fd91192f" path="/var/lib/kubelet/pods/d310a62f-9f54-4000-b280-0581fd91192f/volumes" Oct 31 00:06:19.377416 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:19.377297 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:06:19.480364 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:19.456565 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-d32a7cc0-76fd97d5d6-62rcv" podUID="d0961830-61e4-4528-a704-82a109b453fb" containerName="payload-processing" containerID="cri-o://49a24120ab046f00ad3aa39f7eb40170389c9d593df058a3552798bf916642c1" gracePeriod=25 Oct 31 00:06:19.574370 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:19.572975 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-d32a7cc0-6b7768cb67-ntrqr" podUID="ea4dd2c2-8b72-4ec5-b43d-5bc1fe96f391" containerName="payload-pre-processing" containerID="cri-o://c3414683b862bd58304224fd39aa282d41e9f177168b9f828f096bf69e01163b" gracePeriod=25 Oct 31 00:06:19.752549 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:19.750624 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:06:19.867998 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:19.867929 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-09efcb6d-c7bfbb5cf-fsvrl"] Oct 31 00:06:19.871055 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:19.869610 4927 cpu_manager.go:414] "containerMap: removing container" podUID="d310a62f-9f54-4000-b280-0581fd91192f" containerName="istio-proxy" Oct 31 00:06:19.871055 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:19.869636 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d310a62f-9f54-4000-b280-0581fd91192f" containerName="istio-proxy" Oct 31 00:06:19.871055 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:19.869665 4927 cpu_manager.go:414] "containerMap: removing container" podUID="d357695b-ec8c-435b-aeea-35e7409bfdd5" containerName="istio-proxy" Oct 31 00:06:19.871055 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:19.869672 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d357695b-ec8c-435b-aeea-35e7409bfdd5" containerName="istio-proxy" Oct 31 00:06:19.871055 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:19.869964 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="d310a62f-9f54-4000-b280-0581fd91192f" containerName="istio-proxy" Oct 31 00:06:19.871055 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:19.869995 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="d310a62f-9f54-4000-b280-0581fd91192f" containerName="istio-proxy" Oct 31 00:06:19.871055 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:19.870006 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="d357695b-ec8c-435b-aeea-35e7409bfdd5" containerName="istio-proxy" Oct 31 00:06:19.875246 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:06:19.875222 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sqv69" containerName="extractor" Oct 31 00:06:19.886951 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:19.876259 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-09efcb6d-c7bfbb5cf-fsvrl"] Oct 31 00:06:19.886951 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:19.876408 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-09efcb6d-c7bfbb5cf-fsvrl" Oct 31 00:06:19.953872 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:19.953809 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-09efcb6d-dockercfg-pjfxv\"" Oct 31 00:06:19.954588 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:19.954488 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-09efcb6d\"" Oct 31 00:06:20.029742 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:20.006817 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37daf0f5-13de-43e2-a2a2-d82036836415-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-09efcb6d-c7bfbb5cf-fsvrl\" (UID: \"37daf0f5-13de-43e2-a2a2-d82036836415\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-09efcb6d-c7bfbb5cf-fsvrl" Oct 31 00:06:20.029742 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:20.006909 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7864m\" (UniqueName: \"kubernetes.io/projected/37daf0f5-13de-43e2-a2a2-d82036836415-kube-api-access-7864m\") pod \"payload-pre-processing-e2e-derive-09efcb6d-c7bfbb5cf-fsvrl\" (UID: \"37daf0f5-13de-43e2-a2a2-d82036836415\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-09efcb6d-c7bfbb5cf-fsvrl" Oct 31 00:06:20.132017 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:20.111044 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37daf0f5-13de-43e2-a2a2-d82036836415-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-09efcb6d-c7bfbb5cf-fsvrl\" (UID: \"37daf0f5-13de-43e2-a2a2-d82036836415\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-09efcb6d-c7bfbb5cf-fsvrl" Oct 31 00:06:20.132017 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:20.131165 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7864m\" (UniqueName: \"kubernetes.io/projected/37daf0f5-13de-43e2-a2a2-d82036836415-kube-api-access-7864m\") pod \"payload-pre-processing-e2e-derive-09efcb6d-c7bfbb5cf-fsvrl\" (UID: \"37daf0f5-13de-43e2-a2a2-d82036836415\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-09efcb6d-c7bfbb5cf-fsvrl" Oct 31 00:06:20.153231 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:20.153189 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37daf0f5-13de-43e2-a2a2-d82036836415-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-09efcb6d-c7bfbb5cf-fsvrl\" (UID: \"37daf0f5-13de-43e2-a2a2-d82036836415\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-09efcb6d-c7bfbb5cf-fsvrl" Oct 31 00:06:20.183035 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:20.182588 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-09efcb6d-699987c7fd-j2v69"] Oct 31 00:06:20.184306 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:20.184209 4927 cpu_manager.go:414] "containerMap: removing container" podUID="d310a62f-9f54-4000-b280-0581fd91192f" containerName="istio-proxy" Oct 31 00:06:20.184306 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:20.184236 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d310a62f-9f54-4000-b280-0581fd91192f" containerName="istio-proxy" Oct 31 00:06:20.210447 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:20.209835 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-09efcb6d-699987c7fd-j2v69" Oct 31 00:06:20.298666 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:20.298242 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7864m\" (UniqueName: \"kubernetes.io/projected/37daf0f5-13de-43e2-a2a2-d82036836415-kube-api-access-7864m\") pod \"payload-pre-processing-e2e-derive-09efcb6d-c7bfbb5cf-fsvrl\" (UID: \"37daf0f5-13de-43e2-a2a2-d82036836415\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-09efcb6d-c7bfbb5cf-fsvrl" Oct 31 00:06:20.308460 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:20.305635 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zqg5j\" (UniqueName: \"kubernetes.io/projected/ee43bf6b-c4d3-46f6-8808-8064e3412906-kube-api-access-zqg5j\") pod \"payload-processing-e2e-derive-09efcb6d-699987c7fd-j2v69\" (UID: \"ee43bf6b-c4d3-46f6-8808-8064e3412906\") " pod="openshift-ingress/payload-processing-e2e-derive-09efcb6d-699987c7fd-j2v69" Oct 31 00:06:20.308460 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:20.305820 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee43bf6b-c4d3-46f6-8808-8064e3412906-plugins-config-volume\") pod \"payload-processing-e2e-derive-09efcb6d-699987c7fd-j2v69\" (UID: \"ee43bf6b-c4d3-46f6-8808-8064e3412906\") " pod="openshift-ingress/payload-processing-e2e-derive-09efcb6d-699987c7fd-j2v69" Oct 31 00:06:20.424050 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:20.407661 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqg5j\" (UniqueName: \"kubernetes.io/projected/ee43bf6b-c4d3-46f6-8808-8064e3412906-kube-api-access-zqg5j\") pod \"payload-processing-e2e-derive-09efcb6d-699987c7fd-j2v69\" (UID: \"ee43bf6b-c4d3-46f6-8808-8064e3412906\") " pod="openshift-ingress/payload-processing-e2e-derive-09efcb6d-699987c7fd-j2v69" Oct 31 00:06:20.449994 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:20.449948 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee43bf6b-c4d3-46f6-8808-8064e3412906-plugins-config-volume\") pod \"payload-processing-e2e-derive-09efcb6d-699987c7fd-j2v69\" (UID: \"ee43bf6b-c4d3-46f6-8808-8064e3412906\") " pod="openshift-ingress/payload-processing-e2e-derive-09efcb6d-699987c7fd-j2v69" Oct 31 00:06:20.451482 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:20.451185 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee43bf6b-c4d3-46f6-8808-8064e3412906-plugins-config-volume\") pod \"payload-processing-e2e-derive-09efcb6d-699987c7fd-j2v69\" (UID: \"ee43bf6b-c4d3-46f6-8808-8064e3412906\") " pod="openshift-ingress/payload-processing-e2e-derive-09efcb6d-699987c7fd-j2v69" Oct 31 00:06:20.549951 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:20.501310 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-09efcb6d-c7bfbb5cf-fsvrl" Oct 31 00:06:20.592407 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:20.563812 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqg5j\" (UniqueName: \"kubernetes.io/projected/ee43bf6b-c4d3-46f6-8808-8064e3412906-kube-api-access-zqg5j\") pod \"payload-processing-e2e-derive-09efcb6d-699987c7fd-j2v69\" (UID: \"ee43bf6b-c4d3-46f6-8808-8064e3412906\") " pod="openshift-ingress/payload-processing-e2e-derive-09efcb6d-699987c7fd-j2v69" Oct 31 00:06:20.656349 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:20.655710 4927 generic.go:356] "Generic (PLEG): container finished" podID="d0961830-61e4-4528-a704-82a109b453fb" containerID="49a24120ab046f00ad3aa39f7eb40170389c9d593df058a3552798bf916642c1" exitCode=0 Oct 31 00:06:20.656349 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:20.656197 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-d32a7cc0-76fd97d5d6-62rcv" event={"ID":"d0961830-61e4-4528-a704-82a109b453fb","Type":"ContainerDied","Data":"49a24120ab046f00ad3aa39f7eb40170389c9d593df058a3552798bf916642c1"} Oct 31 00:06:20.660102 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:20.660058 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-09efcb6d-699987c7fd-j2v69" Oct 31 00:06:20.664220 ip-10-0-141-180 kubenswrapper[4927]: W1031 00:06:20.663552 4927 logging.go:55] [core] [Channel #3088 SubChannel #3089]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.157:50052", ServerName: "10.132.0.157:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.157:50052: connect: connection refused" Oct 31 00:06:20.665585 ip-10-0-141-180 kubenswrapper[4927]: W1031 00:06:20.664994 4927 logging.go:55] [core] [Channel #3090 SubChannel #3091]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.158:50052", ServerName: "10.132.0.158:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.158:50052: connect: connection refused" Oct 31 00:06:20.694488 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:20.694455 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-09efcb6d-699987c7fd-j2v69"] Oct 31 00:06:20.915524 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:20.915495 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-d32a7cc0-76fd97d5d6-62rcv" Oct 31 00:06:20.961281 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:20.961251 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-d32a7cc0-6b7768cb67-ntrqr" Oct 31 00:06:21.003480 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:21.003445 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0961830-61e4-4528-a704-82a109b453fb-kube-api-access-8t8kn\" (UniqueName: \"kubernetes.io/projected/d0961830-61e4-4528-a704-82a109b453fb-kube-api-access-8t8kn\") pod \"d0961830-61e4-4528-a704-82a109b453fb\" (UID: \"d0961830-61e4-4528-a704-82a109b453fb\") " Oct 31 00:06:21.003640 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:21.003506 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea4dd2c2-8b72-4ec5-b43d-5bc1fe96f391-kube-api-access-sb6f7\" (UniqueName: \"kubernetes.io/projected/ea4dd2c2-8b72-4ec5-b43d-5bc1fe96f391-kube-api-access-sb6f7\") pod \"ea4dd2c2-8b72-4ec5-b43d-5bc1fe96f391\" (UID: \"ea4dd2c2-8b72-4ec5-b43d-5bc1fe96f391\") " Oct 31 00:06:21.003714 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:21.003646 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d0961830-61e4-4528-a704-82a109b453fb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0961830-61e4-4528-a704-82a109b453fb-plugins-config-volume\") pod \"d0961830-61e4-4528-a704-82a109b453fb\" (UID: \"d0961830-61e4-4528-a704-82a109b453fb\") " Oct 31 00:06:21.003714 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:21.003696 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea4dd2c2-8b72-4ec5-b43d-5bc1fe96f391-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea4dd2c2-8b72-4ec5-b43d-5bc1fe96f391-plugins-config-volume\") pod \"ea4dd2c2-8b72-4ec5-b43d-5bc1fe96f391\" (UID: \"ea4dd2c2-8b72-4ec5-b43d-5bc1fe96f391\") " Oct 31 00:06:21.004285 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:21.004240 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0961830-61e4-4528-a704-82a109b453fb-plugins-config-volume" pod "d0961830-61e4-4528-a704-82a109b453fb" (UID: "d0961830-61e4-4528-a704-82a109b453fb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:06:21.004522 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:21.004473 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea4dd2c2-8b72-4ec5-b43d-5bc1fe96f391-plugins-config-volume" pod "ea4dd2c2-8b72-4ec5-b43d-5bc1fe96f391" (UID: "ea4dd2c2-8b72-4ec5-b43d-5bc1fe96f391"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:06:21.006277 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:21.006246 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0961830-61e4-4528-a704-82a109b453fb-kube-api-access-8t8kn" pod "d0961830-61e4-4528-a704-82a109b453fb" (UID: "d0961830-61e4-4528-a704-82a109b453fb"). InnerVolumeSpecName "kube-api-access-8t8kn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:06:21.006466 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:21.006442 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea4dd2c2-8b72-4ec5-b43d-5bc1fe96f391-kube-api-access-sb6f7" pod "ea4dd2c2-8b72-4ec5-b43d-5bc1fe96f391" (UID: "ea4dd2c2-8b72-4ec5-b43d-5bc1fe96f391"). InnerVolumeSpecName "kube-api-access-sb6f7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:06:21.105490 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:21.105458 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8t8kn\" (UniqueName: \"kubernetes.io/projected/d0961830-61e4-4528-a704-82a109b453fb-kube-api-access-8t8kn\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:06:21.105628 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:21.105493 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sb6f7\" (UniqueName: \"kubernetes.io/projected/ea4dd2c2-8b72-4ec5-b43d-5bc1fe96f391-kube-api-access-sb6f7\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:06:21.105628 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:21.105511 4927 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0961830-61e4-4528-a704-82a109b453fb-plugins-config-volume\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:06:21.105628 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:21.105526 4927 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea4dd2c2-8b72-4ec5-b43d-5bc1fe96f391-plugins-config-volume\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:06:21.154735 ip-10-0-141-180 kubenswrapper[4927]: W1031 00:06:21.154706 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37daf0f5_13de_43e2_a2a2_d82036836415.slice/crio-29bafd1d56c24685fcc9fc819586983bcab70e4e928593539a785da5efe07b90 WatchSource:0}: Error finding container 29bafd1d56c24685fcc9fc819586983bcab70e4e928593539a785da5efe07b90: Status 404 returned error can't find the container with id 29bafd1d56c24685fcc9fc819586983bcab70e4e928593539a785da5efe07b90 Oct 31 00:06:21.160445 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:21.160423 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-09efcb6d-c7bfbb5cf-fsvrl"] Oct 31 00:06:21.188882 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:21.188854 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:06:21.205645 ip-10-0-141-180 kubenswrapper[4927]: W1031 00:06:21.205616 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee43bf6b_c4d3_46f6_8808_8064e3412906.slice/crio-ddd0e5273bebfabd1da5796ea1da977daa4520fdd954047459b8591220395f95 WatchSource:0}: Error finding container ddd0e5273bebfabd1da5796ea1da977daa4520fdd954047459b8591220395f95: Status 404 returned error can't find the container with id ddd0e5273bebfabd1da5796ea1da977daa4520fdd954047459b8591220395f95 Oct 31 00:06:21.210854 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:21.210831 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-09efcb6d-699987c7fd-j2v69"] Oct 31 00:06:21.662910 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:21.662873 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-migrate-d32a7cc0-76fd97d5d6-62rcv" podUID="d0961830-61e4-4528-a704-82a109b453fb" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.157:50052\" within 1s: context deadline exceeded" Oct 31 00:06:21.664073 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:21.664041 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-migrate-d32a7cc0-6b7768cb67-ntrqr" podUID="ea4dd2c2-8b72-4ec5-b43d-5bc1fe96f391" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.158:50052\" within 1s: context deadline exceeded" Oct 31 00:06:21.668878 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:21.668851 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-09efcb6d-c7bfbb5cf-fsvrl" event={"ID":"37daf0f5-13de-43e2-a2a2-d82036836415","Type":"ContainerStarted","Data":"41a4c7a18cafccee7073638afa6e7bc78cfe1ffdb9c682af735571d521e5d65c"} Oct 31 00:06:21.669008 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:21.668905 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-09efcb6d-c7bfbb5cf-fsvrl" event={"ID":"37daf0f5-13de-43e2-a2a2-d82036836415","Type":"ContainerStarted","Data":"29bafd1d56c24685fcc9fc819586983bcab70e4e928593539a785da5efe07b90"} Oct 31 00:06:21.669008 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:21.668963 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-09efcb6d-c7bfbb5cf-fsvrl" Oct 31 00:06:21.670462 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:21.670437 4927 generic.go:356] "Generic (PLEG): container finished" podID="ea4dd2c2-8b72-4ec5-b43d-5bc1fe96f391" containerID="c3414683b862bd58304224fd39aa282d41e9f177168b9f828f096bf69e01163b" exitCode=0 Oct 31 00:06:21.670566 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:21.670483 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-d32a7cc0-6b7768cb67-ntrqr" Oct 31 00:06:21.670566 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:21.670513 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-d32a7cc0-6b7768cb67-ntrqr" event={"ID":"ea4dd2c2-8b72-4ec5-b43d-5bc1fe96f391","Type":"ContainerDied","Data":"c3414683b862bd58304224fd39aa282d41e9f177168b9f828f096bf69e01163b"} Oct 31 00:06:21.670665 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:21.670564 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-d32a7cc0-6b7768cb67-ntrqr" event={"ID":"ea4dd2c2-8b72-4ec5-b43d-5bc1fe96f391","Type":"ContainerDied","Data":"394e776fec5d1e36d4349f1d2238ed5fd5b5b43613c64dd67cbc885b7452903f"} Oct 31 00:06:21.670665 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:21.670592 4927 scope.go:122] "RemoveContainer" containerID="c3414683b862bd58304224fd39aa282d41e9f177168b9f828f096bf69e01163b" Oct 31 00:06:21.672246 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:21.672222 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-d32a7cc0-76fd97d5d6-62rcv" Oct 31 00:06:21.672365 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:21.672229 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-d32a7cc0-76fd97d5d6-62rcv" event={"ID":"d0961830-61e4-4528-a704-82a109b453fb","Type":"ContainerDied","Data":"b6867d288486ccf8eb336934acb3a7335d2e8016dd2041d036adcb8ea381c46e"} Oct 31 00:06:21.674568 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:21.674543 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-09efcb6d-699987c7fd-j2v69" event={"ID":"ee43bf6b-c4d3-46f6-8808-8064e3412906","Type":"ContainerStarted","Data":"c3e66009bb474ae5f4aa57b627aaa2ccb072b197139d61f21fa44f8e2c9effc1"} Oct 31 00:06:21.674695 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:21.674590 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-09efcb6d-699987c7fd-j2v69" event={"ID":"ee43bf6b-c4d3-46f6-8808-8064e3412906","Type":"ContainerStarted","Data":"ddd0e5273bebfabd1da5796ea1da977daa4520fdd954047459b8591220395f95"} Oct 31 00:06:21.674695 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:21.674640 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-09efcb6d-699987c7fd-j2v69" Oct 31 00:06:21.682900 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:21.682881 4927 scope.go:122] "RemoveContainer" containerID="c3414683b862bd58304224fd39aa282d41e9f177168b9f828f096bf69e01163b" Oct 31 00:06:21.683178 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:06:21.683158 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3414683b862bd58304224fd39aa282d41e9f177168b9f828f096bf69e01163b\": container with ID starting with c3414683b862bd58304224fd39aa282d41e9f177168b9f828f096bf69e01163b not found: ID does not exist" containerID="c3414683b862bd58304224fd39aa282d41e9f177168b9f828f096bf69e01163b" Oct 31 00:06:21.683278 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:21.683187 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3414683b862bd58304224fd39aa282d41e9f177168b9f828f096bf69e01163b"} err="failed to get container status \"c3414683b862bd58304224fd39aa282d41e9f177168b9f828f096bf69e01163b\": rpc error: code = NotFound desc = could not find container \"c3414683b862bd58304224fd39aa282d41e9f177168b9f828f096bf69e01163b\": container with ID starting with c3414683b862bd58304224fd39aa282d41e9f177168b9f828f096bf69e01163b not found: ID does not exist" Oct 31 00:06:21.683278 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:21.683223 4927 scope.go:122] "RemoveContainer" containerID="49a24120ab046f00ad3aa39f7eb40170389c9d593df058a3552798bf916642c1" Oct 31 00:06:21.792076 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:21.792027 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-09efcb6d-c7bfbb5cf-fsvrl" podStartSLOduration=2.7920108089999998 podStartE2EDuration="2.792010809s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:06:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:06:21.790440091 +0000 UTC m=+1494.826770924" watchObservedRunningTime="2026-10-31 00:06:21.792010809 +0000 UTC m=+1494.828341631" Oct 31 00:06:22.012471 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:22.012379 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-d32a7cc0-6b7768cb67-ntrqr"] Oct 31 00:06:22.030759 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:22.030717 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-d32a7cc0-6b7768cb67-ntrqr"] Oct 31 00:06:22.115417 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:22.115381 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-d32a7cc0-76fd97d5d6-62rcv"] Oct 31 00:06:22.122045 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:22.122015 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-d32a7cc0-76fd97d5d6-62rcv"] Oct 31 00:06:22.236307 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:22.236253 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-09efcb6d-699987c7fd-j2v69" podStartSLOduration=3.236240767 podStartE2EDuration="3.236240767s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:06:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:06:22.234171693 +0000 UTC m=+1495.270502531" watchObservedRunningTime="2026-10-31 00:06:22.236240767 +0000 UTC m=+1495.272571590" Oct 31 00:06:23.091746 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:23.091710 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d0961830-61e4-4528-a704-82a109b453fb" path="/var/lib/kubelet/pods/d0961830-61e4-4528-a704-82a109b453fb/volumes" Oct 31 00:06:23.092789 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:23.092762 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea4dd2c2-8b72-4ec5-b43d-5bc1fe96f391" path="/var/lib/kubelet/pods/ea4dd2c2-8b72-4ec5-b43d-5bc1fe96f391/volumes" Oct 31 00:06:24.004041 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:24.004002 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:06:24.235356 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:24.227981 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:06:24.487357 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:24.481279 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-7c8802f8-84c649965f-8cltq"] Oct 31 00:06:24.487357 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:24.482604 4927 cpu_manager.go:414] "containerMap: removing container" podUID="d0961830-61e4-4528-a704-82a109b453fb" containerName="payload-processing" Oct 31 00:06:24.487357 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:24.482630 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0961830-61e4-4528-a704-82a109b453fb" containerName="payload-processing" Oct 31 00:06:24.487357 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:24.482665 4927 cpu_manager.go:414] "containerMap: removing container" podUID="ea4dd2c2-8b72-4ec5-b43d-5bc1fe96f391" containerName="payload-pre-processing" Oct 31 00:06:24.487357 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:24.482673 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea4dd2c2-8b72-4ec5-b43d-5bc1fe96f391" containerName="payload-pre-processing" Oct 31 00:06:24.508853 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:24.490610 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea4dd2c2-8b72-4ec5-b43d-5bc1fe96f391" containerName="payload-pre-processing" Oct 31 00:06:24.508853 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:24.490676 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0961830-61e4-4528-a704-82a109b453fb" containerName="payload-processing" Oct 31 00:06:24.545372 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:24.539140 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-7c8802f8-84c649965f-8cltq" Oct 31 00:06:24.584887 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:24.578537 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-7c8802f8-84c649965f-8cltq"] Oct 31 00:06:24.629185 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:24.623534 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-7c8802f8\"" Oct 31 00:06:24.629185 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:24.624458 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-7c8802f8-dockercfg-j9bjs\"" Oct 31 00:06:24.650311 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:24.649681 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6mxn\" (UniqueName: \"kubernetes.io/projected/819a7f5a-86da-4db4-8674-e260009f2a61-kube-api-access-z6mxn\") pod \"payload-pre-processing-e2e-ait-7c8802f8-84c649965f-8cltq\" (UID: \"819a7f5a-86da-4db4-8674-e260009f2a61\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-7c8802f8-84c649965f-8cltq" Oct 31 00:06:24.650311 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:24.650029 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/819a7f5a-86da-4db4-8674-e260009f2a61-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-7c8802f8-84c649965f-8cltq\" (UID: \"819a7f5a-86da-4db4-8674-e260009f2a61\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-7c8802f8-84c649965f-8cltq" Oct 31 00:06:24.830297 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:24.830198 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6mxn\" (UniqueName: \"kubernetes.io/projected/819a7f5a-86da-4db4-8674-e260009f2a61-kube-api-access-z6mxn\") pod \"payload-pre-processing-e2e-ait-7c8802f8-84c649965f-8cltq\" (UID: \"819a7f5a-86da-4db4-8674-e260009f2a61\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-7c8802f8-84c649965f-8cltq" Oct 31 00:06:24.830973 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:24.830946 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/819a7f5a-86da-4db4-8674-e260009f2a61-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-7c8802f8-84c649965f-8cltq\" (UID: \"819a7f5a-86da-4db4-8674-e260009f2a61\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-7c8802f8-84c649965f-8cltq" Oct 31 00:06:24.831984 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:24.831960 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/819a7f5a-86da-4db4-8674-e260009f2a61-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-7c8802f8-84c649965f-8cltq\" (UID: \"819a7f5a-86da-4db4-8674-e260009f2a61\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-7c8802f8-84c649965f-8cltq" Oct 31 00:06:25.073046 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:25.069241 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-7c8802f8-5db6686694-5hbns"] Oct 31 00:06:25.111462 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:25.105542 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-7c8802f8-5db6686694-5hbns" Oct 31 00:06:25.165947 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:25.165912 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6mxn\" (UniqueName: \"kubernetes.io/projected/819a7f5a-86da-4db4-8674-e260009f2a61-kube-api-access-z6mxn\") pod \"payload-pre-processing-e2e-ait-7c8802f8-84c649965f-8cltq\" (UID: \"819a7f5a-86da-4db4-8674-e260009f2a61\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-7c8802f8-84c649965f-8cltq" Oct 31 00:06:25.215858 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:25.215610 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-7c8802f8-5db6686694-5hbns"] Oct 31 00:06:25.292704 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:25.292668 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43c9272e-aafb-4785-954c-4fd628242852-plugins-config-volume\") pod \"payload-processing-e2e-ait-7c8802f8-5db6686694-5hbns\" (UID: \"43c9272e-aafb-4785-954c-4fd628242852\") " pod="openshift-ingress/payload-processing-e2e-ait-7c8802f8-5db6686694-5hbns" Oct 31 00:06:25.292916 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:25.292725 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bt5nv\" (UniqueName: \"kubernetes.io/projected/43c9272e-aafb-4785-954c-4fd628242852-kube-api-access-bt5nv\") pod \"payload-processing-e2e-ait-7c8802f8-5db6686694-5hbns\" (UID: \"43c9272e-aafb-4785-954c-4fd628242852\") " pod="openshift-ingress/payload-processing-e2e-ait-7c8802f8-5db6686694-5hbns" Oct 31 00:06:25.384985 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:25.384890 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-7c8802f8-84c649965f-8cltq" Oct 31 00:06:25.393870 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:25.393845 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43c9272e-aafb-4785-954c-4fd628242852-plugins-config-volume\") pod \"payload-processing-e2e-ait-7c8802f8-5db6686694-5hbns\" (UID: \"43c9272e-aafb-4785-954c-4fd628242852\") " pod="openshift-ingress/payload-processing-e2e-ait-7c8802f8-5db6686694-5hbns" Oct 31 00:06:25.393967 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:25.393894 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bt5nv\" (UniqueName: \"kubernetes.io/projected/43c9272e-aafb-4785-954c-4fd628242852-kube-api-access-bt5nv\") pod \"payload-processing-e2e-ait-7c8802f8-5db6686694-5hbns\" (UID: \"43c9272e-aafb-4785-954c-4fd628242852\") " pod="openshift-ingress/payload-processing-e2e-ait-7c8802f8-5db6686694-5hbns" Oct 31 00:06:25.394504 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:25.394482 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43c9272e-aafb-4785-954c-4fd628242852-plugins-config-volume\") pod \"payload-processing-e2e-ait-7c8802f8-5db6686694-5hbns\" (UID: \"43c9272e-aafb-4785-954c-4fd628242852\") " pod="openshift-ingress/payload-processing-e2e-ait-7c8802f8-5db6686694-5hbns" Oct 31 00:06:25.460743 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:25.460709 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bt5nv\" (UniqueName: \"kubernetes.io/projected/43c9272e-aafb-4785-954c-4fd628242852-kube-api-access-bt5nv\") pod \"payload-processing-e2e-ait-7c8802f8-5db6686694-5hbns\" (UID: \"43c9272e-aafb-4785-954c-4fd628242852\") " pod="openshift-ingress/payload-processing-e2e-ait-7c8802f8-5db6686694-5hbns" Oct 31 00:06:25.517514 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:25.517470 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-7c8802f8-5db6686694-5hbns" Oct 31 00:06:25.776809 ip-10-0-141-180 kubenswrapper[4927]: W1031 00:06:25.776773 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod819a7f5a_86da_4db4_8674_e260009f2a61.slice/crio-e6020d65c5dd8246eb88c6caed7cccd19191fbc25c31f888b9a66b661bd63d8b WatchSource:0}: Error finding container e6020d65c5dd8246eb88c6caed7cccd19191fbc25c31f888b9a66b661bd63d8b: Status 404 returned error can't find the container with id e6020d65c5dd8246eb88c6caed7cccd19191fbc25c31f888b9a66b661bd63d8b Oct 31 00:06:25.779196 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:25.779169 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-7c8802f8-84c649965f-8cltq"] Oct 31 00:06:25.881413 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:25.881377 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:06:25.914277 ip-10-0-141-180 kubenswrapper[4927]: W1031 00:06:25.914237 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43c9272e_aafb_4785_954c_4fd628242852.slice/crio-fce115129d290f071c234b9eb9f4ba43fecbbad953f28f1257f370473ad6d53a WatchSource:0}: Error finding container fce115129d290f071c234b9eb9f4ba43fecbbad953f28f1257f370473ad6d53a: Status 404 returned error can't find the container with id fce115129d290f071c234b9eb9f4ba43fecbbad953f28f1257f370473ad6d53a Oct 31 00:06:25.927598 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:25.918480 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-7c8802f8-5db6686694-5hbns"] Oct 31 00:06:26.115174 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:26.115097 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:06:26.152178 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:26.152141 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-7c8802f8-5db6686694-5hbns" event={"ID":"43c9272e-aafb-4785-954c-4fd628242852","Type":"ContainerStarted","Data":"fce115129d290f071c234b9eb9f4ba43fecbbad953f28f1257f370473ad6d53a"} Oct 31 00:06:26.154289 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:26.154261 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-7c8802f8-84c649965f-8cltq" event={"ID":"819a7f5a-86da-4db4-8674-e260009f2a61","Type":"ContainerStarted","Data":"c8f7486d43ca36e4e150ee3121fadd1ef4ff6c719a4b6b0f1bfa4d5d97f27945"} Oct 31 00:06:26.154493 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:26.154474 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-7c8802f8-84c649965f-8cltq" event={"ID":"819a7f5a-86da-4db4-8674-e260009f2a61","Type":"ContainerStarted","Data":"e6020d65c5dd8246eb88c6caed7cccd19191fbc25c31f888b9a66b661bd63d8b"} Oct 31 00:06:26.154629 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:26.154616 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-7c8802f8-84c649965f-8cltq" Oct 31 00:06:26.296409 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:26.296355 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-7c8802f8-84c649965f-8cltq" podStartSLOduration=2.296324438 podStartE2EDuration="2.296324438s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:06:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:06:26.294557783 +0000 UTC m=+1499.330888602" watchObservedRunningTime="2026-10-31 00:06:26.296324438 +0000 UTC m=+1499.332655259" Oct 31 00:06:26.389047 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:26.388968 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:06:27.169435 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:27.169387 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-7c8802f8-5db6686694-5hbns" event={"ID":"43c9272e-aafb-4785-954c-4fd628242852","Type":"ContainerStarted","Data":"9e8851d1a744adead476d34bc873e3c6cc40036956b9c742aef39d331efa1cef"} Oct 31 00:06:27.169896 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:27.169830 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-7c8802f8-5db6686694-5hbns" Oct 31 00:06:27.291616 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:27.291555 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-7c8802f8-5db6686694-5hbns" podStartSLOduration=3.2915366710000002 podStartE2EDuration="3.291536671s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:06:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:06:27.288216952 +0000 UTC m=+1500.324547775" watchObservedRunningTime="2026-10-31 00:06:27.291536671 +0000 UTC m=+1500.327867496" Oct 31 00:06:28.417369 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:28.417316 4927 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-pqt2c_6382dfb5-a363-47a2-bc79-0b96f7bec0c3/console-operator/3.log" Oct 31 00:06:28.853282 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:28.853193 4927 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-pqt2c_6382dfb5-a363-47a2-bc79-0b96f7bec0c3/console-operator/3.log" Oct 31 00:06:30.060460 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:30.060419 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b4d752ef-g2sz5"] Oct 31 00:06:30.110752 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:30.110709 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b4d752ef-g2sz5"] Oct 31 00:06:30.431353 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:30.430957 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:06:31.090424 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:31.090388 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bcbadb2f-3539-4d46-9100-2bae9ba06d14" path="/var/lib/kubelet/pods/bcbadb2f-3539-4d46-9100-2bae9ba06d14/volumes" Oct 31 00:06:32.689411 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:32.689383 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-09efcb6d-699987c7fd-j2v69" Oct 31 00:06:32.689826 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:32.689464 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-09efcb6d-c7bfbb5cf-fsvrl" Oct 31 00:06:34.209006 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:34.208965 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:06:34.359960 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:34.359916 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:06:35.053649 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:35.053611 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:06:35.498880 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:35.498838 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:06:35.697508 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:35.697461 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:06:37.172635 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:37.172607 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-7c8802f8-84c649965f-8cltq" Oct 31 00:06:37.893286 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:37.893247 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a380d1e-c9kqq"] Oct 31 00:06:37.900004 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:37.899977 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a380d1e-c9kqq" Oct 31 00:06:37.950643 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:37.950526 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c127cf0-aa5c-4d3b-bdb5-704bf0931c92-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3a380d1e-c9kqq\" (UID: \"1c127cf0-aa5c-4d3b-bdb5-704bf0931c92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a380d1e-c9kqq" Oct 31 00:06:37.950643 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:37.950643 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c127cf0-aa5c-4d3b-bdb5-704bf0931c92-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3a380d1e-c9kqq\" (UID: \"1c127cf0-aa5c-4d3b-bdb5-704bf0931c92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a380d1e-c9kqq" Oct 31 00:06:37.950863 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:37.950715 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j26q6\" (UniqueName: \"kubernetes.io/projected/1c127cf0-aa5c-4d3b-bdb5-704bf0931c92-kube-api-access-j26q6\") pod \"maas-api-revoke-sub-3a380d1e-c9kqq\" (UID: \"1c127cf0-aa5c-4d3b-bdb5-704bf0931c92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a380d1e-c9kqq" Oct 31 00:06:37.957009 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:37.956976 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a380d1e-c9kqq"] Oct 31 00:06:37.999318 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:37.999284 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:06:37.999518 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:37.999486 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4fdzs\"" Oct 31 00:06:38.052766 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:38.052724 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c127cf0-aa5c-4d3b-bdb5-704bf0931c92-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3a380d1e-c9kqq\" (UID: \"1c127cf0-aa5c-4d3b-bdb5-704bf0931c92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a380d1e-c9kqq" Oct 31 00:06:38.052947 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:38.052835 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j26q6\" (UniqueName: \"kubernetes.io/projected/1c127cf0-aa5c-4d3b-bdb5-704bf0931c92-kube-api-access-j26q6\") pod \"maas-api-revoke-sub-3a380d1e-c9kqq\" (UID: \"1c127cf0-aa5c-4d3b-bdb5-704bf0931c92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a380d1e-c9kqq" Oct 31 00:06:38.053172 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:38.053150 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c127cf0-aa5c-4d3b-bdb5-704bf0931c92-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3a380d1e-c9kqq\" (UID: \"1c127cf0-aa5c-4d3b-bdb5-704bf0931c92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a380d1e-c9kqq" Oct 31 00:06:38.053506 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:38.053482 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c127cf0-aa5c-4d3b-bdb5-704bf0931c92-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3a380d1e-c9kqq\" (UID: \"1c127cf0-aa5c-4d3b-bdb5-704bf0931c92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a380d1e-c9kqq" Oct 31 00:06:38.055669 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:38.055642 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c127cf0-aa5c-4d3b-bdb5-704bf0931c92-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3a380d1e-c9kqq\" (UID: \"1c127cf0-aa5c-4d3b-bdb5-704bf0931c92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a380d1e-c9kqq" Oct 31 00:06:38.115703 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:38.115666 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j26q6\" (UniqueName: \"kubernetes.io/projected/1c127cf0-aa5c-4d3b-bdb5-704bf0931c92-kube-api-access-j26q6\") pod \"maas-api-revoke-sub-3a380d1e-c9kqq\" (UID: \"1c127cf0-aa5c-4d3b-bdb5-704bf0931c92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a380d1e-c9kqq" Oct 31 00:06:38.181743 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:38.181713 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-7c8802f8-5db6686694-5hbns" Oct 31 00:06:38.219172 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:38.219140 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a380d1e-c9kqq" Oct 31 00:06:38.624996 ip-10-0-141-180 kubenswrapper[4927]: W1031 00:06:38.624952 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c127cf0_aa5c_4d3b_bdb5_704bf0931c92.slice/crio-a6f3da520ddbe8a25c7cd780da4e3be9e6b46add59aa56ee2976fb4ccb82a5b7 WatchSource:0}: Error finding container a6f3da520ddbe8a25c7cd780da4e3be9e6b46add59aa56ee2976fb4ccb82a5b7: Status 404 returned error can't find the container with id a6f3da520ddbe8a25c7cd780da4e3be9e6b46add59aa56ee2976fb4ccb82a5b7 Oct 31 00:06:38.627819 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:38.627797 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a380d1e-c9kqq"] Oct 31 00:06:39.321484 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:39.321397 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a380d1e-c9kqq" event={"ID":"1c127cf0-aa5c-4d3b-bdb5-704bf0931c92","Type":"ContainerStarted","Data":"801fbd810ffe2c4a894321745973a5a789caed00ad1b177930dc6e6a9a01e689"} Oct 31 00:06:39.321484 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:39.321453 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a380d1e-c9kqq" event={"ID":"1c127cf0-aa5c-4d3b-bdb5-704bf0931c92","Type":"ContainerStarted","Data":"a6f3da520ddbe8a25c7cd780da4e3be9e6b46add59aa56ee2976fb4ccb82a5b7"} Oct 31 00:06:39.445445 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:39.445386 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a380d1e-c9kqq" podStartSLOduration=2.445372648 podStartE2EDuration="2.445372648s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:06:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:06:39.44000064 +0000 UTC m=+1512.476331463" watchObservedRunningTime="2026-10-31 00:06:39.445372648 +0000 UTC m=+1512.481703470" Oct 31 00:06:39.703940 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:39.703903 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:06:40.068442 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:40.068411 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6014ecc2-lgxwz"] Oct 31 00:06:40.068903 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:40.068624 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:06:40.072640 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:40.072615 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6014ecc2-lgxwz"] Oct 31 00:06:40.261415 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:40.261373 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:06:40.336451 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:40.336367 4927 generic.go:356] "Generic (PLEG): container finished" podID="1c127cf0-aa5c-4d3b-bdb5-704bf0931c92" containerID="801fbd810ffe2c4a894321745973a5a789caed00ad1b177930dc6e6a9a01e689" exitCode=0 Oct 31 00:06:40.336838 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:40.336453 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a380d1e-c9kqq" event={"ID":"1c127cf0-aa5c-4d3b-bdb5-704bf0931c92","Type":"ContainerDied","Data":"801fbd810ffe2c4a894321745973a5a789caed00ad1b177930dc6e6a9a01e689"} Oct 31 00:06:41.091098 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:41.091058 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="51c36120-3c7f-497c-84a2-e23674aae599" path="/var/lib/kubelet/pods/51c36120-3c7f-497c-84a2-e23674aae599/volumes" Oct 31 00:06:41.537759 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:41.537734 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a380d1e-c9kqq" Oct 31 00:06:41.614978 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:41.614904 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1c127cf0-aa5c-4d3b-bdb5-704bf0931c92-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c127cf0-aa5c-4d3b-bdb5-704bf0931c92-maas-api-xks-ca\") pod \"1c127cf0-aa5c-4d3b-bdb5-704bf0931c92\" (UID: \"1c127cf0-aa5c-4d3b-bdb5-704bf0931c92\") " Oct 31 00:06:41.615134 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:41.615014 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c127cf0-aa5c-4d3b-bdb5-704bf0931c92-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c127cf0-aa5c-4d3b-bdb5-704bf0931c92-maas-api-openshift-ca\") pod \"1c127cf0-aa5c-4d3b-bdb5-704bf0931c92\" (UID: \"1c127cf0-aa5c-4d3b-bdb5-704bf0931c92\") " Oct 31 00:06:41.615134 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:41.615101 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c127cf0-aa5c-4d3b-bdb5-704bf0931c92-kube-api-access-j26q6\" (UniqueName: \"kubernetes.io/projected/1c127cf0-aa5c-4d3b-bdb5-704bf0931c92-kube-api-access-j26q6\") pod \"1c127cf0-aa5c-4d3b-bdb5-704bf0931c92\" (UID: \"1c127cf0-aa5c-4d3b-bdb5-704bf0931c92\") " Oct 31 00:06:41.615477 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:41.615433 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c127cf0-aa5c-4d3b-bdb5-704bf0931c92-maas-api-openshift-ca" pod "1c127cf0-aa5c-4d3b-bdb5-704bf0931c92" (UID: "1c127cf0-aa5c-4d3b-bdb5-704bf0931c92"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:06:41.615977 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:41.615955 4927 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c127cf0-aa5c-4d3b-bdb5-704bf0931c92-maas-api-openshift-ca\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:06:41.617251 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:41.617230 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c127cf0-aa5c-4d3b-bdb5-704bf0931c92-maas-api-xks-ca" pod "1c127cf0-aa5c-4d3b-bdb5-704bf0931c92" (UID: "1c127cf0-aa5c-4d3b-bdb5-704bf0931c92"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:06:41.617657 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:41.617634 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c127cf0-aa5c-4d3b-bdb5-704bf0931c92-kube-api-access-j26q6" pod "1c127cf0-aa5c-4d3b-bdb5-704bf0931c92" (UID: "1c127cf0-aa5c-4d3b-bdb5-704bf0931c92"). InnerVolumeSpecName "kube-api-access-j26q6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:06:41.718037 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:41.717996 4927 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c127cf0-aa5c-4d3b-bdb5-704bf0931c92-maas-api-xks-ca\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:06:41.718037 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:41.718035 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j26q6\" (UniqueName: \"kubernetes.io/projected/1c127cf0-aa5c-4d3b-bdb5-704bf0931c92-kube-api-access-j26q6\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:06:42.060402 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:42.060359 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ce935ca-nbf9s"] Oct 31 00:06:42.066007 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:42.065981 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ce935ca-nbf9s"] Oct 31 00:06:42.359341 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:42.359243 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a380d1e-c9kqq" Oct 31 00:06:42.359341 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:42.359282 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a380d1e-c9kqq" event={"ID":"1c127cf0-aa5c-4d3b-bdb5-704bf0931c92","Type":"ContainerDied","Data":"a6f3da520ddbe8a25c7cd780da4e3be9e6b46add59aa56ee2976fb4ccb82a5b7"} Oct 31 00:06:42.359546 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:42.359378 4927 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a6f3da520ddbe8a25c7cd780da4e3be9e6b46add59aa56ee2976fb4ccb82a5b7" Oct 31 00:06:43.089149 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:43.089118 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5aeb8b0-448b-4ead-aab0-5052033ec9f4" path="/var/lib/kubelet/pods/e5aeb8b0-448b-4ead-aab0-5052033ec9f4/volumes" Oct 31 00:06:43.669078 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:43.669037 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:06:45.209461 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:45.209419 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:06:45.413683 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:45.413647 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:06:45.583415 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:45.581820 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:06:45.678514 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:45.678477 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d32a7cc0-6h24p"] Oct 31 00:06:45.726453 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:45.726416 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d32a7cc0-6h24p"] Oct 31 00:06:45.837377 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:45.837287 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:06:46.196357 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:46.187977 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:06:46.745067 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:46.706900 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:06:46.921106 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:46.920953 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:06:47.091557 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:47.091474 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f032e374-6050-4177-b404-a4f3e3bc50cf" path="/var/lib/kubelet/pods/f032e374-6050-4177-b404-a4f3e3bc50cf/volumes" Oct 31 00:06:47.390272 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:47.390169 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:06:48.628056 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:48.628021 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:06:49.054953 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:49.054919 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e9419c8d-xp797"] Oct 31 00:06:49.106444 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:49.106414 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e9419c8d-xp797"] Oct 31 00:06:49.681762 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:06:49.681726 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sqv69" containerName="extractor" Oct 31 00:06:50.595057 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:50.595023 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:06:51.091114 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:51.091083 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa9d6354-c4e7-40fd-b163-103d75f0e34e" path="/var/lib/kubelet/pods/fa9d6354-c4e7-40fd-b163-103d75f0e34e/volumes" Oct 31 00:06:51.521798 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:51.521757 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:06:51.708261 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:51.708205 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:06:51.873534 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:51.873434 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:06:54.132715 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:54.132681 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:06:54.292490 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:54.292459 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:06:54.661683 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:54.661641 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:06:54.895351 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:54.890188 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:06:55.140405 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:55.139952 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-f7b8d8-576dfbf9f9-d9d8l"] Oct 31 00:06:55.141398 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:55.141213 4927 cpu_manager.go:414] "containerMap: removing container" podUID="1c127cf0-aa5c-4d3b-bdb5-704bf0931c92" containerName="revoke-keys" Oct 31 00:06:55.141398 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:55.141241 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c127cf0-aa5c-4d3b-bdb5-704bf0931c92" containerName="revoke-keys" Oct 31 00:06:55.141635 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:55.141495 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c127cf0-aa5c-4d3b-bdb5-704bf0931c92" containerName="revoke-keys" Oct 31 00:06:55.161947 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:55.159912 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-f7b8d8-576dfbf9f9-d9d8l" Oct 31 00:06:55.211420 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:55.206699 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j5mdr\" (UniqueName: \"kubernetes.io/projected/4644c742-d4f8-4b7c-89c6-05e38fc39063-kube-api-access-j5mdr\") pod \"payload-pre-processing-e2e-shared-a-w6-f7b8d8-576dfbf9f9-d9d8l\" (UID: \"4644c742-d4f8-4b7c-89c6-05e38fc39063\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-f7b8d8-576dfbf9f9-d9d8l" Oct 31 00:06:55.226626 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:55.218438 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4644c742-d4f8-4b7c-89c6-05e38fc39063-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w6-f7b8d8-576dfbf9f9-d9d8l\" (UID: \"4644c742-d4f8-4b7c-89c6-05e38fc39063\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-f7b8d8-576dfbf9f9-d9d8l" Oct 31 00:06:55.226626 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:55.218344 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w6-f7b8d8\"" Oct 31 00:06:55.297363 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:55.284221 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w6-f7b8d8-dockercfg-5vzdt\"" Oct 31 00:06:55.297363 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:55.258323 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-f7b8d8-576dfbf9f9-d9d8l"] Oct 31 00:06:55.327036 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:55.323991 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5mdr\" (UniqueName: \"kubernetes.io/projected/4644c742-d4f8-4b7c-89c6-05e38fc39063-kube-api-access-j5mdr\") pod \"payload-pre-processing-e2e-shared-a-w6-f7b8d8-576dfbf9f9-d9d8l\" (UID: \"4644c742-d4f8-4b7c-89c6-05e38fc39063\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-f7b8d8-576dfbf9f9-d9d8l" Oct 31 00:06:55.327036 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:55.324060 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4644c742-d4f8-4b7c-89c6-05e38fc39063-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w6-f7b8d8-576dfbf9f9-d9d8l\" (UID: \"4644c742-d4f8-4b7c-89c6-05e38fc39063\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-f7b8d8-576dfbf9f9-d9d8l" Oct 31 00:06:55.361848 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:55.351301 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4644c742-d4f8-4b7c-89c6-05e38fc39063-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w6-f7b8d8-576dfbf9f9-d9d8l\" (UID: \"4644c742-d4f8-4b7c-89c6-05e38fc39063\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-f7b8d8-576dfbf9f9-d9d8l" Oct 31 00:06:55.725486 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:55.710727 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5mdr\" (UniqueName: \"kubernetes.io/projected/4644c742-d4f8-4b7c-89c6-05e38fc39063-kube-api-access-j5mdr\") pod \"payload-pre-processing-e2e-shared-a-w6-f7b8d8-576dfbf9f9-d9d8l\" (UID: \"4644c742-d4f8-4b7c-89c6-05e38fc39063\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-f7b8d8-576dfbf9f9-d9d8l" Oct 31 00:06:55.835676 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:55.835632 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-f7b8d8-576dfbf9f9-d9d8l" Oct 31 00:06:55.913469 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:55.913360 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:06:56.338995 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:56.338888 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:06:56.689315 ip-10-0-141-180 kubenswrapper[4927]: W1031 00:06:56.689280 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4644c742_d4f8_4b7c_89c6_05e38fc39063.slice/crio-d4799cc809c9403c58819283754d9cb0a145672104855bf2514cd031d9cf84f5 WatchSource:0}: Error finding container d4799cc809c9403c58819283754d9cb0a145672104855bf2514cd031d9cf84f5: Status 404 returned error can't find the container with id d4799cc809c9403c58819283754d9cb0a145672104855bf2514cd031d9cf84f5 Oct 31 00:06:56.692731 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:56.692703 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-f7b8d8-576dfbf9f9-d9d8l"] Oct 31 00:06:57.479139 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:57.479097 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-f7b8d8-576dfbf9f9-d9d8l" event={"ID":"4644c742-d4f8-4b7c-89c6-05e38fc39063","Type":"ContainerStarted","Data":"12fede5ab004f064eaab33f83a2b575896370cea5ba7d5d5ef00c0a6c2caa99e"} Oct 31 00:06:57.479561 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:57.479170 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-f7b8d8-576dfbf9f9-d9d8l" Oct 31 00:06:57.479561 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:57.479193 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-f7b8d8-576dfbf9f9-d9d8l" event={"ID":"4644c742-d4f8-4b7c-89c6-05e38fc39063","Type":"ContainerStarted","Data":"d4799cc809c9403c58819283754d9cb0a145672104855bf2514cd031d9cf84f5"} Oct 31 00:06:57.598568 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:57.598510 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-f7b8d8-576dfbf9f9-d9d8l" podStartSLOduration=2.59849268 podStartE2EDuration="2.59849268s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:06:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:06:57.597432451 +0000 UTC m=+1530.633763273" watchObservedRunningTime="2026-10-31 00:06:57.59849268 +0000 UTC m=+1530.634823504" Oct 31 00:06:59.381317 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:06:59.381274 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:07:00.057897 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:07:00.057776 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:07:00.425728 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:07:00.425444 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:07:00.624398 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:07:00.600740 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:07:00.769479 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:07:00.769390 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:07:04.551885 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:07:04.551846 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:07:04.712998 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:07:04.712959 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:07:05.051300 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:07:05.051262 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:07:05.289460 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:07:05.289415 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:07:05.471358 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:07:05.466670 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:07:05.659262 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:07:05.658797 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:07:08.505316 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:07:08.505288 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-f7b8d8-576dfbf9f9-d9d8l" Oct 31 00:07:10.409281 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:07:10.409227 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:07:10.689409 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:07:10.675564 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:07:10.896787 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:07:10.896755 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:07:10.969998 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:07:10.949418 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-a251fdfb-6f8487684c-6qzl9"] Oct 31 00:07:11.003684 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:07:11.003635 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-a251fdfb-6f8487684c-6qzl9" Oct 31 00:07:11.050556 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:07:11.049149 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-a251fdfb-6f8487684c-6qzl9"] Oct 31 00:07:11.150998 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:07:11.150153 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-skxk5\" (UniqueName: \"kubernetes.io/projected/f742e7e8-ad0d-4693-8c12-5b2940957133-kube-api-access-skxk5\") pod \"payload-pre-processing-e2e-mt-a251fdfb-6f8487684c-6qzl9\" (UID: \"f742e7e8-ad0d-4693-8c12-5b2940957133\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-a251fdfb-6f8487684c-6qzl9" Oct 31 00:07:11.150998 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:07:11.150277 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f742e7e8-ad0d-4693-8c12-5b2940957133-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-a251fdfb-6f8487684c-6qzl9\" (UID: \"f742e7e8-ad0d-4693-8c12-5b2940957133\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-a251fdfb-6f8487684c-6qzl9" Oct 31 00:07:11.180353 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:07:11.166651 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-a251fdfb-dockercfg-vw4bp\"" Oct 31 00:07:11.199617 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:07:11.199546 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-a251fdfb\"" Oct 31 00:07:11.294870 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:07:11.284086 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-skxk5\" (UniqueName: \"kubernetes.io/projected/f742e7e8-ad0d-4693-8c12-5b2940957133-kube-api-access-skxk5\") pod \"payload-pre-processing-e2e-mt-a251fdfb-6f8487684c-6qzl9\" (UID: \"f742e7e8-ad0d-4693-8c12-5b2940957133\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-a251fdfb-6f8487684c-6qzl9" Oct 31 00:07:11.303797 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:07:11.303759 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f742e7e8-ad0d-4693-8c12-5b2940957133-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-a251fdfb-6f8487684c-6qzl9\" (UID: \"f742e7e8-ad0d-4693-8c12-5b2940957133\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-a251fdfb-6f8487684c-6qzl9" Oct 31 00:07:11.359357 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:07:11.334302 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f742e7e8-ad0d-4693-8c12-5b2940957133-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-a251fdfb-6f8487684c-6qzl9\" (UID: \"f742e7e8-ad0d-4693-8c12-5b2940957133\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-a251fdfb-6f8487684c-6qzl9" Oct 31 00:07:11.379987 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:07:11.379949 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-a251fdfb-6fb6bf7d6b-xmmgr"] Oct 31 00:07:11.622404 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:07:11.613620 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-a251fdfb-6fb6bf7d6b-xmmgr" Oct 31 00:07:11.703766 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:07:11.703696 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-skxk5\" (UniqueName: \"kubernetes.io/projected/f742e7e8-ad0d-4693-8c12-5b2940957133-kube-api-access-skxk5\") pod \"payload-pre-processing-e2e-mt-a251fdfb-6f8487684c-6qzl9\" (UID: \"f742e7e8-ad0d-4693-8c12-5b2940957133\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-a251fdfb-6f8487684c-6qzl9" Oct 31 00:07:11.754387 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:07:11.752379 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4143d909-tk5lx"] Oct 31 00:07:11.754387 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:07:11.753785 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c690fe76-022e-44e3-97f3-485dacb13fb9-plugins-config-volume\") pod \"payload-processing-e2e-mt-a251fdfb-6fb6bf7d6b-xmmgr\" (UID: \"c690fe76-022e-44e3-97f3-485dacb13fb9\") " pod="openshift-ingress/payload-processing-e2e-mt-a251fdfb-6fb6bf7d6b-xmmgr" Oct 31 00:07:11.754387 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:07:11.753832 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wpt64\" (UniqueName: \"kubernetes.io/projected/c690fe76-022e-44e3-97f3-485dacb13fb9-kube-api-access-wpt64\") pod \"payload-processing-e2e-mt-a251fdfb-6fb6bf7d6b-xmmgr\" (UID: \"c690fe76-022e-44e3-97f3-485dacb13fb9\") " pod="openshift-ingress/payload-processing-e2e-mt-a251fdfb-6fb6bf7d6b-xmmgr" Oct 31 00:07:11.788358 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:07:11.780387 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6d701af0-6r8hs"] Oct 31 00:07:11.821355 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:07:11.821136 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4143d909-tk5lx"] Oct 31 00:07:11.861357 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:07:11.856786 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c690fe76-022e-44e3-97f3-485dacb13fb9-plugins-config-volume\") pod \"payload-processing-e2e-mt-a251fdfb-6fb6bf7d6b-xmmgr\" (UID: \"c690fe76-022e-44e3-97f3-485dacb13fb9\") " pod="openshift-ingress/payload-processing-e2e-mt-a251fdfb-6fb6bf7d6b-xmmgr" Oct 31 00:07:11.861357 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:07:11.856841 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wpt64\" (UniqueName: \"kubernetes.io/projected/c690fe76-022e-44e3-97f3-485dacb13fb9-kube-api-access-wpt64\") pod \"payload-processing-e2e-mt-a251fdfb-6fb6bf7d6b-xmmgr\" (UID: \"c690fe76-022e-44e3-97f3-485dacb13fb9\") " pod="openshift-ingress/payload-processing-e2e-mt-a251fdfb-6fb6bf7d6b-xmmgr" Oct 31 00:07:11.861357 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:07:11.858474 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c690fe76-022e-44e3-97f3-485dacb13fb9-plugins-config-volume\") pod \"payload-processing-e2e-mt-a251fdfb-6fb6bf7d6b-xmmgr\" (UID: \"c690fe76-022e-44e3-97f3-485dacb13fb9\") " pod="openshift-ingress/payload-processing-e2e-mt-a251fdfb-6fb6bf7d6b-xmmgr" Oct 31 00:07:11.867127 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:07:11.865773 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-a251fdfb-6f8487684c-6qzl9" Oct 31 00:07:11.867127 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:07:11.866161 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6d701af0-6r8hs"] Oct 31 00:07:12.000354 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:07:11.989597 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wpt64\" (UniqueName: \"kubernetes.io/projected/c690fe76-022e-44e3-97f3-485dacb13fb9-kube-api-access-wpt64\") pod \"payload-processing-e2e-mt-a251fdfb-6fb6bf7d6b-xmmgr\" (UID: \"c690fe76-022e-44e3-97f3-485dacb13fb9\") " pod="openshift-ingress/payload-processing-e2e-mt-a251fdfb-6fb6bf7d6b-xmmgr" Oct 31 00:07:12.020992 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:07:12.005382 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-a251fdfb-6fb6bf7d6b-xmmgr"] Oct 31 00:07:12.020992 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:07:12.018618 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-a251fdfb-6fb6bf7d6b-xmmgr" Oct 31 00:07:12.676360 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:07:12.675433 4927 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-zvnnz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:07:12.676360 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:07:12.675508 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-zvnnz" podUID="6ba0721f-a4d2-49f6-b161-6d6f0f8ae4c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:07:13.254406 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:07:13.253792 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-27e82b/distinct-w2-27e82b-kserve-75f475fcd9-mccsz" podUID="4ba8ced4-e93f-46ac-87c5-3d0b7fadc02d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:07:13.279357 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:07:13.261684 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5c3374/premium-w1-5c3374-kserve-6c56d4dcd-llwcl" podUID="dc813f02-e077-4251-8c1e-fea0986b0f36" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:07:13.279357 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:07:13.270802 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-35a187/distinct-2-w6-35a187-kserve-7ff4b6dc55-944g6" podUID="429ee092-42d8-475e-b0c6-b5a9930766fc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.113:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:07:13.279357 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:07:13.271212 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fe01fa/distinct-2-w5-fe01fa-kserve-79b99966d8-dkhzd" podUID="d5c1c9e8-ce47-4868-860d-b8413ff74ab5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.111:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:07:13.317355 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:07:13.305751 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-46b586/simulator-w0-46b586-kserve-67588b8dbd-z8xmn" podUID="9a25e252-223d-41f2-abf7-632067c77e0b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded" Oct 31 00:07:13.351356 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:07:13.348613 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f4cd0f96-bac0-49bc-a27f-4c2ae5e7aaf7" path="/var/lib/kubelet/pods/f4cd0f96-bac0-49bc-a27f-4c2ae5e7aaf7/volumes" Oct 31 00:07:13.351356 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:07:13.349569 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fdd8d855-2e0f-4843-9705-dc15f15625d1" path="/var/lib/kubelet/pods/fdd8d855-2e0f-4843-9705-dc15f15625d1/volumes" Oct 31 00:07:13.546407 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:07:13.545966 4927 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-dpcqs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:07:13.546407 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:07:13.546032 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-dpcqs" podUID="8328651d-33b5-4160-8f99-9d6de87ba69f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:07:13.757425 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:07:13.757148 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-a251fdfb-6f8487684c-6qzl9"] Oct 31 00:07:14.027118 ip-10-0-141-180 kubenswrapper[4927]: W1031 00:07:14.011352 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc690fe76_022e_44e3_97f3_485dacb13fb9.slice/crio-b35d71a69c17001cbb8637ebf94c1624c7a936f4fa860a43a65da00b207d14e5 WatchSource:0}: Error finding container b35d71a69c17001cbb8637ebf94c1624c7a936f4fa860a43a65da00b207d14e5: Status 404 returned error can't find the container with id b35d71a69c17001cbb8637ebf94c1624c7a936f4fa860a43a65da00b207d14e5 Oct 31 00:07:14.027118 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:07:14.024158 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-a251fdfb-6fb6bf7d6b-xmmgr"] Oct 31 00:07:14.070349 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:07:14.065803 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b383413a-nvgmn"] Oct 31 00:07:14.085546 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:07:14.081585 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b383413a-nvgmn"] Oct 31 00:07:14.335877 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:07:14.335777 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-a251fdfb-6fb6bf7d6b-xmmgr" event={"ID":"c690fe76-022e-44e3-97f3-485dacb13fb9","Type":"ContainerStarted","Data":"43283d1910c02712d40fa345f0f5087fcb4b39bb083fea208a8fb941db2246fe"} Oct 31 00:07:14.335877 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:07:14.335838 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-a251fdfb-6fb6bf7d6b-xmmgr" event={"ID":"c690fe76-022e-44e3-97f3-485dacb13fb9","Type":"ContainerStarted","Data":"b35d71a69c17001cbb8637ebf94c1624c7a936f4fa860a43a65da00b207d14e5"} Oct 31 00:07:14.335877 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:07:14.335868 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-a251fdfb-6fb6bf7d6b-xmmgr" Oct 31 00:07:14.337847 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:07:14.337815 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-a251fdfb-6f8487684c-6qzl9" event={"ID":"f742e7e8-ad0d-4693-8c12-5b2940957133","Type":"ContainerStarted","Data":"ec6b17c2ceea177eb3f8efca903aaec07945a781ce20aa3a572eb4610861da8e"} Oct 31 00:07:14.337959 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:07:14.337868 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-a251fdfb-6f8487684c-6qzl9" event={"ID":"f742e7e8-ad0d-4693-8c12-5b2940957133","Type":"ContainerStarted","Data":"e29884b7ba0c9ca298962593b469ca67522a6686a3ee1527b159d7e5ebde48e5"} Oct 31 00:07:14.337959 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:07:14.337893 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-a251fdfb-6f8487684c-6qzl9" Oct 31 00:07:14.456043 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:07:14.455993 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-a251fdfb-6fb6bf7d6b-xmmgr" podStartSLOduration=4.455975677 podStartE2EDuration="4.455975677s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:07:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:07:14.453662045 +0000 UTC m=+1547.489992868" watchObservedRunningTime="2026-10-31 00:07:14.455975677 +0000 UTC m=+1547.492306500" Oct 31 00:07:14.592465 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:07:14.589778 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-a251fdfb-6f8487684c-6qzl9" podStartSLOduration=4.5897552600000004 podStartE2EDuration="4.58975526s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:07:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:07:14.575833724 +0000 UTC m=+1547.612164563" watchObservedRunningTime="2026-10-31 00:07:14.58975526 +0000 UTC m=+1547.626086083" Oct 31 00:07:14.770618 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:07:14.770581 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e0751b65-5lgb6"] Oct 31 00:07:14.777573 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:07:14.777543 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e0751b65-5lgb6" Oct 31 00:07:14.825664 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:07:14.824401 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e0751b65-5lgb6"] Oct 31 00:07:14.861406 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:07:14.853543 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:07:14.861406 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:07:14.853673 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4fdzs\"" Oct 31 00:07:14.963877 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:07:14.963834 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a735e34-9f5b-44bc-8155-cfec5319b710-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-e0751b65-5lgb6\" (UID: \"0a735e34-9f5b-44bc-8155-cfec5319b710\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e0751b65-5lgb6" Oct 31 00:07:14.976370 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:07:14.976160 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79wc9\" (UniqueName: \"kubernetes.io/projected/0a735e34-9f5b-44bc-8155-cfec5319b710-kube-api-access-79wc9\") pod \"maas-api-revoke-keys-e2e-mt-e0751b65-5lgb6\" (UID: \"0a735e34-9f5b-44bc-8155-cfec5319b710\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e0751b65-5lgb6" Oct 31 00:07:14.976761 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:07:14.976728 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a735e34-9f5b-44bc-8155-cfec5319b710-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-e0751b65-5lgb6\" (UID: \"0a735e34-9f5b-44bc-8155-cfec5319b710\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e0751b65-5lgb6" Oct 31 00:07:15.079762 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:07:15.079733 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a735e34-9f5b-44bc-8155-cfec5319b710-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-e0751b65-5lgb6\" (UID: \"0a735e34-9f5b-44bc-8155-cfec5319b710\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e0751b65-5lgb6" Oct 31 00:07:15.079964 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:07:15.079765 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79wc9\" (UniqueName: \"kubernetes.io/projected/0a735e34-9f5b-44bc-8155-cfec5319b710-kube-api-access-79wc9\") pod \"maas-api-revoke-keys-e2e-mt-e0751b65-5lgb6\" (UID: \"0a735e34-9f5b-44bc-8155-cfec5319b710\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e0751b65-5lgb6" Oct 31 00:07:15.079964 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:07:15.079810 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a735e34-9f5b-44bc-8155-cfec5319b710-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-e0751b65-5lgb6\" (UID: \"0a735e34-9f5b-44bc-8155-cfec5319b710\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e0751b65-5lgb6" Oct 31 00:07:15.080542 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:07:15.080516 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a735e34-9f5b-44bc-8155-cfec5319b710-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-e0751b65-5lgb6\" (UID: \"0a735e34-9f5b-44bc-8155-cfec5319b710\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e0751b65-5lgb6" Oct 31 00:07:15.082304 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:07:15.082277 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a735e34-9f5b-44bc-8155-cfec5319b710-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-e0751b65-5lgb6\" (UID: \"0a735e34-9f5b-44bc-8155-cfec5319b710\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e0751b65-5lgb6" Oct 31 00:07:15.094739 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:07:15.094709 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="29750093-15c3-4899-a7b2-b8f46713c2ee" path="/var/lib/kubelet/pods/29750093-15c3-4899-a7b2-b8f46713c2ee/volumes" Oct 31 00:07:15.145014 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:07:15.144935 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-79wc9\" (UniqueName: \"kubernetes.io/projected/0a735e34-9f5b-44bc-8155-cfec5319b710-kube-api-access-79wc9\") pod \"maas-api-revoke-keys-e2e-mt-e0751b65-5lgb6\" (UID: \"0a735e34-9f5b-44bc-8155-cfec5319b710\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e0751b65-5lgb6" Oct 31 00:07:15.436439 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:07:15.436394 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e0751b65-5lgb6" Oct 31 00:07:15.822344 ip-10-0-141-180 kubenswrapper[4927]: W1031 00:07:15.822298 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a735e34_9f5b_44bc_8155_cfec5319b710.slice/crio-7e1288fd4822103e183e13ec36f862ba00632090be496fca2304d59796bd8b59 WatchSource:0}: Error finding container 7e1288fd4822103e183e13ec36f862ba00632090be496fca2304d59796bd8b59: Status 404 returned error can't find the container with id 7e1288fd4822103e183e13ec36f862ba00632090be496fca2304d59796bd8b59 Oct 31 00:07:15.824685 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:07:15.824658 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e0751b65-5lgb6"] Oct 31 00:07:15.903359 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:07:15.896487 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:07:16.065416 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:07:16.065379 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c81c200d-pjrnb"] Oct 31 00:07:16.069509 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:07:16.069442 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c81c200d-pjrnb"] Oct 31 00:07:16.364862 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:07:16.364759 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e0751b65-5lgb6" event={"ID":"0a735e34-9f5b-44bc-8155-cfec5319b710","Type":"ContainerStarted","Data":"c91807b9bc94a00f3405da15edff2caa442cc5056215fb0d07b9826453e94a78"} Oct 31 00:07:16.364862 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:07:16.364826 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e0751b65-5lgb6" event={"ID":"0a735e34-9f5b-44bc-8155-cfec5319b710","Type":"ContainerStarted","Data":"7e1288fd4822103e183e13ec36f862ba00632090be496fca2304d59796bd8b59"} Oct 31 00:07:16.483933 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:07:16.483880 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e0751b65-5lgb6" podStartSLOduration=2.483866654 podStartE2EDuration="2.483866654s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:07:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:07:16.482930984 +0000 UTC m=+1549.519261822" watchObservedRunningTime="2026-10-31 00:07:16.483866654 +0000 UTC m=+1549.520197475" Oct 31 00:07:17.091364 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:07:17.091280 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a54b0594-3037-4e79-83bf-f9e13b14d204" path="/var/lib/kubelet/pods/a54b0594-3037-4e79-83bf-f9e13b14d204/volumes" Oct 31 00:07:17.376216 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:07:17.376119 4927 generic.go:356] "Generic (PLEG): container finished" podID="0a735e34-9f5b-44bc-8155-cfec5319b710" containerID="c91807b9bc94a00f3405da15edff2caa442cc5056215fb0d07b9826453e94a78" exitCode=0 Oct 31 00:07:17.376416 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:07:17.376209 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e0751b65-5lgb6" event={"ID":"0a735e34-9f5b-44bc-8155-cfec5319b710","Type":"ContainerDied","Data":"c91807b9bc94a00f3405da15edff2caa442cc5056215fb0d07b9826453e94a78"} Oct 31 00:07:18.590133 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:07:18.590109 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e0751b65-5lgb6" Oct 31 00:07:18.757910 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:07:18.757868 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0a735e34-9f5b-44bc-8155-cfec5319b710-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a735e34-9f5b-44bc-8155-cfec5319b710-maas-api-xks-ca\") pod \"0a735e34-9f5b-44bc-8155-cfec5319b710\" (UID: \"0a735e34-9f5b-44bc-8155-cfec5319b710\") " Oct 31 00:07:18.758242 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:07:18.758222 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a735e34-9f5b-44bc-8155-cfec5319b710-kube-api-access-79wc9\" (UniqueName: \"kubernetes.io/projected/0a735e34-9f5b-44bc-8155-cfec5319b710-kube-api-access-79wc9\") pod \"0a735e34-9f5b-44bc-8155-cfec5319b710\" (UID: \"0a735e34-9f5b-44bc-8155-cfec5319b710\") " Oct 31 00:07:18.758405 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:07:18.758296 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a735e34-9f5b-44bc-8155-cfec5319b710-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a735e34-9f5b-44bc-8155-cfec5319b710-maas-api-openshift-ca\") pod \"0a735e34-9f5b-44bc-8155-cfec5319b710\" (UID: \"0a735e34-9f5b-44bc-8155-cfec5319b710\") " Oct 31 00:07:18.759004 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:07:18.758964 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a735e34-9f5b-44bc-8155-cfec5319b710-maas-api-openshift-ca" pod "0a735e34-9f5b-44bc-8155-cfec5319b710" (UID: "0a735e34-9f5b-44bc-8155-cfec5319b710"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:07:18.759348 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:07:18.759309 4927 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a735e34-9f5b-44bc-8155-cfec5319b710-maas-api-openshift-ca\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:07:18.762738 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:07:18.762709 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a735e34-9f5b-44bc-8155-cfec5319b710-kube-api-access-79wc9" pod "0a735e34-9f5b-44bc-8155-cfec5319b710" (UID: "0a735e34-9f5b-44bc-8155-cfec5319b710"). InnerVolumeSpecName "kube-api-access-79wc9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:07:18.762853 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:07:18.762822 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a735e34-9f5b-44bc-8155-cfec5319b710-maas-api-xks-ca" pod "0a735e34-9f5b-44bc-8155-cfec5319b710" (UID: "0a735e34-9f5b-44bc-8155-cfec5319b710"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:07:18.861299 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:07:18.861265 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-79wc9\" (UniqueName: \"kubernetes.io/projected/0a735e34-9f5b-44bc-8155-cfec5319b710-kube-api-access-79wc9\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:07:18.861299 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:07:18.861296 4927 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a735e34-9f5b-44bc-8155-cfec5319b710-maas-api-xks-ca\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:07:19.397754 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:07:19.397671 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e0751b65-5lgb6" Oct 31 00:07:19.397754 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:07:19.397704 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e0751b65-5lgb6" event={"ID":"0a735e34-9f5b-44bc-8155-cfec5319b710","Type":"ContainerDied","Data":"7e1288fd4822103e183e13ec36f862ba00632090be496fca2304d59796bd8b59"} Oct 31 00:07:19.397961 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:07:19.397772 4927 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7e1288fd4822103e183e13ec36f862ba00632090be496fca2304d59796bd8b59" Oct 31 00:07:19.680108 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:07:19.680070 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sqv69" containerName="extractor" Oct 31 00:07:20.784521 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:07:20.784485 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:07:25.353832 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:07:25.353800 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-a251fdfb-6fb6bf7d6b-xmmgr" Oct 31 00:07:25.354414 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:07:25.353876 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-a251fdfb-6f8487684c-6qzl9" Oct 31 00:07:25.786905 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:07:25.786861 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:07:26.058369 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:07:26.058250 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-09732b29-8wg8w"] Oct 31 00:07:26.064292 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:07:26.064261 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-09732b29-8wg8w"] Oct 31 00:07:26.501823 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:07:26.501783 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:07:26.819688 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:07:26.819586 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-e0751b65-7d7cd79459-76xvm"] Oct 31 00:07:27.072085 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:07:27.071998 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:07:27.089987 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:07:27.089953 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ed957ff-0566-4f8a-a9fe-21a5a6b428e9" path="/var/lib/kubelet/pods/5ed957ff-0566-4f8a-a9fe-21a5a6b428e9/volumes" Oct 31 00:07:27.441294 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:07:27.435707 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:07:29.074418 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:07:29.074379 4927 scope.go:122] "RemoveContainer" containerID="24151378351b9d041b33b1050925d646d547b81e1fa479e376e84f7ca6df394f" Oct 31 00:07:29.088807 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:07:29.088783 4927 scope.go:122] "RemoveContainer" containerID="0a21023ac4719723f21af0c0e10d57cd8ac19b11e93eb5444f5a0b08de348335" Oct 31 00:07:30.376606 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:07:30.376548 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:07:31.076399 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:07:31.076365 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01559921-kq4sn"] Oct 31 00:07:31.189352 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:07:31.175704 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01559921-kq4sn"] Oct 31 00:07:31.831868 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:07:31.831810 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-e0751b65-7d7cd79459-76xvm" podUID="ecce302f-aa28-48f3-86b1-374399cd6540" containerName="payload-pre-processing" containerID="cri-o://188a1e9b3dd78fe1bb546ad232bf8e0c359561dc0b114ef937b2023182f1eba6" gracePeriod=25 Oct 31 00:07:32.083659 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:07:32.078105 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-27170a5e-j56gh"] Oct 31 00:07:32.142820 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:07:32.142761 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-27170a5e-j56gh"] Oct 31 00:07:32.721858 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:07:32.721817 4927 generic.go:356] "Generic (PLEG): container finished" podID="ecce302f-aa28-48f3-86b1-374399cd6540" containerID="188a1e9b3dd78fe1bb546ad232bf8e0c359561dc0b114ef937b2023182f1eba6" exitCode=0 Oct 31 00:07:32.722137 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:07:32.722067 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-e0751b65-7d7cd79459-76xvm" event={"ID":"ecce302f-aa28-48f3-86b1-374399cd6540","Type":"ContainerDied","Data":"188a1e9b3dd78fe1bb546ad232bf8e0c359561dc0b114ef937b2023182f1eba6"} Oct 31 00:07:32.722137 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:07:32.722123 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-e0751b65-7d7cd79459-76xvm" event={"ID":"ecce302f-aa28-48f3-86b1-374399cd6540","Type":"ContainerDied","Data":"9e50bd7996162a381067877dcad4c1ab24d0bef97743d8e9f13eed169b6abcb6"} Oct 31 00:07:32.722410 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:07:32.722140 4927 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9e50bd7996162a381067877dcad4c1ab24d0bef97743d8e9f13eed169b6abcb6" Oct 31 00:07:32.725835 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:07:32.725816 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-e0751b65-7d7cd79459-76xvm" Oct 31 00:07:32.834014 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:07:32.833980 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ecce302f-aa28-48f3-86b1-374399cd6540-kube-api-access-gszwr\" (UniqueName: \"kubernetes.io/projected/ecce302f-aa28-48f3-86b1-374399cd6540-kube-api-access-gszwr\") pod \"ecce302f-aa28-48f3-86b1-374399cd6540\" (UID: \"ecce302f-aa28-48f3-86b1-374399cd6540\") " Oct 31 00:07:32.834526 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:07:32.834028 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ecce302f-aa28-48f3-86b1-374399cd6540-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ecce302f-aa28-48f3-86b1-374399cd6540-plugins-config-volume\") pod \"ecce302f-aa28-48f3-86b1-374399cd6540\" (UID: \"ecce302f-aa28-48f3-86b1-374399cd6540\") " Oct 31 00:07:32.834602 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:07:32.834524 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ecce302f-aa28-48f3-86b1-374399cd6540-plugins-config-volume" pod "ecce302f-aa28-48f3-86b1-374399cd6540" (UID: "ecce302f-aa28-48f3-86b1-374399cd6540"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:07:32.834721 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:07:32.834700 4927 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ecce302f-aa28-48f3-86b1-374399cd6540-plugins-config-volume\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:07:32.836236 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:07:32.836214 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ecce302f-aa28-48f3-86b1-374399cd6540-kube-api-access-gszwr" pod "ecce302f-aa28-48f3-86b1-374399cd6540" (UID: "ecce302f-aa28-48f3-86b1-374399cd6540"). InnerVolumeSpecName "kube-api-access-gszwr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:07:32.936429 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:07:32.936391 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gszwr\" (UniqueName: \"kubernetes.io/projected/ecce302f-aa28-48f3-86b1-374399cd6540-kube-api-access-gszwr\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:07:33.088922 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:07:33.088882 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="26b1d951-786a-4e89-8117-4158af8b15f1" path="/var/lib/kubelet/pods/26b1d951-786a-4e89-8117-4158af8b15f1/volumes" Oct 31 00:07:33.089536 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:07:33.089518 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c899441-013e-4ff6-a7a2-d8ea415215dd" path="/var/lib/kubelet/pods/2c899441-013e-4ff6-a7a2-d8ea415215dd/volumes" Oct 31 00:07:33.746085 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:07:33.746046 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-e0751b65-7d7cd79459-76xvm" Oct 31 00:07:33.941073 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:07:33.941033 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-e0751b65-7d7cd79459-76xvm"] Oct 31 00:07:33.987858 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:07:33.987818 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-e0751b65-7d7cd79459-76xvm"] Oct 31 00:07:34.937256 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:07:34.937213 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:07:35.175197 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:07:35.175159 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ecce302f-aa28-48f3-86b1-374399cd6540" path="/var/lib/kubelet/pods/ecce302f-aa28-48f3-86b1-374399cd6540/volumes" Oct 31 00:07:39.478558 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:07:39.478523 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:07:40.054598 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:07:40.054556 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-270eef34-4b2nh"] Oct 31 00:07:40.105580 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:07:40.105542 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-270eef34-4b2nh"] Oct 31 00:07:41.093157 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:07:41.093113 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="32855a90-baa7-4b9a-b1cb-c97ab9ca8e7d" path="/var/lib/kubelet/pods/32855a90-baa7-4b9a-b1cb-c97ab9ca8e7d/volumes" Oct 31 00:07:43.921117 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:07:43.921079 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:07:44.074882 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:07:44.074842 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:07:44.282145 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:07:44.282065 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:07:45.014845 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:07:45.014812 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:07:45.052788 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:07:45.052755 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bdd5efd5-9957g"] Oct 31 00:07:45.103497 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:07:45.103461 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bdd5efd5-9957g"] Oct 31 00:07:45.926518 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:07:45.926477 4927 patch_prober.go:29] interesting pod/e2e-derive-09efcb6d-openshift-default-7c46fd677c-6wtzg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:07:45.926688 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:07:45.926549 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-09efcb6d-openshift-default-7c46fd677c-6wtzg" podUID="d403f39c-fdf7-4a18-b6c6-60425475307d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:07:47.134153 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:07:47.134038 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4baf4a61-aae7-4f15-9b1a-fff0ed014c9b" path="/var/lib/kubelet/pods/4baf4a61-aae7-4f15-9b1a-fff0ed014c9b/volumes" Oct 31 00:07:47.907035 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:07:47.907002 4927 generic.go:356] "Generic (PLEG): container finished" podID="d403f39c-fdf7-4a18-b6c6-60425475307d" containerID="813b90da108fa46f8396f51017fb233e30c03277f9b21df73a24237ba27c8b95" exitCode=0 Oct 31 00:07:47.907213 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:07:47.907082 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-09efcb6d-openshift-default-7c46fd677c-6wtzg" event={"ID":"d403f39c-fdf7-4a18-b6c6-60425475307d","Type":"ContainerDied","Data":"813b90da108fa46f8396f51017fb233e30c03277f9b21df73a24237ba27c8b95"} Oct 31 00:07:47.907691 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:07:47.907664 4927 scope.go:122] "RemoveContainer" containerID="813b90da108fa46f8396f51017fb233e30c03277f9b21df73a24237ba27c8b95" Oct 31 00:07:47.909572 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:07:47.909540 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:07:47.909679 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:07:47.909613 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:07:47.909679 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:07:47.909645 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:07:48.666547 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:07:48.666510 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:07:48.921779 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:07:48.921675 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-09efcb6d-openshift-default-7c46fd677c-6wtzg" event={"ID":"d403f39c-fdf7-4a18-b6c6-60425475307d","Type":"ContainerStarted","Data":"3592cc00707b3971a162844ca12c8950340c36d51e3906f5d9316c542b0cfff4"} Oct 31 00:07:49.679822 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:07:49.679785 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sqv69" containerName="extractor" Oct 31 00:07:49.881424 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:07:49.881390 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-09efcb6d-openshift-default-7c46fd677c-6wtzg" Oct 31 00:07:49.881710 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:07:49.881680 4927 patch_prober.go:29] interesting pod/e2e-derive-09efcb6d-openshift-default-7c46fd677c-6wtzg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body= Oct 31 00:07:49.881835 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:07:49.881725 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-09efcb6d-openshift-default-7c46fd677c-6wtzg" podUID="d403f39c-fdf7-4a18-b6c6-60425475307d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" Oct 31 00:07:50.881673 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:07:50.881634 4927 patch_prober.go:29] interesting pod/e2e-derive-09efcb6d-openshift-default-7c46fd677c-6wtzg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body= Oct 31 00:07:50.882098 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:07:50.881691 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-09efcb6d-openshift-default-7c46fd677c-6wtzg" podUID="d403f39c-fdf7-4a18-b6c6-60425475307d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" Oct 31 00:07:51.881793 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:07:51.881757 4927 patch_prober.go:29] interesting pod/e2e-derive-09efcb6d-openshift-default-7c46fd677c-6wtzg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body= Oct 31 00:07:51.882181 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:07:51.881816 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-09efcb6d-openshift-default-7c46fd677c-6wtzg" podUID="d403f39c-fdf7-4a18-b6c6-60425475307d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" Oct 31 00:07:52.060047 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:07:52.060004 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fd61788b-nmjjt"] Oct 31 00:07:52.119356 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:07:52.117379 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-98d91d7d-b7th9"] Oct 31 00:07:52.142812 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:07:52.142722 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fd61788b-nmjjt"] Oct 31 00:07:52.149688 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:07:52.149656 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-98d91d7d-b7th9"] Oct 31 00:07:52.882520 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:07:52.882482 4927 patch_prober.go:29] interesting pod/e2e-derive-09efcb6d-openshift-default-7c46fd677c-6wtzg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body= Oct 31 00:07:52.882928 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:07:52.882537 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-09efcb6d-openshift-default-7c46fd677c-6wtzg" podUID="d403f39c-fdf7-4a18-b6c6-60425475307d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" Oct 31 00:07:53.090256 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:07:53.090224 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="39cb8c11-9c39-48ef-b94e-822f423e464b" path="/var/lib/kubelet/pods/39cb8c11-9c39-48ef-b94e-822f423e464b/volumes" Oct 31 00:07:53.090761 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:07:53.090743 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da3ace09-2b4e-4ccd-9fb3-31473672a48f" path="/var/lib/kubelet/pods/da3ace09-2b4e-4ccd-9fb3-31473672a48f/volumes" Oct 31 00:07:53.200105 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:07:53.200046 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:07:53.882389 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:07:53.882324 4927 patch_prober.go:29] interesting pod/e2e-derive-09efcb6d-openshift-default-7c46fd677c-6wtzg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body= Oct 31 00:07:53.882547 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:07:53.882412 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-09efcb6d-openshift-default-7c46fd677c-6wtzg" podUID="d403f39c-fdf7-4a18-b6c6-60425475307d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" Oct 31 00:07:54.885701 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:07:54.885670 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-09efcb6d-openshift-default-7c46fd677c-6wtzg" Oct 31 00:07:54.886230 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:07:54.886204 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-09efcb6d-openshift-default-7c46fd677c-6wtzg" Oct 31 00:07:54.886940 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:07:54.886916 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-09efcb6d-openshift-default-7c46fd677c-6wtzg" Oct 31 00:07:57.814860 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:07:57.814821 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:07:58.056420 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:07:58.056387 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d93b6188-cbrjh"] Oct 31 00:07:58.105998 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:07:58.105928 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d93b6188-cbrjh"] Oct 31 00:07:59.056131 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:07:59.056101 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-499c7567-66x7m"] Oct 31 00:07:59.091145 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:07:59.091112 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ccaf3489-e47d-4dfb-8d46-aa559aeeeb5b" path="/var/lib/kubelet/pods/ccaf3489-e47d-4dfb-8d46-aa559aeeeb5b/volumes" Oct 31 00:07:59.106996 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:07:59.106969 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-499c7567-66x7m"] Oct 31 00:08:01.089993 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:01.089959 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4178174c-aa4a-4bcf-8147-6f9d5987d9ca" path="/var/lib/kubelet/pods/4178174c-aa4a-4bcf-8147-6f9d5987d9ca/volumes" Oct 31 00:08:02.271215 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:02.271178 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:08:02.474362 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:02.472863 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:08:02.637408 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:02.637269 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:08:03.532261 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:03.532223 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-09efcb6d-lt68p"] Oct 31 00:08:03.533358 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:03.533316 4927 cpu_manager.go:414] "containerMap: removing container" podUID="ecce302f-aa28-48f3-86b1-374399cd6540" containerName="payload-pre-processing" Oct 31 00:08:03.533475 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:03.533358 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ecce302f-aa28-48f3-86b1-374399cd6540" containerName="payload-pre-processing" Oct 31 00:08:03.533475 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:03.533445 4927 cpu_manager.go:414] "containerMap: removing container" podUID="0a735e34-9f5b-44bc-8155-cfec5319b710" containerName="revoke-keys" Oct 31 00:08:03.533475 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:03.533453 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a735e34-9f5b-44bc-8155-cfec5319b710" containerName="revoke-keys" Oct 31 00:08:03.533653 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:03.533629 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a735e34-9f5b-44bc-8155-cfec5319b710" containerName="revoke-keys" Oct 31 00:08:03.533653 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:03.533642 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="ecce302f-aa28-48f3-86b1-374399cd6540" containerName="payload-pre-processing" Oct 31 00:08:03.539490 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:03.539462 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-09efcb6d-lt68p" Oct 31 00:08:03.595808 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:03.595769 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-09efcb6d-lt68p"] Oct 31 00:08:03.635358 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:03.635294 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96d362b8-d811-4ce5-abee-6b47f1482045-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-09efcb6d-lt68p\" (UID: \"96d362b8-d811-4ce5-abee-6b47f1482045\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-09efcb6d-lt68p" Oct 31 00:08:03.635554 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:03.635380 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96d362b8-d811-4ce5-abee-6b47f1482045-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-09efcb6d-lt68p\" (UID: \"96d362b8-d811-4ce5-abee-6b47f1482045\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-09efcb6d-lt68p" Oct 31 00:08:03.635709 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:03.635682 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4dcw\" (UniqueName: \"kubernetes.io/projected/96d362b8-d811-4ce5-abee-6b47f1482045-kube-api-access-d4dcw\") pod \"maas-api-revoke-keys-e2e-derive-09efcb6d-lt68p\" (UID: \"96d362b8-d811-4ce5-abee-6b47f1482045\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-09efcb6d-lt68p" Oct 31 00:08:03.636597 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:03.636575 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4fdzs\"" Oct 31 00:08:03.636751 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:03.636591 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:08:03.737450 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:03.737391 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4dcw\" (UniqueName: \"kubernetes.io/projected/96d362b8-d811-4ce5-abee-6b47f1482045-kube-api-access-d4dcw\") pod \"maas-api-revoke-keys-e2e-derive-09efcb6d-lt68p\" (UID: \"96d362b8-d811-4ce5-abee-6b47f1482045\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-09efcb6d-lt68p" Oct 31 00:08:03.737638 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:03.737625 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96d362b8-d811-4ce5-abee-6b47f1482045-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-09efcb6d-lt68p\" (UID: \"96d362b8-d811-4ce5-abee-6b47f1482045\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-09efcb6d-lt68p" Oct 31 00:08:03.737687 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:03.737653 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96d362b8-d811-4ce5-abee-6b47f1482045-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-09efcb6d-lt68p\" (UID: \"96d362b8-d811-4ce5-abee-6b47f1482045\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-09efcb6d-lt68p" Oct 31 00:08:03.738290 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:03.738263 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96d362b8-d811-4ce5-abee-6b47f1482045-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-09efcb6d-lt68p\" (UID: \"96d362b8-d811-4ce5-abee-6b47f1482045\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-09efcb6d-lt68p" Oct 31 00:08:03.740083 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:03.740063 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96d362b8-d811-4ce5-abee-6b47f1482045-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-09efcb6d-lt68p\" (UID: \"96d362b8-d811-4ce5-abee-6b47f1482045\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-09efcb6d-lt68p" Oct 31 00:08:03.800143 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:03.800069 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4dcw\" (UniqueName: \"kubernetes.io/projected/96d362b8-d811-4ce5-abee-6b47f1482045-kube-api-access-d4dcw\") pod \"maas-api-revoke-keys-e2e-derive-09efcb6d-lt68p\" (UID: \"96d362b8-d811-4ce5-abee-6b47f1482045\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-09efcb6d-lt68p" Oct 31 00:08:03.861401 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:03.861362 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-09efcb6d-lt68p" Oct 31 00:08:04.252449 ip-10-0-141-180 kubenswrapper[4927]: W1031 00:08:04.252416 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96d362b8_d811_4ce5_abee_6b47f1482045.slice/crio-8c816715bd5be2312746bc8a9a3527e10e2e7040e6fe05a56c4c45fa8b2d02b4 WatchSource:0}: Error finding container 8c816715bd5be2312746bc8a9a3527e10e2e7040e6fe05a56c4c45fa8b2d02b4: Status 404 returned error can't find the container with id 8c816715bd5be2312746bc8a9a3527e10e2e7040e6fe05a56c4c45fa8b2d02b4 Oct 31 00:08:04.255219 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:04.255193 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-09efcb6d-lt68p"] Oct 31 00:08:05.149103 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:05.149064 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-09efcb6d-lt68p" event={"ID":"96d362b8-d811-4ce5-abee-6b47f1482045","Type":"ContainerStarted","Data":"c0300d5394dd3988178b39aa232395473496f27563c61f79ff1baca9b0574276"} Oct 31 00:08:05.149538 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:05.149134 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-09efcb6d-lt68p" event={"ID":"96d362b8-d811-4ce5-abee-6b47f1482045","Type":"ContainerStarted","Data":"8c816715bd5be2312746bc8a9a3527e10e2e7040e6fe05a56c4c45fa8b2d02b4"} Oct 31 00:08:06.053821 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:06.053764 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-09efcb6d-lt68p" podStartSLOduration=3.05374707 podStartE2EDuration="3.05374707s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:08:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:08:05.310857736 +0000 UTC m=+1598.347188604" watchObservedRunningTime="2026-10-31 00:08:06.05374707 +0000 UTC m=+1599.090077936" Oct 31 00:08:06.054507 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:06.054488 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dcc30ad5-tjf6s"] Oct 31 00:08:06.104716 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:06.104688 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dcc30ad5-tjf6s"] Oct 31 00:08:06.160993 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:06.160962 4927 generic.go:356] "Generic (PLEG): container finished" podID="96d362b8-d811-4ce5-abee-6b47f1482045" containerID="c0300d5394dd3988178b39aa232395473496f27563c61f79ff1baca9b0574276" exitCode=0 Oct 31 00:08:06.161429 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:06.161002 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-09efcb6d-lt68p" event={"ID":"96d362b8-d811-4ce5-abee-6b47f1482045","Type":"ContainerDied","Data":"c0300d5394dd3988178b39aa232395473496f27563c61f79ff1baca9b0574276"} Oct 31 00:08:07.089425 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:07.089392 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ada7676a-5992-46c1-8a41-870d23a74253" path="/var/lib/kubelet/pods/ada7676a-5992-46c1-8a41-870d23a74253/volumes" Oct 31 00:08:07.379195 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:07.379169 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-09efcb6d-lt68p" Oct 31 00:08:07.389209 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:07.389184 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:08:07.399700 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:07.399675 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96d362b8-d811-4ce5-abee-6b47f1482045-kube-api-access-d4dcw\" (UniqueName: \"kubernetes.io/projected/96d362b8-d811-4ce5-abee-6b47f1482045-kube-api-access-d4dcw\") pod \"96d362b8-d811-4ce5-abee-6b47f1482045\" (UID: \"96d362b8-d811-4ce5-abee-6b47f1482045\") " Oct 31 00:08:07.399856 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:07.399829 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/96d362b8-d811-4ce5-abee-6b47f1482045-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96d362b8-d811-4ce5-abee-6b47f1482045-maas-api-openshift-ca\") pod \"96d362b8-d811-4ce5-abee-6b47f1482045\" (UID: \"96d362b8-d811-4ce5-abee-6b47f1482045\") " Oct 31 00:08:07.400083 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:07.400034 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/96d362b8-d811-4ce5-abee-6b47f1482045-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96d362b8-d811-4ce5-abee-6b47f1482045-maas-api-xks-ca\") pod \"96d362b8-d811-4ce5-abee-6b47f1482045\" (UID: \"96d362b8-d811-4ce5-abee-6b47f1482045\") " Oct 31 00:08:07.400206 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:07.400180 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96d362b8-d811-4ce5-abee-6b47f1482045-maas-api-openshift-ca" pod "96d362b8-d811-4ce5-abee-6b47f1482045" (UID: "96d362b8-d811-4ce5-abee-6b47f1482045"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:08:07.400562 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:07.400538 4927 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96d362b8-d811-4ce5-abee-6b47f1482045-maas-api-openshift-ca\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:08:07.402020 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:07.401992 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96d362b8-d811-4ce5-abee-6b47f1482045-kube-api-access-d4dcw" pod "96d362b8-d811-4ce5-abee-6b47f1482045" (UID: "96d362b8-d811-4ce5-abee-6b47f1482045"). InnerVolumeSpecName "kube-api-access-d4dcw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:08:07.402020 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:07.402007 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/96d362b8-d811-4ce5-abee-6b47f1482045-maas-api-xks-ca" pod "96d362b8-d811-4ce5-abee-6b47f1482045" (UID: "96d362b8-d811-4ce5-abee-6b47f1482045"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:08:07.503357 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:07.502472 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d4dcw\" (UniqueName: \"kubernetes.io/projected/96d362b8-d811-4ce5-abee-6b47f1482045-kube-api-access-d4dcw\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:08:07.503357 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:07.502511 4927 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96d362b8-d811-4ce5-abee-6b47f1482045-maas-api-xks-ca\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:08:08.182496 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:08.182467 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-09efcb6d-lt68p" Oct 31 00:08:08.182496 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:08.182491 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-09efcb6d-lt68p" event={"ID":"96d362b8-d811-4ce5-abee-6b47f1482045","Type":"ContainerDied","Data":"8c816715bd5be2312746bc8a9a3527e10e2e7040e6fe05a56c4c45fa8b2d02b4"} Oct 31 00:08:08.182713 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:08.182538 4927 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8c816715bd5be2312746bc8a9a3527e10e2e7040e6fe05a56c4c45fa8b2d02b4" Oct 31 00:08:09.053373 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:09.053297 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79cd756e-jfn6r"] Oct 31 00:08:09.104314 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:09.104274 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79cd756e-jfn6r"] Oct 31 00:08:11.007193 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:11.007150 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:08:11.089980 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:11.089928 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b3f747b-72be-4a13-a670-e150d716f39e" path="/var/lib/kubelet/pods/6b3f747b-72be-4a13-a670-e150d716f39e/volumes" Oct 31 00:08:11.171921 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:11.171886 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:08:12.053551 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:12.053514 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8b7fb02e-q6fx9"] Oct 31 00:08:12.103880 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:12.103844 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8b7fb02e-q6fx9"] Oct 31 00:08:12.347323 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:12.347225 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:08:12.575290 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:12.575256 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:08:13.089186 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:13.089150 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f00f65ee-2871-4e51-bcde-123f07c1d865" path="/var/lib/kubelet/pods/f00f65ee-2871-4e51-bcde-123f07c1d865/volumes" Oct 31 00:08:15.852435 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:15.852398 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7c8802f8-vtdwf"] Oct 31 00:08:15.853213 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:15.853197 4927 cpu_manager.go:414] "containerMap: removing container" podUID="96d362b8-d811-4ce5-abee-6b47f1482045" containerName="revoke-keys" Oct 31 00:08:15.853269 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:15.853214 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96d362b8-d811-4ce5-abee-6b47f1482045" containerName="revoke-keys" Oct 31 00:08:15.853409 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:15.853398 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="96d362b8-d811-4ce5-abee-6b47f1482045" containerName="revoke-keys" Oct 31 00:08:15.858727 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:15.858699 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7c8802f8-vtdwf" Oct 31 00:08:15.945249 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:15.945205 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7c8802f8-vtdwf"] Oct 31 00:08:15.962770 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:15.962738 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:08:15.963015 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:15.962826 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4fdzs\"" Oct 31 00:08:16.264364 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:16.250030 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e5c0632-30c2-4d21-a27b-eb252afe5747-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-7c8802f8-vtdwf\" (UID: \"7e5c0632-30c2-4d21-a27b-eb252afe5747\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7c8802f8-vtdwf" Oct 31 00:08:16.264364 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:16.250297 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ftrrw\" (UniqueName: \"kubernetes.io/projected/7e5c0632-30c2-4d21-a27b-eb252afe5747-kube-api-access-ftrrw\") pod \"maas-api-revoke-keys-e2e-ait-7c8802f8-vtdwf\" (UID: \"7e5c0632-30c2-4d21-a27b-eb252afe5747\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7c8802f8-vtdwf" Oct 31 00:08:16.264364 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:16.250479 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e5c0632-30c2-4d21-a27b-eb252afe5747-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-7c8802f8-vtdwf\" (UID: \"7e5c0632-30c2-4d21-a27b-eb252afe5747\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7c8802f8-vtdwf" Oct 31 00:08:16.350548 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:16.350504 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0ef2f748-jlssh"] Oct 31 00:08:16.353348 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:16.353295 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e5c0632-30c2-4d21-a27b-eb252afe5747-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-7c8802f8-vtdwf\" (UID: \"7e5c0632-30c2-4d21-a27b-eb252afe5747\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7c8802f8-vtdwf" Oct 31 00:08:16.353708 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:16.353688 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftrrw\" (UniqueName: \"kubernetes.io/projected/7e5c0632-30c2-4d21-a27b-eb252afe5747-kube-api-access-ftrrw\") pod \"maas-api-revoke-keys-e2e-ait-7c8802f8-vtdwf\" (UID: \"7e5c0632-30c2-4d21-a27b-eb252afe5747\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7c8802f8-vtdwf" Oct 31 00:08:16.372699 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:16.372657 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e5c0632-30c2-4d21-a27b-eb252afe5747-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-7c8802f8-vtdwf\" (UID: \"7e5c0632-30c2-4d21-a27b-eb252afe5747\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7c8802f8-vtdwf" Oct 31 00:08:16.385816 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:16.356387 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e5c0632-30c2-4d21-a27b-eb252afe5747-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-7c8802f8-vtdwf\" (UID: \"7e5c0632-30c2-4d21-a27b-eb252afe5747\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7c8802f8-vtdwf" Oct 31 00:08:16.433942 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:16.433907 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e5c0632-30c2-4d21-a27b-eb252afe5747-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-7c8802f8-vtdwf\" (UID: \"7e5c0632-30c2-4d21-a27b-eb252afe5747\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7c8802f8-vtdwf" Oct 31 00:08:16.459379 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:16.459155 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ftrrw\" (UniqueName: \"kubernetes.io/projected/7e5c0632-30c2-4d21-a27b-eb252afe5747-kube-api-access-ftrrw\") pod \"maas-api-revoke-keys-e2e-ait-7c8802f8-vtdwf\" (UID: \"7e5c0632-30c2-4d21-a27b-eb252afe5747\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7c8802f8-vtdwf" Oct 31 00:08:16.489366 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:16.489228 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7c8802f8-vtdwf" Oct 31 00:08:16.504396 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:16.500384 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0ef2f748-jlssh"] Oct 31 00:08:16.575786 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:16.575687 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-09efcb6d-699987c7fd-j2v69"] Oct 31 00:08:16.662668 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:16.662632 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-09efcb6d-c7bfbb5cf-fsvrl"] Oct 31 00:08:16.704608 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:16.704171 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:08:16.885821 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:16.885680 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:08:17.123980 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:17.099899 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="289b001c-26fa-413b-957e-d7928ca24c1b" path="/var/lib/kubelet/pods/289b001c-26fa-413b-957e-d7928ca24c1b/volumes" Oct 31 00:08:17.584365 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:17.581611 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7c8802f8-vtdwf"] Oct 31 00:08:17.619873 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:17.619826 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:08:18.163487 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:18.163451 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4d65879a-l2xz9"] Oct 31 00:08:18.313106 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:18.313068 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4d65879a-l2xz9"] Oct 31 00:08:18.699223 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:18.699187 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7c8802f8-vtdwf" event={"ID":"7e5c0632-30c2-4d21-a27b-eb252afe5747","Type":"ContainerStarted","Data":"09df47c3c1ba9856824b0c38fa11576d0ca01e8f24f94c845879102727695432"} Oct 31 00:08:18.699438 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:18.699239 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7c8802f8-vtdwf" event={"ID":"7e5c0632-30c2-4d21-a27b-eb252afe5747","Type":"ContainerStarted","Data":"b010cc6dde4db8c4016745364c064ce355282f204c72d1f05cd83bac84a5eb5e"} Oct 31 00:08:18.835117 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:18.835055 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7c8802f8-vtdwf" podStartSLOduration=3.835035299 podStartE2EDuration="3.835035299s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:08:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:08:18.831490732 +0000 UTC m=+1611.867821554" watchObservedRunningTime="2026-10-31 00:08:18.835035299 +0000 UTC m=+1611.871366122" Oct 31 00:08:19.091842 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:19.091808 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b6c30728-965b-4edf-a59b-38d13d7459ea" path="/var/lib/kubelet/pods/b6c30728-965b-4edf-a59b-38d13d7459ea/volumes" Oct 31 00:08:19.356825 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:19.356748 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-16364156-gx8fb"] Oct 31 00:08:19.362969 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:19.362946 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16364156-gx8fb" Oct 31 00:08:19.417450 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:19.417421 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-16364156-gx8fb"] Oct 31 00:08:19.512309 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:19.512277 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cec08908-5b14-4d8c-a101-174d1496e41c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-16364156-gx8fb\" (UID: \"cec08908-5b14-4d8c-a101-174d1496e41c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16364156-gx8fb" Oct 31 00:08:19.512449 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:19.512355 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cec08908-5b14-4d8c-a101-174d1496e41c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-16364156-gx8fb\" (UID: \"cec08908-5b14-4d8c-a101-174d1496e41c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16364156-gx8fb" Oct 31 00:08:19.512618 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:19.512595 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4bdz6\" (UniqueName: \"kubernetes.io/projected/cec08908-5b14-4d8c-a101-174d1496e41c-kube-api-access-4bdz6\") pod \"maas-api-revoke-sub-16364156-gx8fb\" (UID: \"cec08908-5b14-4d8c-a101-174d1496e41c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16364156-gx8fb" Oct 31 00:08:19.599870 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:19.599837 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c8d1a25-ns4dx"] Oct 31 00:08:19.604861 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:19.604842 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c8d1a25-ns4dx" Oct 31 00:08:19.613994 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:19.613963 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cec08908-5b14-4d8c-a101-174d1496e41c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-16364156-gx8fb\" (UID: \"cec08908-5b14-4d8c-a101-174d1496e41c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16364156-gx8fb" Oct 31 00:08:19.614084 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:19.614051 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cec08908-5b14-4d8c-a101-174d1496e41c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-16364156-gx8fb\" (UID: \"cec08908-5b14-4d8c-a101-174d1496e41c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16364156-gx8fb" Oct 31 00:08:19.614380 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:19.614362 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bdz6\" (UniqueName: \"kubernetes.io/projected/cec08908-5b14-4d8c-a101-174d1496e41c-kube-api-access-4bdz6\") pod \"maas-api-revoke-sub-16364156-gx8fb\" (UID: \"cec08908-5b14-4d8c-a101-174d1496e41c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16364156-gx8fb" Oct 31 00:08:19.614833 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:19.614807 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cec08908-5b14-4d8c-a101-174d1496e41c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-16364156-gx8fb\" (UID: \"cec08908-5b14-4d8c-a101-174d1496e41c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16364156-gx8fb" Oct 31 00:08:19.616425 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:19.616397 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cec08908-5b14-4d8c-a101-174d1496e41c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-16364156-gx8fb\" (UID: \"cec08908-5b14-4d8c-a101-174d1496e41c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16364156-gx8fb" Oct 31 00:08:19.663268 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:19.663230 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c8d1a25-ns4dx"] Oct 31 00:08:19.680138 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:19.680110 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4bdz6\" (UniqueName: \"kubernetes.io/projected/cec08908-5b14-4d8c-a101-174d1496e41c-kube-api-access-4bdz6\") pod \"maas-api-revoke-sub-16364156-gx8fb\" (UID: \"cec08908-5b14-4d8c-a101-174d1496e41c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16364156-gx8fb" Oct 31 00:08:19.686055 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:08:19.686020 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sqv69" containerName="extractor" Oct 31 00:08:19.713707 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:19.713678 4927 generic.go:356] "Generic (PLEG): container finished" podID="7e5c0632-30c2-4d21-a27b-eb252afe5747" containerID="09df47c3c1ba9856824b0c38fa11576d0ca01e8f24f94c845879102727695432" exitCode=0 Oct 31 00:08:19.713832 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:19.713763 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7c8802f8-vtdwf" event={"ID":"7e5c0632-30c2-4d21-a27b-eb252afe5747","Type":"ContainerDied","Data":"09df47c3c1ba9856824b0c38fa11576d0ca01e8f24f94c845879102727695432"} Oct 31 00:08:19.715711 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:19.715693 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f369330e-5800-40d6-929c-bc9996aaa3d9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4c8d1a25-ns4dx\" (UID: \"f369330e-5800-40d6-929c-bc9996aaa3d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c8d1a25-ns4dx" Oct 31 00:08:19.715982 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:19.715960 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bbzdz\" (UniqueName: \"kubernetes.io/projected/f369330e-5800-40d6-929c-bc9996aaa3d9-kube-api-access-bbzdz\") pod \"maas-api-revoke-sub-4c8d1a25-ns4dx\" (UID: \"f369330e-5800-40d6-929c-bc9996aaa3d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c8d1a25-ns4dx" Oct 31 00:08:19.716056 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:19.716037 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f369330e-5800-40d6-929c-bc9996aaa3d9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4c8d1a25-ns4dx\" (UID: \"f369330e-5800-40d6-929c-bc9996aaa3d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c8d1a25-ns4dx" Oct 31 00:08:19.818111 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:19.818076 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bbzdz\" (UniqueName: \"kubernetes.io/projected/f369330e-5800-40d6-929c-bc9996aaa3d9-kube-api-access-bbzdz\") pod \"maas-api-revoke-sub-4c8d1a25-ns4dx\" (UID: \"f369330e-5800-40d6-929c-bc9996aaa3d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c8d1a25-ns4dx" Oct 31 00:08:19.818287 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:19.818124 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f369330e-5800-40d6-929c-bc9996aaa3d9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4c8d1a25-ns4dx\" (UID: \"f369330e-5800-40d6-929c-bc9996aaa3d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c8d1a25-ns4dx" Oct 31 00:08:19.818287 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:19.818192 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f369330e-5800-40d6-929c-bc9996aaa3d9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4c8d1a25-ns4dx\" (UID: \"f369330e-5800-40d6-929c-bc9996aaa3d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c8d1a25-ns4dx" Oct 31 00:08:19.818876 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:19.818851 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f369330e-5800-40d6-929c-bc9996aaa3d9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4c8d1a25-ns4dx\" (UID: \"f369330e-5800-40d6-929c-bc9996aaa3d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c8d1a25-ns4dx" Oct 31 00:08:19.820622 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:19.820599 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f369330e-5800-40d6-929c-bc9996aaa3d9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4c8d1a25-ns4dx\" (UID: \"f369330e-5800-40d6-929c-bc9996aaa3d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c8d1a25-ns4dx" Oct 31 00:08:19.881459 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:19.881368 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bbzdz\" (UniqueName: \"kubernetes.io/projected/f369330e-5800-40d6-929c-bc9996aaa3d9-kube-api-access-bbzdz\") pod \"maas-api-revoke-sub-4c8d1a25-ns4dx\" (UID: \"f369330e-5800-40d6-929c-bc9996aaa3d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c8d1a25-ns4dx" Oct 31 00:08:19.920429 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:19.920390 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c8d1a25-ns4dx" Oct 31 00:08:19.979393 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:19.979365 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16364156-gx8fb" Oct 31 00:08:20.319615 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:20.319580 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c8d1a25-ns4dx"] Oct 31 00:08:20.434899 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:20.434871 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-16364156-gx8fb"] Oct 31 00:08:20.512076 ip-10-0-141-180 kubenswrapper[4927]: W1031 00:08:20.512032 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcec08908_5b14_4d8c_a101_174d1496e41c.slice/crio-47df82ac06bc3ebe5de28dbccae74e38bbdc1991b0842531517fea8945de298a WatchSource:0}: Error finding container 47df82ac06bc3ebe5de28dbccae74e38bbdc1991b0842531517fea8945de298a: Status 404 returned error can't find the container with id 47df82ac06bc3ebe5de28dbccae74e38bbdc1991b0842531517fea8945de298a Oct 31 00:08:20.731038 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:20.730984 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c8d1a25-ns4dx" event={"ID":"f369330e-5800-40d6-929c-bc9996aaa3d9","Type":"ContainerStarted","Data":"0e0d626956e8eefaa1076574b2549686604efb5314bd74d30821ccea36860bdf"} Oct 31 00:08:20.732981 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:20.732949 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16364156-gx8fb" event={"ID":"cec08908-5b14-4d8c-a101-174d1496e41c","Type":"ContainerStarted","Data":"47df82ac06bc3ebe5de28dbccae74e38bbdc1991b0842531517fea8945de298a"} Oct 31 00:08:20.922681 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:20.922652 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7c8802f8-vtdwf" Oct 31 00:08:21.041213 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:21.041180 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7e5c0632-30c2-4d21-a27b-eb252afe5747-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e5c0632-30c2-4d21-a27b-eb252afe5747-maas-api-xks-ca\") pod \"7e5c0632-30c2-4d21-a27b-eb252afe5747\" (UID: \"7e5c0632-30c2-4d21-a27b-eb252afe5747\") " Oct 31 00:08:21.041467 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:21.041451 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e5c0632-30c2-4d21-a27b-eb252afe5747-kube-api-access-ftrrw\" (UniqueName: \"kubernetes.io/projected/7e5c0632-30c2-4d21-a27b-eb252afe5747-kube-api-access-ftrrw\") pod \"7e5c0632-30c2-4d21-a27b-eb252afe5747\" (UID: \"7e5c0632-30c2-4d21-a27b-eb252afe5747\") " Oct 31 00:08:21.041555 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:21.041540 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e5c0632-30c2-4d21-a27b-eb252afe5747-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e5c0632-30c2-4d21-a27b-eb252afe5747-maas-api-openshift-ca\") pod \"7e5c0632-30c2-4d21-a27b-eb252afe5747\" (UID: \"7e5c0632-30c2-4d21-a27b-eb252afe5747\") " Oct 31 00:08:21.041947 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:21.041918 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e5c0632-30c2-4d21-a27b-eb252afe5747-maas-api-openshift-ca" pod "7e5c0632-30c2-4d21-a27b-eb252afe5747" (UID: "7e5c0632-30c2-4d21-a27b-eb252afe5747"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:08:21.042302 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:21.042282 4927 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e5c0632-30c2-4d21-a27b-eb252afe5747-maas-api-openshift-ca\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:08:21.068649 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:21.068608 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e5c0632-30c2-4d21-a27b-eb252afe5747-maas-api-xks-ca" pod "7e5c0632-30c2-4d21-a27b-eb252afe5747" (UID: "7e5c0632-30c2-4d21-a27b-eb252afe5747"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:08:21.068803 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:21.068690 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e5c0632-30c2-4d21-a27b-eb252afe5747-kube-api-access-ftrrw" pod "7e5c0632-30c2-4d21-a27b-eb252afe5747" (UID: "7e5c0632-30c2-4d21-a27b-eb252afe5747"). InnerVolumeSpecName "kube-api-access-ftrrw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:08:21.143834 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:21.143799 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ftrrw\" (UniqueName: \"kubernetes.io/projected/7e5c0632-30c2-4d21-a27b-eb252afe5747-kube-api-access-ftrrw\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:08:21.143834 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:21.143835 4927 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e5c0632-30c2-4d21-a27b-eb252afe5747-maas-api-xks-ca\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:08:21.576743 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:21.576699 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-09efcb6d-699987c7fd-j2v69" podUID="ee43bf6b-c4d3-46f6-8808-8064e3412906" containerName="payload-processing" containerID="cri-o://c3e66009bb474ae5f4aa57b627aaa2ccb072b197139d61f21fa44f8e2c9effc1" gracePeriod=25 Oct 31 00:08:21.663346 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:21.663297 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-09efcb6d-c7bfbb5cf-fsvrl" podUID="37daf0f5-13de-43e2-a2a2-d82036836415" containerName="payload-pre-processing" containerID="cri-o://41a4c7a18cafccee7073638afa6e7bc78cfe1ffdb9c682af735571d521e5d65c" gracePeriod=25 Oct 31 00:08:21.745568 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:21.745539 4927 generic.go:356] "Generic (PLEG): container finished" podID="ee43bf6b-c4d3-46f6-8808-8064e3412906" containerID="c3e66009bb474ae5f4aa57b627aaa2ccb072b197139d61f21fa44f8e2c9effc1" exitCode=0 Oct 31 00:08:21.745697 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:21.745610 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-09efcb6d-699987c7fd-j2v69" event={"ID":"ee43bf6b-c4d3-46f6-8808-8064e3412906","Type":"ContainerDied","Data":"c3e66009bb474ae5f4aa57b627aaa2ccb072b197139d61f21fa44f8e2c9effc1"} Oct 31 00:08:21.747202 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:21.747185 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7c8802f8-vtdwf" Oct 31 00:08:21.747323 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:21.747187 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7c8802f8-vtdwf" event={"ID":"7e5c0632-30c2-4d21-a27b-eb252afe5747","Type":"ContainerDied","Data":"b010cc6dde4db8c4016745364c064ce355282f204c72d1f05cd83bac84a5eb5e"} Oct 31 00:08:21.747323 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:21.747308 4927 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b010cc6dde4db8c4016745364c064ce355282f204c72d1f05cd83bac84a5eb5e" Oct 31 00:08:21.748891 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:21.748866 4927 generic.go:356] "Generic (PLEG): container finished" podID="cec08908-5b14-4d8c-a101-174d1496e41c" containerID="3d26b0780840c981c1c50310e82bbea3d89cae5e5e35ee90328360c736e85ced" exitCode=0 Oct 31 00:08:21.748987 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:21.748945 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16364156-gx8fb" event={"ID":"cec08908-5b14-4d8c-a101-174d1496e41c","Type":"ContainerDied","Data":"3d26b0780840c981c1c50310e82bbea3d89cae5e5e35ee90328360c736e85ced"} Oct 31 00:08:21.750694 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:21.750671 4927 generic.go:356] "Generic (PLEG): container finished" podID="f369330e-5800-40d6-929c-bc9996aaa3d9" containerID="a8bf905ff0c4c6178d3399e1f4d9e62a66d247667635d1123edd168d82f9162e" exitCode=0 Oct 31 00:08:21.750781 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:21.750746 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c8d1a25-ns4dx" event={"ID":"f369330e-5800-40d6-929c-bc9996aaa3d9","Type":"ContainerDied","Data":"a8bf905ff0c4c6178d3399e1f4d9e62a66d247667635d1123edd168d82f9162e"} Oct 31 00:08:22.051751 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:22.051677 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-09efcb6d-699987c7fd-j2v69" Oct 31 00:08:22.056883 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:22.056853 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:08:22.064204 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:22.064174 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-47cd94c7-bvjxk"] Oct 31 00:08:22.079449 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:22.075808 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-47cd94c7-bvjxk"] Oct 31 00:08:22.137397 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:22.137373 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-09efcb6d-c7bfbb5cf-fsvrl" Oct 31 00:08:22.168110 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:22.168077 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee43bf6b-c4d3-46f6-8808-8064e3412906-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee43bf6b-c4d3-46f6-8808-8064e3412906-plugins-config-volume\") pod \"ee43bf6b-c4d3-46f6-8808-8064e3412906\" (UID: \"ee43bf6b-c4d3-46f6-8808-8064e3412906\") " Oct 31 00:08:22.168426 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:22.168400 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee43bf6b-c4d3-46f6-8808-8064e3412906-kube-api-access-zqg5j\" (UniqueName: \"kubernetes.io/projected/ee43bf6b-c4d3-46f6-8808-8064e3412906-kube-api-access-zqg5j\") pod \"ee43bf6b-c4d3-46f6-8808-8064e3412906\" (UID: \"ee43bf6b-c4d3-46f6-8808-8064e3412906\") " Oct 31 00:08:22.168519 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:22.168487 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee43bf6b-c4d3-46f6-8808-8064e3412906-plugins-config-volume" pod "ee43bf6b-c4d3-46f6-8808-8064e3412906" (UID: "ee43bf6b-c4d3-46f6-8808-8064e3412906"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:08:22.169599 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:22.169571 4927 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee43bf6b-c4d3-46f6-8808-8064e3412906-plugins-config-volume\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:08:22.188540 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:22.180100 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee43bf6b-c4d3-46f6-8808-8064e3412906-kube-api-access-zqg5j" pod "ee43bf6b-c4d3-46f6-8808-8064e3412906" (UID: "ee43bf6b-c4d3-46f6-8808-8064e3412906"). InnerVolumeSpecName "kube-api-access-zqg5j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:08:22.328408 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:22.270991 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/37daf0f5-13de-43e2-a2a2-d82036836415-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37daf0f5-13de-43e2-a2a2-d82036836415-plugins-config-volume\") pod \"37daf0f5-13de-43e2-a2a2-d82036836415\" (UID: \"37daf0f5-13de-43e2-a2a2-d82036836415\") " Oct 31 00:08:22.328882 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:22.328842 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37daf0f5-13de-43e2-a2a2-d82036836415-kube-api-access-7864m\" (UniqueName: \"kubernetes.io/projected/37daf0f5-13de-43e2-a2a2-d82036836415-kube-api-access-7864m\") pod \"37daf0f5-13de-43e2-a2a2-d82036836415\" (UID: \"37daf0f5-13de-43e2-a2a2-d82036836415\") " Oct 31 00:08:22.401312 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:22.396289 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37daf0f5-13de-43e2-a2a2-d82036836415-plugins-config-volume" pod "37daf0f5-13de-43e2-a2a2-d82036836415" (UID: "37daf0f5-13de-43e2-a2a2-d82036836415"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:08:22.403039 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:22.403011 4927 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37daf0f5-13de-43e2-a2a2-d82036836415-plugins-config-volume\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:08:22.403244 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:22.403229 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zqg5j\" (UniqueName: \"kubernetes.io/projected/ee43bf6b-c4d3-46f6-8808-8064e3412906-kube-api-access-zqg5j\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:08:22.412745 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:22.407121 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37daf0f5-13de-43e2-a2a2-d82036836415-kube-api-access-7864m" pod "37daf0f5-13de-43e2-a2a2-d82036836415" (UID: "37daf0f5-13de-43e2-a2a2-d82036836415"). InnerVolumeSpecName "kube-api-access-7864m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:08:22.505842 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:22.505792 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7864m\" (UniqueName: \"kubernetes.io/projected/37daf0f5-13de-43e2-a2a2-d82036836415-kube-api-access-7864m\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:08:22.811153 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:22.811126 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-09efcb6d-c7bfbb5cf-fsvrl" Oct 31 00:08:22.811721 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:22.811182 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-09efcb6d-c7bfbb5cf-fsvrl" event={"ID":"37daf0f5-13de-43e2-a2a2-d82036836415","Type":"ContainerDied","Data":"41a4c7a18cafccee7073638afa6e7bc78cfe1ffdb9c682af735571d521e5d65c"} Oct 31 00:08:22.811721 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:22.811271 4927 scope.go:122] "RemoveContainer" containerID="41a4c7a18cafccee7073638afa6e7bc78cfe1ffdb9c682af735571d521e5d65c" Oct 31 00:08:22.812404 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:22.812375 4927 generic.go:356] "Generic (PLEG): container finished" podID="37daf0f5-13de-43e2-a2a2-d82036836415" containerID="41a4c7a18cafccee7073638afa6e7bc78cfe1ffdb9c682af735571d521e5d65c" exitCode=0 Oct 31 00:08:22.812650 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:22.812627 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-09efcb6d-c7bfbb5cf-fsvrl" event={"ID":"37daf0f5-13de-43e2-a2a2-d82036836415","Type":"ContainerDied","Data":"29bafd1d56c24685fcc9fc819586983bcab70e4e928593539a785da5efe07b90"} Oct 31 00:08:22.815057 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:22.815032 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-09efcb6d-699987c7fd-j2v69" Oct 31 00:08:22.815257 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:22.815235 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-09efcb6d-699987c7fd-j2v69" event={"ID":"ee43bf6b-c4d3-46f6-8808-8064e3412906","Type":"ContainerDied","Data":"ddd0e5273bebfabd1da5796ea1da977daa4520fdd954047459b8591220395f95"} Oct 31 00:08:22.844111 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:22.844077 4927 scope.go:122] "RemoveContainer" containerID="41a4c7a18cafccee7073638afa6e7bc78cfe1ffdb9c682af735571d521e5d65c" Oct 31 00:08:22.844696 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:08:22.844550 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41a4c7a18cafccee7073638afa6e7bc78cfe1ffdb9c682af735571d521e5d65c\": container with ID starting with 41a4c7a18cafccee7073638afa6e7bc78cfe1ffdb9c682af735571d521e5d65c not found: ID does not exist" containerID="41a4c7a18cafccee7073638afa6e7bc78cfe1ffdb9c682af735571d521e5d65c" Oct 31 00:08:22.844696 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:22.844582 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41a4c7a18cafccee7073638afa6e7bc78cfe1ffdb9c682af735571d521e5d65c"} err="failed to get container status \"41a4c7a18cafccee7073638afa6e7bc78cfe1ffdb9c682af735571d521e5d65c\": rpc error: code = NotFound desc = could not find container \"41a4c7a18cafccee7073638afa6e7bc78cfe1ffdb9c682af735571d521e5d65c\": container with ID starting with 41a4c7a18cafccee7073638afa6e7bc78cfe1ffdb9c682af735571d521e5d65c not found: ID does not exist" Oct 31 00:08:22.844696 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:22.844610 4927 scope.go:122] "RemoveContainer" containerID="c3e66009bb474ae5f4aa57b627aaa2ccb072b197139d61f21fa44f8e2c9effc1" Oct 31 00:08:23.444476 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:23.444438 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5655083c-8498-4488-8ec7-49e4de5d40fd" path="/var/lib/kubelet/pods/5655083c-8498-4488-8ec7-49e4de5d40fd/volumes" Oct 31 00:08:23.445305 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:23.445279 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-09efcb6d-699987c7fd-j2v69"] Oct 31 00:08:23.445542 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:23.445526 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-09efcb6d-699987c7fd-j2v69"] Oct 31 00:08:23.445677 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:23.445663 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-09efcb6d-c7bfbb5cf-fsvrl"] Oct 31 00:08:23.445794 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:23.445782 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-09efcb6d-c7bfbb5cf-fsvrl"] Oct 31 00:08:23.521916 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:23.521884 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16364156-gx8fb" Oct 31 00:08:23.533688 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:23.533658 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c8d1a25-ns4dx" Oct 31 00:08:23.587778 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:23.587706 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f369330e-5800-40d6-929c-bc9996aaa3d9-kube-api-access-bbzdz\" (UniqueName: \"kubernetes.io/projected/f369330e-5800-40d6-929c-bc9996aaa3d9-kube-api-access-bbzdz\") pod \"f369330e-5800-40d6-929c-bc9996aaa3d9\" (UID: \"f369330e-5800-40d6-929c-bc9996aaa3d9\") " Oct 31 00:08:23.587778 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:23.587749 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f369330e-5800-40d6-929c-bc9996aaa3d9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f369330e-5800-40d6-929c-bc9996aaa3d9-maas-api-openshift-ca\") pod \"f369330e-5800-40d6-929c-bc9996aaa3d9\" (UID: \"f369330e-5800-40d6-929c-bc9996aaa3d9\") " Oct 31 00:08:23.587778 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:23.587769 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f369330e-5800-40d6-929c-bc9996aaa3d9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f369330e-5800-40d6-929c-bc9996aaa3d9-maas-api-xks-ca\") pod \"f369330e-5800-40d6-929c-bc9996aaa3d9\" (UID: \"f369330e-5800-40d6-929c-bc9996aaa3d9\") " Oct 31 00:08:23.588066 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:23.587805 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cec08908-5b14-4d8c-a101-174d1496e41c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cec08908-5b14-4d8c-a101-174d1496e41c-maas-api-xks-ca\") pod \"cec08908-5b14-4d8c-a101-174d1496e41c\" (UID: \"cec08908-5b14-4d8c-a101-174d1496e41c\") " Oct 31 00:08:23.588066 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:23.587921 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cec08908-5b14-4d8c-a101-174d1496e41c-kube-api-access-4bdz6\" (UniqueName: \"kubernetes.io/projected/cec08908-5b14-4d8c-a101-174d1496e41c-kube-api-access-4bdz6\") pod \"cec08908-5b14-4d8c-a101-174d1496e41c\" (UID: \"cec08908-5b14-4d8c-a101-174d1496e41c\") " Oct 31 00:08:23.588187 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:23.588127 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cec08908-5b14-4d8c-a101-174d1496e41c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cec08908-5b14-4d8c-a101-174d1496e41c-maas-api-openshift-ca\") pod \"cec08908-5b14-4d8c-a101-174d1496e41c\" (UID: \"cec08908-5b14-4d8c-a101-174d1496e41c\") " Oct 31 00:08:23.588482 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:23.588439 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f369330e-5800-40d6-929c-bc9996aaa3d9-maas-api-openshift-ca" pod "f369330e-5800-40d6-929c-bc9996aaa3d9" (UID: "f369330e-5800-40d6-929c-bc9996aaa3d9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:08:23.588593 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:23.588498 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cec08908-5b14-4d8c-a101-174d1496e41c-maas-api-openshift-ca" pod "cec08908-5b14-4d8c-a101-174d1496e41c" (UID: "cec08908-5b14-4d8c-a101-174d1496e41c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:08:23.589000 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:23.588970 4927 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f369330e-5800-40d6-929c-bc9996aaa3d9-maas-api-openshift-ca\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:08:23.589116 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:23.589002 4927 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cec08908-5b14-4d8c-a101-174d1496e41c-maas-api-openshift-ca\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:08:23.590284 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:23.590258 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cec08908-5b14-4d8c-a101-174d1496e41c-maas-api-xks-ca" pod "cec08908-5b14-4d8c-a101-174d1496e41c" (UID: "cec08908-5b14-4d8c-a101-174d1496e41c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:08:23.590625 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:23.590487 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f369330e-5800-40d6-929c-bc9996aaa3d9-kube-api-access-bbzdz" pod "f369330e-5800-40d6-929c-bc9996aaa3d9" (UID: "f369330e-5800-40d6-929c-bc9996aaa3d9"). InnerVolumeSpecName "kube-api-access-bbzdz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:08:23.590625 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:23.590545 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f369330e-5800-40d6-929c-bc9996aaa3d9-maas-api-xks-ca" pod "f369330e-5800-40d6-929c-bc9996aaa3d9" (UID: "f369330e-5800-40d6-929c-bc9996aaa3d9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:08:23.590811 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:23.590785 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cec08908-5b14-4d8c-a101-174d1496e41c-kube-api-access-4bdz6" pod "cec08908-5b14-4d8c-a101-174d1496e41c" (UID: "cec08908-5b14-4d8c-a101-174d1496e41c"). InnerVolumeSpecName "kube-api-access-4bdz6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:08:23.690248 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:23.690208 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bbzdz\" (UniqueName: \"kubernetes.io/projected/f369330e-5800-40d6-929c-bc9996aaa3d9-kube-api-access-bbzdz\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:08:23.690248 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:23.690249 4927 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f369330e-5800-40d6-929c-bc9996aaa3d9-maas-api-xks-ca\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:08:23.690248 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:23.690266 4927 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cec08908-5b14-4d8c-a101-174d1496e41c-maas-api-xks-ca\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:08:23.690537 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:23.690280 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4bdz6\" (UniqueName: \"kubernetes.io/projected/cec08908-5b14-4d8c-a101-174d1496e41c-kube-api-access-4bdz6\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:08:23.710923 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:23.710841 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:08:23.826578 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:23.826542 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16364156-gx8fb" Oct 31 00:08:23.826578 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:23.826570 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16364156-gx8fb" event={"ID":"cec08908-5b14-4d8c-a101-174d1496e41c","Type":"ContainerDied","Data":"47df82ac06bc3ebe5de28dbccae74e38bbdc1991b0842531517fea8945de298a"} Oct 31 00:08:23.827032 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:23.826619 4927 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="47df82ac06bc3ebe5de28dbccae74e38bbdc1991b0842531517fea8945de298a" Oct 31 00:08:23.828297 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:23.828271 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c8d1a25-ns4dx" Oct 31 00:08:23.828481 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:23.828311 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c8d1a25-ns4dx" event={"ID":"f369330e-5800-40d6-929c-bc9996aaa3d9","Type":"ContainerDied","Data":"0e0d626956e8eefaa1076574b2549686604efb5314bd74d30821ccea36860bdf"} Oct 31 00:08:23.828481 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:23.828371 4927 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0e0d626956e8eefaa1076574b2549686604efb5314bd74d30821ccea36860bdf" Oct 31 00:08:23.978649 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:23.978609 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:08:24.242130 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:24.242051 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:08:24.254403 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:24.254282 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e0751b65-5lgb6"] Oct 31 00:08:24.262380 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:24.262351 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e0751b65-5lgb6"] Oct 31 00:08:24.626361 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:24.626242 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:08:25.089192 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:25.089161 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a735e34-9f5b-44bc-8155-cfec5319b710" path="/var/lib/kubelet/pods/0a735e34-9f5b-44bc-8155-cfec5319b710/volumes" Oct 31 00:08:25.089816 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:25.089712 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="37daf0f5-13de-43e2-a2a2-d82036836415" path="/var/lib/kubelet/pods/37daf0f5-13de-43e2-a2a2-d82036836415/volumes" Oct 31 00:08:25.090177 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:25.090161 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee43bf6b-c4d3-46f6-8808-8064e3412906" path="/var/lib/kubelet/pods/ee43bf6b-c4d3-46f6-8808-8064e3412906/volumes" Oct 31 00:08:26.862405 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:26.862343 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:08:26.866307 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:26.866280 4927 generic.go:356] "Generic (PLEG): container finished" podID="85533e83-d620-4bd3-8290-408d0c41164c" containerID="6459ff1422fee57db06f23be1010e336bf13a79c0d217e76dcff62c9d8f7296c" exitCode=0 Oct 31 00:08:26.866466 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:26.866364 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-7c8802f8-openshift-default-74499c95cc-v5qrl" event={"ID":"85533e83-d620-4bd3-8290-408d0c41164c","Type":"ContainerDied","Data":"6459ff1422fee57db06f23be1010e336bf13a79c0d217e76dcff62c9d8f7296c"} Oct 31 00:08:26.866844 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:26.866827 4927 scope.go:122] "RemoveContainer" containerID="6459ff1422fee57db06f23be1010e336bf13a79c0d217e76dcff62c9d8f7296c" Oct 31 00:08:26.868450 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:26.868416 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:08:26.868529 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:26.868494 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:08:26.868585 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:26.868535 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:08:27.262181 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:27.262146 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-7c8802f8-5db6686694-5hbns"] Oct 31 00:08:27.364362 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:27.364166 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-7c8802f8-84c649965f-8cltq"] Oct 31 00:08:27.641096 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:27.639170 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:08:27.838370 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:27.818346 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:08:28.030356 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:28.007769 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-7c8802f8-openshift-default-74499c95cc-v5qrl" event={"ID":"85533e83-d620-4bd3-8290-408d0c41164c","Type":"ContainerStarted","Data":"dbe5de481eae525d6ef94eb186fd51421928cd6be4e48593e55c0da8a239f7f1"} Oct 31 00:08:28.050742 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:28.050699 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-7c8802f8-openshift-default-74499c95cc-v5qrl" Oct 31 00:08:28.051794 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:28.051762 4927 patch_prober.go:29] interesting pod/e2e-ait-7c8802f8-openshift-default-74499c95cc-v5qrl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" start-of-body= Oct 31 00:08:28.051941 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:28.051830 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-7c8802f8-openshift-default-74499c95cc-v5qrl" podUID="85533e83-d620-4bd3-8290-408d0c41164c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" Oct 31 00:08:29.115262 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:29.111554 4927 patch_prober.go:29] interesting pod/e2e-ait-7c8802f8-openshift-default-74499c95cc-v5qrl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" start-of-body= Oct 31 00:08:29.115262 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:29.111637 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-7c8802f8-openshift-default-74499c95cc-v5qrl" podUID="85533e83-d620-4bd3-8290-408d0c41164c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" Oct 31 00:08:29.200720 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:29.198307 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:08:29.271411 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:29.265545 4927 scope.go:122] "RemoveContainer" containerID="a0e7e4d9c971111513db178ecf9667f819eb1c765085a9019ad15ca0ca63bea2" Oct 31 00:08:29.401276 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:29.401237 4927 scope.go:122] "RemoveContainer" containerID="b6e2f488c4668340ae0be6ecf66735b3596aa898b457aa664c61c51296804f77" Oct 31 00:08:29.562359 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:29.550532 4927 scope.go:122] "RemoveContainer" containerID="6d9d2bbef7fd0732a1f1728da6ff740e00762e63e0c5da1eec8a43e5c582d796" Oct 31 00:08:29.666794 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:29.666758 4927 scope.go:122] "RemoveContainer" containerID="8f5773172dce3bbbae288a3ee03ed7287c9f9cf1baa707197487a2fdcf542e2a" Oct 31 00:08:29.702561 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:29.702167 4927 scope.go:122] "RemoveContainer" containerID="7b5aa63c0b6630abe4b023572b18a1858f2ec88be22b58c0ea310439124e1f58" Oct 31 00:08:29.748612 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:29.748586 4927 scope.go:122] "RemoveContainer" containerID="936c08a97e8b38e293cec6cdba70a9e7d73f7b9df6bdcf07b2dcadfffd793d42" Oct 31 00:08:29.762480 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:29.762407 4927 scope.go:122] "RemoveContainer" containerID="22a2be1a89949d0a19071805fa1a872deafb7323b2455d4b3619589d1a246af5" Oct 31 00:08:29.777083 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:29.777060 4927 scope.go:122] "RemoveContainer" containerID="41ee9ee3779e3afe1a36fb2ec8d1a1df6e11a177c7fdde9fd8d2cc7012d11b8e" Oct 31 00:08:29.787657 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:29.787625 4927 scope.go:122] "RemoveContainer" containerID="ebc28877756ebf7357a2d3a230972d1a059bffa2cb7314187f02786d10711f75" Oct 31 00:08:29.798419 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:29.798375 4927 scope.go:122] "RemoveContainer" containerID="db9f426aad062a104cc9fe6aba026c4c50a645af3c2f97bfc927c88bd016e37a" Oct 31 00:08:30.050113 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:30.050076 4927 patch_prober.go:29] interesting pod/e2e-ait-7c8802f8-openshift-default-74499c95cc-v5qrl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" start-of-body= Oct 31 00:08:30.050295 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:30.050140 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-7c8802f8-openshift-default-74499c95cc-v5qrl" podUID="85533e83-d620-4bd3-8290-408d0c41164c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" Oct 31 00:08:30.084966 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:30.084935 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:08:30.261520 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:30.261441 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:08:31.005482 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:31.005443 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:08:31.050852 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:31.050815 4927 patch_prober.go:29] interesting pod/e2e-ait-7c8802f8-openshift-default-74499c95cc-v5qrl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" start-of-body= Oct 31 00:08:31.051131 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:31.051101 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-7c8802f8-openshift-default-74499c95cc-v5qrl" podUID="85533e83-d620-4bd3-8290-408d0c41164c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" Oct 31 00:08:31.206682 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:31.206646 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-7c8802f8-openshift-default-74499c95cc-v5qrl" Oct 31 00:08:31.284829 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:31.284736 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:08:31.447073 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:31.447038 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:08:31.937379 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:31.926583 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:08:32.062095 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:32.062058 4927 patch_prober.go:29] interesting pod/e2e-ait-7c8802f8-openshift-default-74499c95cc-v5qrl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" start-of-body= Oct 31 00:08:32.062302 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:32.062116 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-7c8802f8-openshift-default-74499c95cc-v5qrl" podUID="85533e83-d620-4bd3-8290-408d0c41164c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" Oct 31 00:08:32.063425 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:32.063402 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c9c66457-4rw24"] Oct 31 00:08:32.083115 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:32.083086 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:08:32.143804 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:32.143766 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c9c66457-4rw24"] Oct 31 00:08:32.263543 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:32.263435 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-7c8802f8-5db6686694-5hbns" podUID="43c9272e-aafb-4785-954c-4fd628242852" containerName="payload-processing" containerID="cri-o://9e8851d1a744adead476d34bc873e3c6cc40036956b9c742aef39d331efa1cef" gracePeriod=25 Oct 31 00:08:32.365437 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:32.365395 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-7c8802f8-84c649965f-8cltq" podUID="819a7f5a-86da-4db4-8674-e260009f2a61" containerName="payload-pre-processing" containerID="cri-o://c8f7486d43ca36e4e150ee3121fadd1ef4ff6c719a4b6b0f1bfa4d5d97f27945" gracePeriod=25 Oct 31 00:08:32.446115 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:32.445120 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:08:32.453020 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:32.452982 4927 generic.go:356] "Generic (PLEG): container finished" podID="43c9272e-aafb-4785-954c-4fd628242852" containerID="9e8851d1a744adead476d34bc873e3c6cc40036956b9c742aef39d331efa1cef" exitCode=0 Oct 31 00:08:32.453221 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:32.453030 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-7c8802f8-5db6686694-5hbns" event={"ID":"43c9272e-aafb-4785-954c-4fd628242852","Type":"ContainerDied","Data":"9e8851d1a744adead476d34bc873e3c6cc40036956b9c742aef39d331efa1cef"} Oct 31 00:08:32.748583 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:32.748548 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-7c8802f8-5db6686694-5hbns" Oct 31 00:08:32.769378 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:32.761607 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/43c9272e-aafb-4785-954c-4fd628242852-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43c9272e-aafb-4785-954c-4fd628242852-plugins-config-volume\") pod \"43c9272e-aafb-4785-954c-4fd628242852\" (UID: \"43c9272e-aafb-4785-954c-4fd628242852\") " Oct 31 00:08:32.769378 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:32.763503 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43c9272e-aafb-4785-954c-4fd628242852-plugins-config-volume" pod "43c9272e-aafb-4785-954c-4fd628242852" (UID: "43c9272e-aafb-4785-954c-4fd628242852"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:08:32.769378 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:32.764185 4927 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43c9272e-aafb-4785-954c-4fd628242852-plugins-config-volume\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:08:32.874789 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:32.865714 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43c9272e-aafb-4785-954c-4fd628242852-kube-api-access-bt5nv\" (UniqueName: \"kubernetes.io/projected/43c9272e-aafb-4785-954c-4fd628242852-kube-api-access-bt5nv\") pod \"43c9272e-aafb-4785-954c-4fd628242852\" (UID: \"43c9272e-aafb-4785-954c-4fd628242852\") " Oct 31 00:08:32.879417 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:32.877913 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43c9272e-aafb-4785-954c-4fd628242852-kube-api-access-bt5nv" pod "43c9272e-aafb-4785-954c-4fd628242852" (UID: "43c9272e-aafb-4785-954c-4fd628242852"). InnerVolumeSpecName "kube-api-access-bt5nv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:08:32.881756 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:32.881723 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-7c8802f8-84c649965f-8cltq" Oct 31 00:08:32.976744 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:32.976588 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bt5nv\" (UniqueName: \"kubernetes.io/projected/43c9272e-aafb-4785-954c-4fd628242852-kube-api-access-bt5nv\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:08:33.051036 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:33.050903 4927 patch_prober.go:29] interesting pod/e2e-ait-7c8802f8-openshift-default-74499c95cc-v5qrl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" start-of-body= Oct 31 00:08:33.051036 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:33.050967 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-7c8802f8-openshift-default-74499c95cc-v5qrl" podUID="85533e83-d620-4bd3-8290-408d0c41164c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" Oct 31 00:08:33.078040 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:33.078004 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/819a7f5a-86da-4db4-8674-e260009f2a61-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/819a7f5a-86da-4db4-8674-e260009f2a61-plugins-config-volume\") pod \"819a7f5a-86da-4db4-8674-e260009f2a61\" (UID: \"819a7f5a-86da-4db4-8674-e260009f2a61\") " Oct 31 00:08:33.078388 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:33.078366 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/819a7f5a-86da-4db4-8674-e260009f2a61-kube-api-access-z6mxn\" (UniqueName: \"kubernetes.io/projected/819a7f5a-86da-4db4-8674-e260009f2a61-kube-api-access-z6mxn\") pod \"819a7f5a-86da-4db4-8674-e260009f2a61\" (UID: \"819a7f5a-86da-4db4-8674-e260009f2a61\") " Oct 31 00:08:33.078535 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:33.078505 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/819a7f5a-86da-4db4-8674-e260009f2a61-plugins-config-volume" pod "819a7f5a-86da-4db4-8674-e260009f2a61" (UID: "819a7f5a-86da-4db4-8674-e260009f2a61"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:08:33.079897 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:33.079873 4927 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/819a7f5a-86da-4db4-8674-e260009f2a61-plugins-config-volume\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:08:33.083348 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:33.083305 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/819a7f5a-86da-4db4-8674-e260009f2a61-kube-api-access-z6mxn" pod "819a7f5a-86da-4db4-8674-e260009f2a61" (UID: "819a7f5a-86da-4db4-8674-e260009f2a61"). InnerVolumeSpecName "kube-api-access-z6mxn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:08:33.097128 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:33.097092 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="351c1f92-30d9-4b47-ad7e-70f102f0a280" path="/var/lib/kubelet/pods/351c1f92-30d9-4b47-ad7e-70f102f0a280/volumes" Oct 31 00:08:33.182131 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:33.182089 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z6mxn\" (UniqueName: \"kubernetes.io/projected/819a7f5a-86da-4db4-8674-e260009f2a61-kube-api-access-z6mxn\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:08:33.470658 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:33.470444 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-7c8802f8-5db6686694-5hbns" event={"ID":"43c9272e-aafb-4785-954c-4fd628242852","Type":"ContainerDied","Data":"fce115129d290f071c234b9eb9f4ba43fecbbad953f28f1257f370473ad6d53a"} Oct 31 00:08:33.470658 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:33.470526 4927 scope.go:122] "RemoveContainer" containerID="9e8851d1a744adead476d34bc873e3c6cc40036956b9c742aef39d331efa1cef" Oct 31 00:08:33.472731 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:33.472431 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-7c8802f8-5db6686694-5hbns" Oct 31 00:08:33.473798 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:33.473439 4927 generic.go:356] "Generic (PLEG): container finished" podID="819a7f5a-86da-4db4-8674-e260009f2a61" containerID="c8f7486d43ca36e4e150ee3121fadd1ef4ff6c719a4b6b0f1bfa4d5d97f27945" exitCode=0 Oct 31 00:08:33.473798 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:33.473687 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-7c8802f8-84c649965f-8cltq" event={"ID":"819a7f5a-86da-4db4-8674-e260009f2a61","Type":"ContainerDied","Data":"c8f7486d43ca36e4e150ee3121fadd1ef4ff6c719a4b6b0f1bfa4d5d97f27945"} Oct 31 00:08:33.473798 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:33.473729 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-7c8802f8-84c649965f-8cltq" event={"ID":"819a7f5a-86da-4db4-8674-e260009f2a61","Type":"ContainerDied","Data":"e6020d65c5dd8246eb88c6caed7cccd19191fbc25c31f888b9a66b661bd63d8b"} Oct 31 00:08:33.473798 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:33.473796 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-7c8802f8-84c649965f-8cltq" Oct 31 00:08:33.487927 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:33.487900 4927 scope.go:122] "RemoveContainer" containerID="c8f7486d43ca36e4e150ee3121fadd1ef4ff6c719a4b6b0f1bfa4d5d97f27945" Oct 31 00:08:33.504509 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:33.504484 4927 scope.go:122] "RemoveContainer" containerID="c8f7486d43ca36e4e150ee3121fadd1ef4ff6c719a4b6b0f1bfa4d5d97f27945" Oct 31 00:08:33.504861 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:08:33.504807 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c8f7486d43ca36e4e150ee3121fadd1ef4ff6c719a4b6b0f1bfa4d5d97f27945\": container with ID starting with c8f7486d43ca36e4e150ee3121fadd1ef4ff6c719a4b6b0f1bfa4d5d97f27945 not found: ID does not exist" containerID="c8f7486d43ca36e4e150ee3121fadd1ef4ff6c719a4b6b0f1bfa4d5d97f27945" Oct 31 00:08:33.504952 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:33.504865 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8f7486d43ca36e4e150ee3121fadd1ef4ff6c719a4b6b0f1bfa4d5d97f27945"} err="failed to get container status \"c8f7486d43ca36e4e150ee3121fadd1ef4ff6c719a4b6b0f1bfa4d5d97f27945\": rpc error: code = NotFound desc = could not find container \"c8f7486d43ca36e4e150ee3121fadd1ef4ff6c719a4b6b0f1bfa4d5d97f27945\": container with ID starting with c8f7486d43ca36e4e150ee3121fadd1ef4ff6c719a4b6b0f1bfa4d5d97f27945 not found: ID does not exist" Oct 31 00:08:33.648650 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:33.648553 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-7c8802f8-5db6686694-5hbns"] Oct 31 00:08:33.656074 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:33.656045 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-7c8802f8-5db6686694-5hbns"] Oct 31 00:08:33.771138 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:33.771095 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-7c8802f8-84c649965f-8cltq"] Oct 31 00:08:33.783136 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:33.783097 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-7c8802f8-84c649965f-8cltq"] Oct 31 00:08:34.002482 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:34.002446 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:08:34.098381 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:34.090824 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2609fef-pwzkp"] Oct 31 00:08:34.108357 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:34.107460 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2609fef-pwzkp"] Oct 31 00:08:34.119354 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:34.117707 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-7c8802f8-openshift-default-74499c95cc-v5qrl" Oct 31 00:08:34.216352 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:34.216146 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-7c8802f8-openshift-default-74499c95cc-v5qrl" Oct 31 00:08:35.089106 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:35.089072 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43c9272e-aafb-4785-954c-4fd628242852" path="/var/lib/kubelet/pods/43c9272e-aafb-4785-954c-4fd628242852/volumes" Oct 31 00:08:35.089577 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:35.089560 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="819a7f5a-86da-4db4-8674-e260009f2a61" path="/var/lib/kubelet/pods/819a7f5a-86da-4db4-8674-e260009f2a61/volumes" Oct 31 00:08:35.089990 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:35.089977 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="adaebfaf-92b7-459d-ae77-0d78fb8a46f5" path="/var/lib/kubelet/pods/adaebfaf-92b7-459d-ae77-0d78fb8a46f5/volumes" Oct 31 00:08:37.102291 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:37.102255 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:08:37.635406 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:37.635369 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:08:38.560584 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:38.560546 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:08:38.762998 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:38.762187 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:08:39.047640 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:39.047600 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:08:39.622860 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:39.622809 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-71b76a-openshift-default-77db967c97-lh4ld"] Oct 31 00:08:39.626109 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:39.626078 4927 cpu_manager.go:414] "containerMap: removing container" podUID="ee43bf6b-c4d3-46f6-8808-8064e3412906" containerName="payload-processing" Oct 31 00:08:39.626326 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:39.626306 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee43bf6b-c4d3-46f6-8808-8064e3412906" containerName="payload-processing" Oct 31 00:08:39.626567 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:39.626552 4927 cpu_manager.go:414] "containerMap: removing container" podUID="819a7f5a-86da-4db4-8674-e260009f2a61" containerName="payload-pre-processing" Oct 31 00:08:39.626666 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:39.626654 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="819a7f5a-86da-4db4-8674-e260009f2a61" containerName="payload-pre-processing" Oct 31 00:08:39.626748 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:39.626739 4927 cpu_manager.go:414] "containerMap: removing container" podUID="f369330e-5800-40d6-929c-bc9996aaa3d9" containerName="revoke-keys" Oct 31 00:08:39.626822 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:39.626812 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f369330e-5800-40d6-929c-bc9996aaa3d9" containerName="revoke-keys" Oct 31 00:08:39.626910 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:39.626900 4927 cpu_manager.go:414] "containerMap: removing container" podUID="7e5c0632-30c2-4d21-a27b-eb252afe5747" containerName="revoke-keys" Oct 31 00:08:39.626982 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:39.626973 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e5c0632-30c2-4d21-a27b-eb252afe5747" containerName="revoke-keys" Oct 31 00:08:39.627096 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:39.627086 4927 cpu_manager.go:414] "containerMap: removing container" podUID="43c9272e-aafb-4785-954c-4fd628242852" containerName="payload-processing" Oct 31 00:08:39.627167 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:39.627158 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43c9272e-aafb-4785-954c-4fd628242852" containerName="payload-processing" Oct 31 00:08:39.627238 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:39.627229 4927 cpu_manager.go:414] "containerMap: removing container" podUID="cec08908-5b14-4d8c-a101-174d1496e41c" containerName="revoke-keys" Oct 31 00:08:39.627315 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:39.627305 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cec08908-5b14-4d8c-a101-174d1496e41c" containerName="revoke-keys" Oct 31 00:08:39.627465 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:39.627449 4927 cpu_manager.go:414] "containerMap: removing container" podUID="37daf0f5-13de-43e2-a2a2-d82036836415" containerName="payload-pre-processing" Oct 31 00:08:39.627552 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:39.627536 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37daf0f5-13de-43e2-a2a2-d82036836415" containerName="payload-pre-processing" Oct 31 00:08:39.627913 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:39.627892 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee43bf6b-c4d3-46f6-8808-8064e3412906" containerName="payload-processing" Oct 31 00:08:39.628048 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:39.628037 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="819a7f5a-86da-4db4-8674-e260009f2a61" containerName="payload-pre-processing" Oct 31 00:08:39.628124 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:39.628114 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="f369330e-5800-40d6-929c-bc9996aaa3d9" containerName="revoke-keys" Oct 31 00:08:39.628204 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:39.628195 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e5c0632-30c2-4d21-a27b-eb252afe5747" containerName="revoke-keys" Oct 31 00:08:39.628290 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:39.628281 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="43c9272e-aafb-4785-954c-4fd628242852" containerName="payload-processing" Oct 31 00:08:39.628412 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:39.628376 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="cec08908-5b14-4d8c-a101-174d1496e41c" containerName="revoke-keys" Oct 31 00:08:39.628535 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:39.628520 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="37daf0f5-13de-43e2-a2a2-d82036836415" containerName="payload-pre-processing" Oct 31 00:08:39.636205 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:39.636179 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-71b76a-openshift-default-77db967c97-lh4ld" Oct 31 00:08:39.712403 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:39.707769 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-71b76a-openshift-default-dockercfg-8lzzp\"" Oct 31 00:08:39.777354 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:39.758494 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fl8ph\" (UniqueName: \"kubernetes.io/projected/c51c586a-f2d8-48f9-8813-3b6e4f360e5e-kube-api-access-fl8ph\") pod \"e2e-api-a-71b76a-openshift-default-77db967c97-lh4ld\" (UID: \"c51c586a-f2d8-48f9-8813-3b6e4f360e5e\") " pod="openshift-ingress/e2e-api-a-71b76a-openshift-default-77db967c97-lh4ld" Oct 31 00:08:40.194512 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:39.801080 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:08:40.194922 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:39.906690 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-71b76a-openshift-default-77db967c97-lh4ld"] Oct 31 00:08:40.195628 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:40.195598 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c51c586a-f2d8-48f9-8813-3b6e4f360e5e-istio-token\") pod \"e2e-api-a-71b76a-openshift-default-77db967c97-lh4ld\" (UID: \"c51c586a-f2d8-48f9-8813-3b6e4f360e5e\") " pod="openshift-ingress/e2e-api-a-71b76a-openshift-default-77db967c97-lh4ld" Oct 31 00:08:40.195918 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:40.195897 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c51c586a-f2d8-48f9-8813-3b6e4f360e5e-istio-envoy\") pod \"e2e-api-a-71b76a-openshift-default-77db967c97-lh4ld\" (UID: \"c51c586a-f2d8-48f9-8813-3b6e4f360e5e\") " pod="openshift-ingress/e2e-api-a-71b76a-openshift-default-77db967c97-lh4ld" Oct 31 00:08:40.222712 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:40.206030 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c51c586a-f2d8-48f9-8813-3b6e4f360e5e-istio-podinfo\") pod \"e2e-api-a-71b76a-openshift-default-77db967c97-lh4ld\" (UID: \"c51c586a-f2d8-48f9-8813-3b6e4f360e5e\") " pod="openshift-ingress/e2e-api-a-71b76a-openshift-default-77db967c97-lh4ld" Oct 31 00:08:40.222712 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:40.206372 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c51c586a-f2d8-48f9-8813-3b6e4f360e5e-istio-data\") pod \"e2e-api-a-71b76a-openshift-default-77db967c97-lh4ld\" (UID: \"c51c586a-f2d8-48f9-8813-3b6e4f360e5e\") " pod="openshift-ingress/e2e-api-a-71b76a-openshift-default-77db967c97-lh4ld" Oct 31 00:08:40.222712 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:40.206692 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c51c586a-f2d8-48f9-8813-3b6e4f360e5e-credential-socket\") pod \"e2e-api-a-71b76a-openshift-default-77db967c97-lh4ld\" (UID: \"c51c586a-f2d8-48f9-8813-3b6e4f360e5e\") " pod="openshift-ingress/e2e-api-a-71b76a-openshift-default-77db967c97-lh4ld" Oct 31 00:08:40.222712 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:40.206858 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c51c586a-f2d8-48f9-8813-3b6e4f360e5e-workload-certs\") pod \"e2e-api-a-71b76a-openshift-default-77db967c97-lh4ld\" (UID: \"c51c586a-f2d8-48f9-8813-3b6e4f360e5e\") " pod="openshift-ingress/e2e-api-a-71b76a-openshift-default-77db967c97-lh4ld" Oct 31 00:08:40.244881 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:40.207488 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c51c586a-f2d8-48f9-8813-3b6e4f360e5e-istiod-ca-cert\") pod \"e2e-api-a-71b76a-openshift-default-77db967c97-lh4ld\" (UID: \"c51c586a-f2d8-48f9-8813-3b6e4f360e5e\") " pod="openshift-ingress/e2e-api-a-71b76a-openshift-default-77db967c97-lh4ld" Oct 31 00:08:40.250654 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:40.250621 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c51c586a-f2d8-48f9-8813-3b6e4f360e5e-workload-socket\") pod \"e2e-api-a-71b76a-openshift-default-77db967c97-lh4ld\" (UID: \"c51c586a-f2d8-48f9-8813-3b6e4f360e5e\") " pod="openshift-ingress/e2e-api-a-71b76a-openshift-default-77db967c97-lh4ld" Oct 31 00:08:40.365353 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:40.361625 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:08:40.365353 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:40.362554 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c51c586a-f2d8-48f9-8813-3b6e4f360e5e-istio-data\") pod \"e2e-api-a-71b76a-openshift-default-77db967c97-lh4ld\" (UID: \"c51c586a-f2d8-48f9-8813-3b6e4f360e5e\") " pod="openshift-ingress/e2e-api-a-71b76a-openshift-default-77db967c97-lh4ld" Oct 31 00:08:40.365353 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:40.362603 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c51c586a-f2d8-48f9-8813-3b6e4f360e5e-credential-socket\") pod \"e2e-api-a-71b76a-openshift-default-77db967c97-lh4ld\" (UID: \"c51c586a-f2d8-48f9-8813-3b6e4f360e5e\") " pod="openshift-ingress/e2e-api-a-71b76a-openshift-default-77db967c97-lh4ld" Oct 31 00:08:40.365353 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:40.362652 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c51c586a-f2d8-48f9-8813-3b6e4f360e5e-workload-certs\") pod \"e2e-api-a-71b76a-openshift-default-77db967c97-lh4ld\" (UID: \"c51c586a-f2d8-48f9-8813-3b6e4f360e5e\") " pod="openshift-ingress/e2e-api-a-71b76a-openshift-default-77db967c97-lh4ld" Oct 31 00:08:40.365353 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:40.362766 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c51c586a-f2d8-48f9-8813-3b6e4f360e5e-istiod-ca-cert\") pod \"e2e-api-a-71b76a-openshift-default-77db967c97-lh4ld\" (UID: \"c51c586a-f2d8-48f9-8813-3b6e4f360e5e\") " pod="openshift-ingress/e2e-api-a-71b76a-openshift-default-77db967c97-lh4ld" Oct 31 00:08:40.365353 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:40.362839 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c51c586a-f2d8-48f9-8813-3b6e4f360e5e-workload-socket\") pod \"e2e-api-a-71b76a-openshift-default-77db967c97-lh4ld\" (UID: \"c51c586a-f2d8-48f9-8813-3b6e4f360e5e\") " pod="openshift-ingress/e2e-api-a-71b76a-openshift-default-77db967c97-lh4ld" Oct 31 00:08:40.365353 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:40.362916 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fl8ph\" (UniqueName: \"kubernetes.io/projected/c51c586a-f2d8-48f9-8813-3b6e4f360e5e-kube-api-access-fl8ph\") pod \"e2e-api-a-71b76a-openshift-default-77db967c97-lh4ld\" (UID: \"c51c586a-f2d8-48f9-8813-3b6e4f360e5e\") " pod="openshift-ingress/e2e-api-a-71b76a-openshift-default-77db967c97-lh4ld" Oct 31 00:08:40.365353 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:40.363010 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c51c586a-f2d8-48f9-8813-3b6e4f360e5e-istio-token\") pod \"e2e-api-a-71b76a-openshift-default-77db967c97-lh4ld\" (UID: \"c51c586a-f2d8-48f9-8813-3b6e4f360e5e\") " pod="openshift-ingress/e2e-api-a-71b76a-openshift-default-77db967c97-lh4ld" Oct 31 00:08:40.365353 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:40.363060 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c51c586a-f2d8-48f9-8813-3b6e4f360e5e-istio-envoy\") pod \"e2e-api-a-71b76a-openshift-default-77db967c97-lh4ld\" (UID: \"c51c586a-f2d8-48f9-8813-3b6e4f360e5e\") " pod="openshift-ingress/e2e-api-a-71b76a-openshift-default-77db967c97-lh4ld" Oct 31 00:08:40.365353 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:40.363097 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c51c586a-f2d8-48f9-8813-3b6e4f360e5e-istio-podinfo\") pod \"e2e-api-a-71b76a-openshift-default-77db967c97-lh4ld\" (UID: \"c51c586a-f2d8-48f9-8813-3b6e4f360e5e\") " pod="openshift-ingress/e2e-api-a-71b76a-openshift-default-77db967c97-lh4ld" Oct 31 00:08:40.365353 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:40.363900 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c51c586a-f2d8-48f9-8813-3b6e4f360e5e-istio-data\") pod \"e2e-api-a-71b76a-openshift-default-77db967c97-lh4ld\" (UID: \"c51c586a-f2d8-48f9-8813-3b6e4f360e5e\") " pod="openshift-ingress/e2e-api-a-71b76a-openshift-default-77db967c97-lh4ld" Oct 31 00:08:40.365353 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:40.364096 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c51c586a-f2d8-48f9-8813-3b6e4f360e5e-credential-socket\") pod \"e2e-api-a-71b76a-openshift-default-77db967c97-lh4ld\" (UID: \"c51c586a-f2d8-48f9-8813-3b6e4f360e5e\") " pod="openshift-ingress/e2e-api-a-71b76a-openshift-default-77db967c97-lh4ld" Oct 31 00:08:40.365353 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:40.364602 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c51c586a-f2d8-48f9-8813-3b6e4f360e5e-workload-certs\") pod \"e2e-api-a-71b76a-openshift-default-77db967c97-lh4ld\" (UID: \"c51c586a-f2d8-48f9-8813-3b6e4f360e5e\") " pod="openshift-ingress/e2e-api-a-71b76a-openshift-default-77db967c97-lh4ld" Oct 31 00:08:40.368320 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:40.367258 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c51c586a-f2d8-48f9-8813-3b6e4f360e5e-istiod-ca-cert\") pod \"e2e-api-a-71b76a-openshift-default-77db967c97-lh4ld\" (UID: \"c51c586a-f2d8-48f9-8813-3b6e4f360e5e\") " pod="openshift-ingress/e2e-api-a-71b76a-openshift-default-77db967c97-lh4ld" Oct 31 00:08:40.368320 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:40.368135 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c51c586a-f2d8-48f9-8813-3b6e4f360e5e-workload-socket\") pod \"e2e-api-a-71b76a-openshift-default-77db967c97-lh4ld\" (UID: \"c51c586a-f2d8-48f9-8813-3b6e4f360e5e\") " pod="openshift-ingress/e2e-api-a-71b76a-openshift-default-77db967c97-lh4ld" Oct 31 00:08:40.402086 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:40.401979 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c51c586a-f2d8-48f9-8813-3b6e4f360e5e-istio-envoy\") pod \"e2e-api-a-71b76a-openshift-default-77db967c97-lh4ld\" (UID: \"c51c586a-f2d8-48f9-8813-3b6e4f360e5e\") " pod="openshift-ingress/e2e-api-a-71b76a-openshift-default-77db967c97-lh4ld" Oct 31 00:08:40.436365 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:40.435641 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c51c586a-f2d8-48f9-8813-3b6e4f360e5e-istio-podinfo\") pod \"e2e-api-a-71b76a-openshift-default-77db967c97-lh4ld\" (UID: \"c51c586a-f2d8-48f9-8813-3b6e4f360e5e\") " pod="openshift-ingress/e2e-api-a-71b76a-openshift-default-77db967c97-lh4ld" Oct 31 00:08:40.454240 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:40.443892 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c51c586a-f2d8-48f9-8813-3b6e4f360e5e-istio-token\") pod \"e2e-api-a-71b76a-openshift-default-77db967c97-lh4ld\" (UID: \"c51c586a-f2d8-48f9-8813-3b6e4f360e5e\") " pod="openshift-ingress/e2e-api-a-71b76a-openshift-default-77db967c97-lh4ld" Oct 31 00:08:40.454240 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:40.446960 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fl8ph\" (UniqueName: \"kubernetes.io/projected/c51c586a-f2d8-48f9-8813-3b6e4f360e5e-kube-api-access-fl8ph\") pod \"e2e-api-a-71b76a-openshift-default-77db967c97-lh4ld\" (UID: \"c51c586a-f2d8-48f9-8813-3b6e4f360e5e\") " pod="openshift-ingress/e2e-api-a-71b76a-openshift-default-77db967c97-lh4ld" Oct 31 00:08:40.595553 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:40.595523 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-71b76a-openshift-default-77db967c97-lh4ld" Oct 31 00:08:41.047461 ip-10-0-141-180 kubenswrapper[4927]: W1031 00:08:41.047423 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc51c586a_f2d8_48f9_8813_3b6e4f360e5e.slice/crio-218f4d1e80a3de938f62ea4a5c245e7eeb4c4eecd358cfaab34d7edc2ff76ebf WatchSource:0}: Error finding container 218f4d1e80a3de938f62ea4a5c245e7eeb4c4eecd358cfaab34d7edc2ff76ebf: Status 404 returned error can't find the container with id 218f4d1e80a3de938f62ea4a5c245e7eeb4c4eecd358cfaab34d7edc2ff76ebf Oct 31 00:08:41.048059 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:41.048031 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-71b76a-openshift-default-77db967c97-lh4ld"] Oct 31 00:08:41.051001 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:41.050956 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:08:41.051115 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:41.051045 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:08:41.051115 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:41.051088 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:08:41.585812 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:41.585773 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-71b76a-openshift-default-77db967c97-lh4ld" event={"ID":"c51c586a-f2d8-48f9-8813-3b6e4f360e5e","Type":"ContainerStarted","Data":"aabe1a791b1aa93cf5e47255b266a37d6cdc8ec03231adcc7bae54204476a0b3"} Oct 31 00:08:41.585995 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:41.585840 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-71b76a-openshift-default-77db967c97-lh4ld" event={"ID":"c51c586a-f2d8-48f9-8813-3b6e4f360e5e","Type":"ContainerStarted","Data":"218f4d1e80a3de938f62ea4a5c245e7eeb4c4eecd358cfaab34d7edc2ff76ebf"} Oct 31 00:08:41.596490 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:41.596457 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-71b76a-openshift-default-77db967c97-lh4ld" Oct 31 00:08:41.764634 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:41.764579 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-71b76a-openshift-default-77db967c97-lh4ld" podStartSLOduration=2.7645629290000002 podStartE2EDuration="2.764562929s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:08:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:08:41.759270707 +0000 UTC m=+1634.795601551" watchObservedRunningTime="2026-10-31 00:08:41.764562929 +0000 UTC m=+1634.800893751" Oct 31 00:08:42.601266 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:42.601233 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-71b76a-openshift-default-77db967c97-lh4ld" Oct 31 00:08:43.234626 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:43.234585 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:08:43.603858 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:43.603782 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-71b76a-openshift-default-77db967c97-lh4ld" Oct 31 00:08:43.604706 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:43.604683 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-71b76a-openshift-default-77db967c97-lh4ld" Oct 31 00:08:44.587143 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:44.587107 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:08:44.803076 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:44.803036 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:08:45.089274 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:45.089237 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:08:45.105370 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:45.105303 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ea831844-gf8cw"] Oct 31 00:08:45.155874 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:45.155837 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ea831844-gf8cw"] Oct 31 00:08:47.092372 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:47.092285 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="55ce234b-b2e4-48ca-ae22-57598d4bd8db" path="/var/lib/kubelet/pods/55ce234b-b2e4-48ca-ae22-57598d4bd8db/volumes" Oct 31 00:08:47.473493 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:47.473454 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-xgpsz"] Oct 31 00:08:47.475239 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:47.475202 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-xgpsz" podUID="e6b778b5-4861-4a3f-85cb-5524098583f1" containerName="istio-proxy" containerID="cri-o://dae44a8554e1269e47872205987bfe082bbc0e2df1fa6679ab5137a80ab6ad25" gracePeriod=30 Oct 31 00:08:47.533477 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:47.533389 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-94tkp"] Oct 31 00:08:47.533966 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:47.533929 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-94tkp" podUID="9f348596-394c-4bfb-ba79-4788da8fc6de" containerName="istio-proxy" containerID="cri-o://9ea8e8a0622bd064854d976900a471b591cf0ad747977a10109b496f5289b8df" gracePeriod=30 Oct 31 00:08:47.573805 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:47.572837 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-klh6t"] Oct 31 00:08:47.573805 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:47.573104 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-klh6t" podUID="b7eb6748-2997-4527-91b3-86900ea70eed" containerName="istio-proxy" containerID="cri-o://c4717a5f3b503ab1937f5d3e01e2b12f6a6927be6491842a06b68ade45ca38da" gracePeriod=30 Oct 31 00:08:47.630677 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:47.630629 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-nr4qz"] Oct 31 00:08:47.640464 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:47.640420 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-nr4qz" podUID="333d9301-4a9b-4867-b3d9-88d3766d1acb" containerName="istio-proxy" containerID="cri-o://2cbee7072e17e752090d8a76ddbb931486f62ce02a86fb77ec088ce1899fa412" gracePeriod=30 Oct 31 00:08:47.819627 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:47.819491 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:08:47.999225 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:47.999178 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:08:48.008008 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:48.007961 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-khcqd"] Oct 31 00:08:48.009091 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:48.009046 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-khcqd" podUID="2b317f84-83ac-49aa-921e-0242889f2b4f" containerName="istio-proxy" containerID="cri-o://f2edac246d0fdf2378197ddfb3fb9eab46f7ea47415499c914edaee90412f22b" gracePeriod=30 Oct 31 00:08:48.038781 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:48.038739 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-2qnns"] Oct 31 00:08:48.039097 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:48.039076 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-7gm84"] Oct 31 00:08:48.039539 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:48.039508 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-7gm84" podUID="45977fb0-7f5d-4f55-9530-112d60f269ab" containerName="istio-proxy" containerID="cri-o://646c05c17da8a4f530f45bf9bb4e8348afb3fbe87e9040d1ee2cea4e92dccb62" gracePeriod=30 Oct 31 00:08:48.040678 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:48.040648 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-2qnns" podUID="b7194267-cfef-4252-ae16-a194eccd2a1f" containerName="istio-proxy" containerID="cri-o://6616bfd1ea4a480ce02466ce388380eb31ea8e8030af93e453c8d0c7a09224b2" gracePeriod=30 Oct 31 00:08:48.052601 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:48.052568 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-vb9r5"] Oct 31 00:08:48.052814 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:48.052792 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-vb9r5" podUID="334f41bf-6d81-474d-bddc-d38c3492d586" containerName="istio-proxy" containerID="cri-o://b2188dcfc96b95e4deeaf72da6a224c424e584af088e805d70914d27730aae5e" gracePeriod=30 Oct 31 00:08:48.100251 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:48.100159 4927 patch_prober.go:29] interesting pod/e2e-worker-w6-35a187-openshift-default-5b54c65f75-xgpsz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 00:08:48.100251 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:48.100239 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-xgpsz" podUID="e6b778b5-4861-4a3f-85cb-5524098583f1" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:08:48.150110 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:48.150072 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-71b76a-64f77d669f-hm2nn"] Oct 31 00:08:48.155435 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:48.155410 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-71b76a-64f77d669f-hm2nn" Oct 31 00:08:48.209061 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:48.209030 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-71b76a\"" Oct 31 00:08:48.209061 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:48.209069 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-71b76a-dockercfg-9skgj\"" Oct 31 00:08:48.220295 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:48.220259 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-71b76a-64f77d669f-hm2nn"] Oct 31 00:08:48.252319 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:48.252283 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9612233d-027a-49e1-bb03-c9dc4f8ecb6b-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-71b76a-64f77d669f-hm2nn\" (UID: \"9612233d-027a-49e1-bb03-c9dc4f8ecb6b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-71b76a-64f77d669f-hm2nn" Oct 31 00:08:48.252535 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:48.252436 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48fdn\" (UniqueName: \"kubernetes.io/projected/9612233d-027a-49e1-bb03-c9dc4f8ecb6b-kube-api-access-48fdn\") pod \"payload-pre-processing-e2e-api-a-71b76a-64f77d669f-hm2nn\" (UID: \"9612233d-027a-49e1-bb03-c9dc4f8ecb6b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-71b76a-64f77d669f-hm2nn" Oct 31 00:08:48.260834 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:48.260805 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:08:48.356719 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:48.356624 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48fdn\" (UniqueName: \"kubernetes.io/projected/9612233d-027a-49e1-bb03-c9dc4f8ecb6b-kube-api-access-48fdn\") pod \"payload-pre-processing-e2e-api-a-71b76a-64f77d669f-hm2nn\" (UID: \"9612233d-027a-49e1-bb03-c9dc4f8ecb6b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-71b76a-64f77d669f-hm2nn" Oct 31 00:08:48.357020 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:48.356997 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9612233d-027a-49e1-bb03-c9dc4f8ecb6b-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-71b76a-64f77d669f-hm2nn\" (UID: \"9612233d-027a-49e1-bb03-c9dc4f8ecb6b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-71b76a-64f77d669f-hm2nn" Oct 31 00:08:48.358271 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:48.358098 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9612233d-027a-49e1-bb03-c9dc4f8ecb6b-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-71b76a-64f77d669f-hm2nn\" (UID: \"9612233d-027a-49e1-bb03-c9dc4f8ecb6b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-71b76a-64f77d669f-hm2nn" Oct 31 00:08:48.590359 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:48.554090 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-48fdn\" (UniqueName: \"kubernetes.io/projected/9612233d-027a-49e1-bb03-c9dc4f8ecb6b-kube-api-access-48fdn\") pod \"payload-pre-processing-e2e-api-a-71b76a-64f77d669f-hm2nn\" (UID: \"9612233d-027a-49e1-bb03-c9dc4f8ecb6b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-71b76a-64f77d669f-hm2nn" Oct 31 00:08:48.654560 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:48.654456 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-kzqr4"] Oct 31 00:08:48.654860 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:48.654838 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-jdcvx"] Oct 31 00:08:48.655194 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:48.655166 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-jdcvx" podUID="a520a0b5-deab-4825-9b36-329c399754f6" containerName="istio-proxy" containerID="cri-o://7176a65eae8ecdf19042d0452aac23fa54b6cb2c488a78b8fd12c4e09bc31744" gracePeriod=30 Oct 31 00:08:48.659374 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:48.656143 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-kzqr4" podUID="a550117e-ad41-4e7c-998d-910a8d13d617" containerName="istio-proxy" containerID="cri-o://8351cbcebc39fc6a3f1bd858ac5bebc698fad85fb20361588f689aae67eaefa9" gracePeriod=30 Oct 31 00:08:48.759353 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:48.750127 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-tgc8q"] Oct 31 00:08:48.759353 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:48.750634 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-tgc8q" podUID="3e8b64ac-0b2a-40aa-ab07-25ec13005517" containerName="istio-proxy" containerID="cri-o://d67f2bd16f7e33e47e54489c1a91fb2008dda3cfc24949e624a22d15d0ebf8e1" gracePeriod=30 Oct 31 00:08:48.784353 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:48.782270 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-71b76a-64f77d669f-hm2nn" Oct 31 00:08:48.818356 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:48.813746 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-ccdzj"] Oct 31 00:08:48.818356 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:48.813996 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-ccdzj" podUID="afc963ee-cfd4-441b-9185-a6ae83276e19" containerName="istio-proxy" containerID="cri-o://51b115498114d0ec225e6a94c5934f19833fa1d6d10f859b7f49762a5b8d07df" gracePeriod=30 Oct 31 00:08:49.349092 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:49.347947 4927 patch_prober.go:29] interesting pod/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-7gm84 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 00:08:49.349092 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:49.348016 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-7gm84" podUID="45977fb0-7f5d-4f55-9530-112d60f269ab" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:08:49.731695 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:49.730383 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-71b76a-64f77d669f-hm2nn"] Oct 31 00:08:49.826403 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:49.806414 4927 patch_prober.go:29] interesting pod/e2e-worker-w6-35a187-openshift-default-5b54c65f75-94tkp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 00:08:49.826403 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:49.806563 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-94tkp" podUID="9f348596-394c-4bfb-ba79-4788da8fc6de" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:08:49.983384 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:08:49.976691 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sqv69" containerName="extractor" Oct 31 00:08:50.112381 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:50.112322 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-71b76a-64f77d669f-hm2nn" event={"ID":"9612233d-027a-49e1-bb03-c9dc4f8ecb6b","Type":"ContainerStarted","Data":"77344b1604184daed19b299ebe663a5508b2d1190c28b744609b466c64b4a6ff"} Oct 31 00:08:51.318681 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:51.318630 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-71b76a-64f77d669f-hm2nn" event={"ID":"9612233d-027a-49e1-bb03-c9dc4f8ecb6b","Type":"ContainerStarted","Data":"3e1e344c5b15eadaedb6d60603ae37e296b562951294bc6606209b7eaa47b228"} Oct 31 00:08:51.320742 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:51.320719 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-71b76a-64f77d669f-hm2nn" Oct 31 00:08:51.400494 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:51.374822 4927 patch_prober.go:29] interesting pod/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-2qnns container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 00:08:51.400494 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:51.374885 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-2qnns" podUID="b7194267-cfef-4252-ae16-a194eccd2a1f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:08:51.491551 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:51.491482 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-71b76a-64f77d669f-hm2nn" podStartSLOduration=3.491462277 podStartE2EDuration="3.491462277s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:08:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:08:51.465288705 +0000 UTC m=+1644.501619536" watchObservedRunningTime="2026-10-31 00:08:51.491462277 +0000 UTC m=+1644.527793100" Oct 31 00:08:51.538358 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:51.511304 4927 patch_prober.go:29] interesting pod/e2e-worker-w6-35a187-openshift-default-5b54c65f75-nr4qz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 00:08:51.538918 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:51.538874 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-nr4qz" podUID="333d9301-4a9b-4867-b3d9-88d3766d1acb" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:08:52.167358 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:52.166769 4927 patch_prober.go:29] interesting pod/e2e-worker-w0-46b586-openshift-default-86c598c96c-ccdzj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 00:08:52.167358 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:52.166863 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-ccdzj" podUID="afc963ee-cfd4-441b-9185-a6ae83276e19" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:08:52.302264 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:52.302223 4927 patch_prober.go:29] interesting pod/e2e-worker-w6-35a187-openshift-default-5b54c65f75-klh6t container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 00:08:52.302486 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:52.302293 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-klh6t" podUID="b7eb6748-2997-4527-91b3-86900ea70eed" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:08:52.857278 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:52.857243 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-94tkp" Oct 31 00:08:52.956519 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:52.956489 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-xgpsz" Oct 31 00:08:52.995524 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:52.995483 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f348596-394c-4bfb-ba79-4788da8fc6de-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f348596-394c-4bfb-ba79-4788da8fc6de-istio-data\") pod \"9f348596-394c-4bfb-ba79-4788da8fc6de\" (UID: \"9f348596-394c-4bfb-ba79-4788da8fc6de\") " Oct 31 00:08:52.996152 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:52.996129 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f348596-394c-4bfb-ba79-4788da8fc6de-kube-api-access-ml2l5\" (UniqueName: \"kubernetes.io/projected/9f348596-394c-4bfb-ba79-4788da8fc6de-kube-api-access-ml2l5\") pod \"9f348596-394c-4bfb-ba79-4788da8fc6de\" (UID: \"9f348596-394c-4bfb-ba79-4788da8fc6de\") " Oct 31 00:08:52.997417 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:52.995999 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f348596-394c-4bfb-ba79-4788da8fc6de-istio-data" pod "9f348596-394c-4bfb-ba79-4788da8fc6de" (UID: "9f348596-394c-4bfb-ba79-4788da8fc6de"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:08:53.002310 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:53.002278 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f348596-394c-4bfb-ba79-4788da8fc6de-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f348596-394c-4bfb-ba79-4788da8fc6de-credential-socket\") pod \"9f348596-394c-4bfb-ba79-4788da8fc6de\" (UID: \"9f348596-394c-4bfb-ba79-4788da8fc6de\") " Oct 31 00:08:53.002626 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:53.002607 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f348596-394c-4bfb-ba79-4788da8fc6de-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f348596-394c-4bfb-ba79-4788da8fc6de-istiod-ca-cert\") pod \"9f348596-394c-4bfb-ba79-4788da8fc6de\" (UID: \"9f348596-394c-4bfb-ba79-4788da8fc6de\") " Oct 31 00:08:53.002842 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:53.002825 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f348596-394c-4bfb-ba79-4788da8fc6de-istio-token\" (UniqueName: \"kubernetes.io/projected/9f348596-394c-4bfb-ba79-4788da8fc6de-istio-token\") pod \"9f348596-394c-4bfb-ba79-4788da8fc6de\" (UID: \"9f348596-394c-4bfb-ba79-4788da8fc6de\") " Oct 31 00:08:53.002982 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:53.002966 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f348596-394c-4bfb-ba79-4788da8fc6de-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f348596-394c-4bfb-ba79-4788da8fc6de-istio-envoy\") pod \"9f348596-394c-4bfb-ba79-4788da8fc6de\" (UID: \"9f348596-394c-4bfb-ba79-4788da8fc6de\") " Oct 31 00:08:53.006382 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:53.002603 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f348596-394c-4bfb-ba79-4788da8fc6de-credential-socket" pod "9f348596-394c-4bfb-ba79-4788da8fc6de" (UID: "9f348596-394c-4bfb-ba79-4788da8fc6de"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:08:53.006489 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:53.002865 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f348596-394c-4bfb-ba79-4788da8fc6de-kube-api-access-ml2l5" pod "9f348596-394c-4bfb-ba79-4788da8fc6de" (UID: "9f348596-394c-4bfb-ba79-4788da8fc6de"). InnerVolumeSpecName "kube-api-access-ml2l5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:08:53.006489 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:53.002876 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f348596-394c-4bfb-ba79-4788da8fc6de-istiod-ca-cert" pod "9f348596-394c-4bfb-ba79-4788da8fc6de" (UID: "9f348596-394c-4bfb-ba79-4788da8fc6de"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:08:53.006489 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:53.005576 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f348596-394c-4bfb-ba79-4788da8fc6de-istio-envoy" pod "9f348596-394c-4bfb-ba79-4788da8fc6de" (UID: "9f348596-394c-4bfb-ba79-4788da8fc6de"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:08:53.006717 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:53.006697 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f348596-394c-4bfb-ba79-4788da8fc6de-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f348596-394c-4bfb-ba79-4788da8fc6de-workload-socket\") pod \"9f348596-394c-4bfb-ba79-4788da8fc6de\" (UID: \"9f348596-394c-4bfb-ba79-4788da8fc6de\") " Oct 31 00:08:53.006896 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:53.006877 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f348596-394c-4bfb-ba79-4788da8fc6de-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f348596-394c-4bfb-ba79-4788da8fc6de-workload-certs\") pod \"9f348596-394c-4bfb-ba79-4788da8fc6de\" (UID: \"9f348596-394c-4bfb-ba79-4788da8fc6de\") " Oct 31 00:08:53.007031 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:53.007018 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9f348596-394c-4bfb-ba79-4788da8fc6de-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f348596-394c-4bfb-ba79-4788da8fc6de-istio-podinfo\") pod \"9f348596-394c-4bfb-ba79-4788da8fc6de\" (UID: \"9f348596-394c-4bfb-ba79-4788da8fc6de\") " Oct 31 00:08:53.007299 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:53.006903 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f348596-394c-4bfb-ba79-4788da8fc6de-workload-socket" pod "9f348596-394c-4bfb-ba79-4788da8fc6de" (UID: "9f348596-394c-4bfb-ba79-4788da8fc6de"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:08:53.007299 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:53.007019 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f348596-394c-4bfb-ba79-4788da8fc6de-workload-certs" pod "9f348596-394c-4bfb-ba79-4788da8fc6de" (UID: "9f348596-394c-4bfb-ba79-4788da8fc6de"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:08:53.008226 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:53.008197 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ml2l5\" (UniqueName: \"kubernetes.io/projected/9f348596-394c-4bfb-ba79-4788da8fc6de-kube-api-access-ml2l5\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:08:53.008327 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:53.008230 4927 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f348596-394c-4bfb-ba79-4788da8fc6de-credential-socket\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:08:53.008327 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:53.008263 4927 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f348596-394c-4bfb-ba79-4788da8fc6de-istiod-ca-cert\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:08:53.008327 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:53.008278 4927 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f348596-394c-4bfb-ba79-4788da8fc6de-istio-envoy\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:08:53.008327 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:53.008290 4927 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f348596-394c-4bfb-ba79-4788da8fc6de-workload-socket\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:08:53.008327 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:53.008304 4927 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f348596-394c-4bfb-ba79-4788da8fc6de-workload-certs\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:08:53.008327 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:53.008317 4927 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f348596-394c-4bfb-ba79-4788da8fc6de-istio-data\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:08:53.009018 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:53.008998 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9f348596-394c-4bfb-ba79-4788da8fc6de-istio-podinfo" pod "9f348596-394c-4bfb-ba79-4788da8fc6de" (UID: "9f348596-394c-4bfb-ba79-4788da8fc6de"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:08:53.010572 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:53.010547 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f348596-394c-4bfb-ba79-4788da8fc6de-istio-token" pod "9f348596-394c-4bfb-ba79-4788da8fc6de" (UID: "9f348596-394c-4bfb-ba79-4788da8fc6de"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:08:53.110022 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:53.109938 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6b778b5-4861-4a3f-85cb-5524098583f1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6b778b5-4861-4a3f-85cb-5524098583f1-istio-envoy\") pod \"e6b778b5-4861-4a3f-85cb-5524098583f1\" (UID: \"e6b778b5-4861-4a3f-85cb-5524098583f1\") " Oct 31 00:08:53.110022 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:53.110003 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6b778b5-4861-4a3f-85cb-5524098583f1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6b778b5-4861-4a3f-85cb-5524098583f1-istio-data\") pod \"e6b778b5-4861-4a3f-85cb-5524098583f1\" (UID: \"e6b778b5-4861-4a3f-85cb-5524098583f1\") " Oct 31 00:08:53.110203 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:53.110077 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6b778b5-4861-4a3f-85cb-5524098583f1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6b778b5-4861-4a3f-85cb-5524098583f1-credential-socket\") pod \"e6b778b5-4861-4a3f-85cb-5524098583f1\" (UID: \"e6b778b5-4861-4a3f-85cb-5524098583f1\") " Oct 31 00:08:53.110262 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:53.110202 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6b778b5-4861-4a3f-85cb-5524098583f1-istio-token\" (UniqueName: \"kubernetes.io/projected/e6b778b5-4861-4a3f-85cb-5524098583f1-istio-token\") pod \"e6b778b5-4861-4a3f-85cb-5524098583f1\" (UID: \"e6b778b5-4861-4a3f-85cb-5524098583f1\") " Oct 31 00:08:53.110372 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:53.110279 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e6b778b5-4861-4a3f-85cb-5524098583f1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6b778b5-4861-4a3f-85cb-5524098583f1-istiod-ca-cert\") pod \"e6b778b5-4861-4a3f-85cb-5524098583f1\" (UID: \"e6b778b5-4861-4a3f-85cb-5524098583f1\") " Oct 31 00:08:53.110460 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:53.110388 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e6b778b5-4861-4a3f-85cb-5524098583f1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6b778b5-4861-4a3f-85cb-5524098583f1-istio-podinfo\") pod \"e6b778b5-4861-4a3f-85cb-5524098583f1\" (UID: \"e6b778b5-4861-4a3f-85cb-5524098583f1\") " Oct 31 00:08:53.110591 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:53.110571 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6b778b5-4861-4a3f-85cb-5524098583f1-kube-api-access-mtgd5\" (UniqueName: \"kubernetes.io/projected/e6b778b5-4861-4a3f-85cb-5524098583f1-kube-api-access-mtgd5\") pod \"e6b778b5-4861-4a3f-85cb-5524098583f1\" (UID: \"e6b778b5-4861-4a3f-85cb-5524098583f1\") " Oct 31 00:08:53.110677 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:53.110641 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6b778b5-4861-4a3f-85cb-5524098583f1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6b778b5-4861-4a3f-85cb-5524098583f1-workload-certs\") pod \"e6b778b5-4861-4a3f-85cb-5524098583f1\" (UID: \"e6b778b5-4861-4a3f-85cb-5524098583f1\") " Oct 31 00:08:53.110732 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:53.110677 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6b778b5-4861-4a3f-85cb-5524098583f1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6b778b5-4861-4a3f-85cb-5524098583f1-workload-socket\") pod \"e6b778b5-4861-4a3f-85cb-5524098583f1\" (UID: \"e6b778b5-4861-4a3f-85cb-5524098583f1\") " Oct 31 00:08:53.111292 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:53.111272 4927 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f348596-394c-4bfb-ba79-4788da8fc6de-istio-token\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:08:53.111416 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:53.111296 4927 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f348596-394c-4bfb-ba79-4788da8fc6de-istio-podinfo\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:08:53.111789 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:53.111761 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6b778b5-4861-4a3f-85cb-5524098583f1-workload-socket" pod "e6b778b5-4861-4a3f-85cb-5524098583f1" (UID: "e6b778b5-4861-4a3f-85cb-5524098583f1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:08:53.112479 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:53.112449 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6b778b5-4861-4a3f-85cb-5524098583f1-istio-data" pod "e6b778b5-4861-4a3f-85cb-5524098583f1" (UID: "e6b778b5-4861-4a3f-85cb-5524098583f1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:08:53.112628 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:53.112592 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6b778b5-4861-4a3f-85cb-5524098583f1-workload-certs" pod "e6b778b5-4861-4a3f-85cb-5524098583f1" (UID: "e6b778b5-4861-4a3f-85cb-5524098583f1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:08:53.113079 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:53.113044 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6b778b5-4861-4a3f-85cb-5524098583f1-credential-socket" pod "e6b778b5-4861-4a3f-85cb-5524098583f1" (UID: "e6b778b5-4861-4a3f-85cb-5524098583f1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:08:53.113457 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:53.113435 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6b778b5-4861-4a3f-85cb-5524098583f1-istiod-ca-cert" pod "e6b778b5-4861-4a3f-85cb-5524098583f1" (UID: "e6b778b5-4861-4a3f-85cb-5524098583f1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:08:53.122562 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:53.122531 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6b778b5-4861-4a3f-85cb-5524098583f1-kube-api-access-mtgd5" pod "e6b778b5-4861-4a3f-85cb-5524098583f1" (UID: "e6b778b5-4861-4a3f-85cb-5524098583f1"). InnerVolumeSpecName "kube-api-access-mtgd5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:08:53.138278 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:53.138240 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e6b778b5-4861-4a3f-85cb-5524098583f1-istio-podinfo" pod "e6b778b5-4861-4a3f-85cb-5524098583f1" (UID: "e6b778b5-4861-4a3f-85cb-5524098583f1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:08:53.138436 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:53.138270 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6b778b5-4861-4a3f-85cb-5524098583f1-istio-token" pod "e6b778b5-4861-4a3f-85cb-5524098583f1" (UID: "e6b778b5-4861-4a3f-85cb-5524098583f1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:08:53.140013 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:53.139921 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6b778b5-4861-4a3f-85cb-5524098583f1-istio-envoy" pod "e6b778b5-4861-4a3f-85cb-5524098583f1" (UID: "e6b778b5-4861-4a3f-85cb-5524098583f1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:08:53.214903 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:53.213968 4927 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6b778b5-4861-4a3f-85cb-5524098583f1-istio-envoy\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:08:53.214903 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:53.214010 4927 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6b778b5-4861-4a3f-85cb-5524098583f1-istio-data\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:08:53.214903 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:53.214028 4927 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6b778b5-4861-4a3f-85cb-5524098583f1-credential-socket\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:08:53.214903 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:53.214043 4927 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6b778b5-4861-4a3f-85cb-5524098583f1-istio-token\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:08:53.214903 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:53.214064 4927 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6b778b5-4861-4a3f-85cb-5524098583f1-istiod-ca-cert\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:08:53.214903 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:53.214084 4927 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6b778b5-4861-4a3f-85cb-5524098583f1-istio-podinfo\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:08:53.214903 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:53.214101 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mtgd5\" (UniqueName: \"kubernetes.io/projected/e6b778b5-4861-4a3f-85cb-5524098583f1-kube-api-access-mtgd5\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:08:53.214903 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:53.214118 4927 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6b778b5-4861-4a3f-85cb-5524098583f1-workload-certs\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:08:53.214903 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:53.214131 4927 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6b778b5-4861-4a3f-85cb-5524098583f1-workload-socket\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:08:53.263980 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:53.263520 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-klh6t" Oct 31 00:08:53.279601 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:53.279509 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-nr4qz" Oct 31 00:08:53.310406 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:53.293930 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:08:53.587817 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:53.587530 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b7eb6748-2997-4527-91b3-86900ea70eed-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7eb6748-2997-4527-91b3-86900ea70eed-istiod-ca-cert\") pod \"b7eb6748-2997-4527-91b3-86900ea70eed\" (UID: \"b7eb6748-2997-4527-91b3-86900ea70eed\") " Oct 31 00:08:53.587817 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:53.587627 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7eb6748-2997-4527-91b3-86900ea70eed-istio-token\" (UniqueName: \"kubernetes.io/projected/b7eb6748-2997-4527-91b3-86900ea70eed-istio-token\") pod \"b7eb6748-2997-4527-91b3-86900ea70eed\" (UID: \"b7eb6748-2997-4527-91b3-86900ea70eed\") " Oct 31 00:08:53.587817 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:53.587669 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/333d9301-4a9b-4867-b3d9-88d3766d1acb-kube-api-access-hbr6r\" (UniqueName: \"kubernetes.io/projected/333d9301-4a9b-4867-b3d9-88d3766d1acb-kube-api-access-hbr6r\") pod \"333d9301-4a9b-4867-b3d9-88d3766d1acb\" (UID: \"333d9301-4a9b-4867-b3d9-88d3766d1acb\") " Oct 31 00:08:53.587817 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:53.587698 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b7eb6748-2997-4527-91b3-86900ea70eed-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7eb6748-2997-4527-91b3-86900ea70eed-istio-podinfo\") pod \"b7eb6748-2997-4527-91b3-86900ea70eed\" (UID: \"b7eb6748-2997-4527-91b3-86900ea70eed\") " Oct 31 00:08:53.587817 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:53.587730 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7eb6748-2997-4527-91b3-86900ea70eed-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7eb6748-2997-4527-91b3-86900ea70eed-istio-data\") pod \"b7eb6748-2997-4527-91b3-86900ea70eed\" (UID: \"b7eb6748-2997-4527-91b3-86900ea70eed\") " Oct 31 00:08:53.587817 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:53.587778 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/333d9301-4a9b-4867-b3d9-88d3766d1acb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/333d9301-4a9b-4867-b3d9-88d3766d1acb-istio-data\") pod \"333d9301-4a9b-4867-b3d9-88d3766d1acb\" (UID: \"333d9301-4a9b-4867-b3d9-88d3766d1acb\") " Oct 31 00:08:53.587817 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:53.587813 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/333d9301-4a9b-4867-b3d9-88d3766d1acb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/333d9301-4a9b-4867-b3d9-88d3766d1acb-istiod-ca-cert\") pod \"333d9301-4a9b-4867-b3d9-88d3766d1acb\" (UID: \"333d9301-4a9b-4867-b3d9-88d3766d1acb\") " Oct 31 00:08:53.588295 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:53.587848 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/333d9301-4a9b-4867-b3d9-88d3766d1acb-istio-token\" (UniqueName: \"kubernetes.io/projected/333d9301-4a9b-4867-b3d9-88d3766d1acb-istio-token\") pod \"333d9301-4a9b-4867-b3d9-88d3766d1acb\" (UID: \"333d9301-4a9b-4867-b3d9-88d3766d1acb\") " Oct 31 00:08:53.588295 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:53.587902 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/333d9301-4a9b-4867-b3d9-88d3766d1acb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/333d9301-4a9b-4867-b3d9-88d3766d1acb-istio-podinfo\") pod \"333d9301-4a9b-4867-b3d9-88d3766d1acb\" (UID: \"333d9301-4a9b-4867-b3d9-88d3766d1acb\") " Oct 31 00:08:53.588295 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:53.587987 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/333d9301-4a9b-4867-b3d9-88d3766d1acb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/333d9301-4a9b-4867-b3d9-88d3766d1acb-credential-socket\") pod \"333d9301-4a9b-4867-b3d9-88d3766d1acb\" (UID: \"333d9301-4a9b-4867-b3d9-88d3766d1acb\") " Oct 31 00:08:53.588295 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:53.588013 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7eb6748-2997-4527-91b3-86900ea70eed-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7eb6748-2997-4527-91b3-86900ea70eed-credential-socket\") pod \"b7eb6748-2997-4527-91b3-86900ea70eed\" (UID: \"b7eb6748-2997-4527-91b3-86900ea70eed\") " Oct 31 00:08:53.588295 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:53.588081 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/333d9301-4a9b-4867-b3d9-88d3766d1acb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/333d9301-4a9b-4867-b3d9-88d3766d1acb-workload-socket\") pod \"333d9301-4a9b-4867-b3d9-88d3766d1acb\" (UID: \"333d9301-4a9b-4867-b3d9-88d3766d1acb\") " Oct 31 00:08:53.588295 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:53.588238 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/333d9301-4a9b-4867-b3d9-88d3766d1acb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/333d9301-4a9b-4867-b3d9-88d3766d1acb-workload-certs\") pod \"333d9301-4a9b-4867-b3d9-88d3766d1acb\" (UID: \"333d9301-4a9b-4867-b3d9-88d3766d1acb\") " Oct 31 00:08:53.588295 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:53.588290 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7eb6748-2997-4527-91b3-86900ea70eed-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7eb6748-2997-4527-91b3-86900ea70eed-workload-certs\") pod \"b7eb6748-2997-4527-91b3-86900ea70eed\" (UID: \"b7eb6748-2997-4527-91b3-86900ea70eed\") " Oct 31 00:08:53.588680 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:53.588365 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7eb6748-2997-4527-91b3-86900ea70eed-kube-api-access-dlb46\" (UniqueName: \"kubernetes.io/projected/b7eb6748-2997-4527-91b3-86900ea70eed-kube-api-access-dlb46\") pod \"b7eb6748-2997-4527-91b3-86900ea70eed\" (UID: \"b7eb6748-2997-4527-91b3-86900ea70eed\") " Oct 31 00:08:53.588680 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:53.588474 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7eb6748-2997-4527-91b3-86900ea70eed-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7eb6748-2997-4527-91b3-86900ea70eed-istio-envoy\") pod \"b7eb6748-2997-4527-91b3-86900ea70eed\" (UID: \"b7eb6748-2997-4527-91b3-86900ea70eed\") " Oct 31 00:08:53.588680 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:53.588591 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/333d9301-4a9b-4867-b3d9-88d3766d1acb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/333d9301-4a9b-4867-b3d9-88d3766d1acb-istio-envoy\") pod \"333d9301-4a9b-4867-b3d9-88d3766d1acb\" (UID: \"333d9301-4a9b-4867-b3d9-88d3766d1acb\") " Oct 31 00:08:53.588832 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:53.588805 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7eb6748-2997-4527-91b3-86900ea70eed-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7eb6748-2997-4527-91b3-86900ea70eed-workload-socket\") pod \"b7eb6748-2997-4527-91b3-86900ea70eed\" (UID: \"b7eb6748-2997-4527-91b3-86900ea70eed\") " Oct 31 00:08:53.590112 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:53.589283 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/333d9301-4a9b-4867-b3d9-88d3766d1acb-workload-certs" pod "333d9301-4a9b-4867-b3d9-88d3766d1acb" (UID: "333d9301-4a9b-4867-b3d9-88d3766d1acb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:08:53.590112 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:53.589541 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/333d9301-4a9b-4867-b3d9-88d3766d1acb-credential-socket" pod "333d9301-4a9b-4867-b3d9-88d3766d1acb" (UID: "333d9301-4a9b-4867-b3d9-88d3766d1acb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:08:53.590112 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:53.589836 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7eb6748-2997-4527-91b3-86900ea70eed-workload-certs" pod "b7eb6748-2997-4527-91b3-86900ea70eed" (UID: "b7eb6748-2997-4527-91b3-86900ea70eed"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:08:53.590112 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:53.590016 4927 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/333d9301-4a9b-4867-b3d9-88d3766d1acb-workload-certs\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:08:53.590112 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:53.590039 4927 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7eb6748-2997-4527-91b3-86900ea70eed-workload-certs\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:08:53.590112 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:53.590056 4927 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/333d9301-4a9b-4867-b3d9-88d3766d1acb-credential-socket\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:08:53.590526 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:53.590257 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7eb6748-2997-4527-91b3-86900ea70eed-credential-socket" pod "b7eb6748-2997-4527-91b3-86900ea70eed" (UID: "b7eb6748-2997-4527-91b3-86900ea70eed"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:08:53.590526 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:53.590464 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/333d9301-4a9b-4867-b3d9-88d3766d1acb-workload-socket" pod "333d9301-4a9b-4867-b3d9-88d3766d1acb" (UID: "333d9301-4a9b-4867-b3d9-88d3766d1acb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:08:53.592165 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:53.591607 4927 generic.go:356] "Generic (PLEG): container finished" podID="9f348596-394c-4bfb-ba79-4788da8fc6de" containerID="9ea8e8a0622bd064854d976900a471b591cf0ad747977a10109b496f5289b8df" exitCode=0 Oct 31 00:08:53.592165 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:53.591703 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-94tkp" event={"ID":"9f348596-394c-4bfb-ba79-4788da8fc6de","Type":"ContainerDied","Data":"9ea8e8a0622bd064854d976900a471b591cf0ad747977a10109b496f5289b8df"} Oct 31 00:08:53.592165 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:53.591757 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-94tkp" event={"ID":"9f348596-394c-4bfb-ba79-4788da8fc6de","Type":"ContainerDied","Data":"f83de21595e5c3c41d8b1dc997798e97442e8bb3e5a0cb3c45cbfc86d21c97ac"} Oct 31 00:08:53.592165 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:53.591785 4927 scope.go:122] "RemoveContainer" containerID="9ea8e8a0622bd064854d976900a471b591cf0ad747977a10109b496f5289b8df" Oct 31 00:08:53.592165 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:53.591972 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-94tkp" Oct 31 00:08:53.597933 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:53.597908 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7eb6748-2997-4527-91b3-86900ea70eed-kube-api-access-dlb46" pod "b7eb6748-2997-4527-91b3-86900ea70eed" (UID: "b7eb6748-2997-4527-91b3-86900ea70eed"). InnerVolumeSpecName "kube-api-access-dlb46". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:08:53.603452 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:53.598778 4927 generic.go:356] "Generic (PLEG): container finished" podID="e6b778b5-4861-4a3f-85cb-5524098583f1" containerID="dae44a8554e1269e47872205987bfe082bbc0e2df1fa6679ab5137a80ab6ad25" exitCode=0 Oct 31 00:08:53.603452 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:53.598856 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-xgpsz" event={"ID":"e6b778b5-4861-4a3f-85cb-5524098583f1","Type":"ContainerDied","Data":"dae44a8554e1269e47872205987bfe082bbc0e2df1fa6679ab5137a80ab6ad25"} Oct 31 00:08:53.603452 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:53.598900 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-xgpsz" event={"ID":"e6b778b5-4861-4a3f-85cb-5524098583f1","Type":"ContainerDied","Data":"c80c8ebf1c96d05ba6484b49689cd0bb0896ec05eb54465160de443bb1e3b6ee"} Oct 31 00:08:53.603452 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:53.598970 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-xgpsz" Oct 31 00:08:53.603797 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:53.603773 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7eb6748-2997-4527-91b3-86900ea70eed-workload-socket" pod "b7eb6748-2997-4527-91b3-86900ea70eed" (UID: "b7eb6748-2997-4527-91b3-86900ea70eed"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:08:53.666608 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:53.666171 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7eb6748-2997-4527-91b3-86900ea70eed-istiod-ca-cert" pod "b7eb6748-2997-4527-91b3-86900ea70eed" (UID: "b7eb6748-2997-4527-91b3-86900ea70eed"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:08:53.679769 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:53.679708 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/333d9301-4a9b-4867-b3d9-88d3766d1acb-istio-envoy" pod "333d9301-4a9b-4867-b3d9-88d3766d1acb" (UID: "333d9301-4a9b-4867-b3d9-88d3766d1acb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:08:53.681163 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:53.681110 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/333d9301-4a9b-4867-b3d9-88d3766d1acb-istio-podinfo" pod "333d9301-4a9b-4867-b3d9-88d3766d1acb" (UID: "333d9301-4a9b-4867-b3d9-88d3766d1acb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:08:53.814717 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:53.749701 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7eb6748-2997-4527-91b3-86900ea70eed-istio-envoy" pod "b7eb6748-2997-4527-91b3-86900ea70eed" (UID: "b7eb6748-2997-4527-91b3-86900ea70eed"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:08:53.814717 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:53.771711 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7eb6748-2997-4527-91b3-86900ea70eed-istio-token" pod "b7eb6748-2997-4527-91b3-86900ea70eed" (UID: "b7eb6748-2997-4527-91b3-86900ea70eed"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:08:53.814717 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:53.777882 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/333d9301-4a9b-4867-b3d9-88d3766d1acb-istiod-ca-cert" pod "333d9301-4a9b-4867-b3d9-88d3766d1acb" (UID: "333d9301-4a9b-4867-b3d9-88d3766d1acb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:08:53.814717 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:53.780452 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/333d9301-4a9b-4867-b3d9-88d3766d1acb-istio-data" pod "333d9301-4a9b-4867-b3d9-88d3766d1acb" (UID: "333d9301-4a9b-4867-b3d9-88d3766d1acb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:08:53.814717 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:53.683239 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/333d9301-4a9b-4867-b3d9-88d3766d1acb-kube-api-access-hbr6r" pod "333d9301-4a9b-4867-b3d9-88d3766d1acb" (UID: "333d9301-4a9b-4867-b3d9-88d3766d1acb"). InnerVolumeSpecName "kube-api-access-hbr6r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:08:53.814717 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:53.803134 4927 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/333d9301-4a9b-4867-b3d9-88d3766d1acb-istio-envoy\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:08:53.814717 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:53.803169 4927 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7eb6748-2997-4527-91b3-86900ea70eed-workload-socket\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:08:53.814717 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:53.803186 4927 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7eb6748-2997-4527-91b3-86900ea70eed-istiod-ca-cert\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:08:53.814717 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:53.803201 4927 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7eb6748-2997-4527-91b3-86900ea70eed-istio-token\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:08:53.814717 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:53.803221 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hbr6r\" (UniqueName: \"kubernetes.io/projected/333d9301-4a9b-4867-b3d9-88d3766d1acb-kube-api-access-hbr6r\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:08:53.814717 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:53.803239 4927 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/333d9301-4a9b-4867-b3d9-88d3766d1acb-istio-data\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:08:53.814717 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:53.803369 4927 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/333d9301-4a9b-4867-b3d9-88d3766d1acb-istiod-ca-cert\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:08:53.814717 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:53.803385 4927 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/333d9301-4a9b-4867-b3d9-88d3766d1acb-istio-podinfo\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:08:53.814717 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:53.803407 4927 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7eb6748-2997-4527-91b3-86900ea70eed-credential-socket\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:08:53.814717 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:53.803421 4927 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/333d9301-4a9b-4867-b3d9-88d3766d1acb-workload-socket\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:08:53.814717 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:53.803437 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dlb46\" (UniqueName: \"kubernetes.io/projected/b7eb6748-2997-4527-91b3-86900ea70eed-kube-api-access-dlb46\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:08:53.814717 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:53.803452 4927 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7eb6748-2997-4527-91b3-86900ea70eed-istio-envoy\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:08:53.853747 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:53.827226 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7eb6748-2997-4527-91b3-86900ea70eed-istio-data" pod "b7eb6748-2997-4527-91b3-86900ea70eed" (UID: "b7eb6748-2997-4527-91b3-86900ea70eed"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:08:53.881454 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:53.880302 4927 generic.go:356] "Generic (PLEG): container finished" podID="45977fb0-7f5d-4f55-9530-112d60f269ab" containerID="646c05c17da8a4f530f45bf9bb4e8348afb3fbe87e9040d1ee2cea4e92dccb62" exitCode=0 Oct 31 00:08:53.881454 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:53.880503 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-7gm84" event={"ID":"45977fb0-7f5d-4f55-9530-112d60f269ab","Type":"ContainerDied","Data":"646c05c17da8a4f530f45bf9bb4e8348afb3fbe87e9040d1ee2cea4e92dccb62"} Oct 31 00:08:53.883550 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:53.883419 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:08:53.887218 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:53.887186 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b7eb6748-2997-4527-91b3-86900ea70eed-istio-podinfo" pod "b7eb6748-2997-4527-91b3-86900ea70eed" (UID: "b7eb6748-2997-4527-91b3-86900ea70eed"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:08:53.929488 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:53.916664 4927 patch_prober.go:29] interesting pod/e2e-worker-w0-46b586-openshift-default-86c598c96c-kzqr4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Oct 31 00:08:53.929596 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:53.929530 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-kzqr4" podUID="a550117e-ad41-4e7c-998d-910a8d13d617" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Oct 31 00:08:53.929875 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:53.927571 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/333d9301-4a9b-4867-b3d9-88d3766d1acb-istio-token" pod "333d9301-4a9b-4867-b3d9-88d3766d1acb" (UID: "333d9301-4a9b-4867-b3d9-88d3766d1acb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:08:53.934739 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:53.934704 4927 generic.go:356] "Generic (PLEG): container finished" podID="b7194267-cfef-4252-ae16-a194eccd2a1f" containerID="6616bfd1ea4a480ce02466ce388380eb31ea8e8030af93e453c8d0c7a09224b2" exitCode=0 Oct 31 00:08:53.934949 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:53.934928 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-2qnns" event={"ID":"b7194267-cfef-4252-ae16-a194eccd2a1f","Type":"ContainerDied","Data":"6616bfd1ea4a480ce02466ce388380eb31ea8e8030af93e453c8d0c7a09224b2"} Oct 31 00:08:53.936032 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:53.936007 4927 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7eb6748-2997-4527-91b3-86900ea70eed-istio-podinfo\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:08:53.936121 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:53.936037 4927 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7eb6748-2997-4527-91b3-86900ea70eed-istio-data\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:08:53.936121 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:53.936080 4927 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/333d9301-4a9b-4867-b3d9-88d3766d1acb-istio-token\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:08:53.947508 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:53.947482 4927 scope.go:122] "RemoveContainer" containerID="65e5c2072ab717203bf04484e90c092ed6756580261f380bc47fb601fcba23d1" Oct 31 00:08:53.990555 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:53.989395 4927 generic.go:356] "Generic (PLEG): container finished" podID="333d9301-4a9b-4867-b3d9-88d3766d1acb" containerID="2cbee7072e17e752090d8a76ddbb931486f62ce02a86fb77ec088ce1899fa412" exitCode=0 Oct 31 00:08:53.990555 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:53.989541 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-nr4qz" event={"ID":"333d9301-4a9b-4867-b3d9-88d3766d1acb","Type":"ContainerDied","Data":"2cbee7072e17e752090d8a76ddbb931486f62ce02a86fb77ec088ce1899fa412"} Oct 31 00:08:53.990555 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:53.989592 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-nr4qz" event={"ID":"333d9301-4a9b-4867-b3d9-88d3766d1acb","Type":"ContainerDied","Data":"0f90f2bb757a8342bd6e605c2931a2c8f35924148215d274f4cce376b9fee438"} Oct 31 00:08:53.993685 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:53.991957 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-nr4qz" Oct 31 00:08:54.038961 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:54.020421 4927 generic.go:356] "Generic (PLEG): container finished" podID="2b317f84-83ac-49aa-921e-0242889f2b4f" containerID="f2edac246d0fdf2378197ddfb3fb9eab46f7ea47415499c914edaee90412f22b" exitCode=0 Oct 31 00:08:54.038961 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:54.020556 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-khcqd" event={"ID":"2b317f84-83ac-49aa-921e-0242889f2b4f","Type":"ContainerDied","Data":"f2edac246d0fdf2378197ddfb3fb9eab46f7ea47415499c914edaee90412f22b"} Oct 31 00:08:54.038961 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:54.025908 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-khcqd" Oct 31 00:08:54.057526 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:54.057228 4927 generic.go:356] "Generic (PLEG): container finished" podID="b7eb6748-2997-4527-91b3-86900ea70eed" containerID="c4717a5f3b503ab1937f5d3e01e2b12f6a6927be6491842a06b68ade45ca38da" exitCode=0 Oct 31 00:08:54.063082 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:54.057808 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-klh6t" event={"ID":"b7eb6748-2997-4527-91b3-86900ea70eed","Type":"ContainerDied","Data":"c4717a5f3b503ab1937f5d3e01e2b12f6a6927be6491842a06b68ade45ca38da"} Oct 31 00:08:54.063082 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:54.057870 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-klh6t" event={"ID":"b7eb6748-2997-4527-91b3-86900ea70eed","Type":"ContainerDied","Data":"abcc70304f20a388439e4f3cfb51c9c1f49c64350f375da074508b2bbc46560f"} Oct 31 00:08:54.063082 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:54.057973 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-klh6t" Oct 31 00:08:54.063082 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:54.061894 4927 scope.go:122] "RemoveContainer" containerID="9ea8e8a0622bd064854d976900a471b591cf0ad747977a10109b496f5289b8df" Oct 31 00:08:54.073297 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:54.073259 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-xgpsz"] Oct 31 00:08:54.074294 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:54.074024 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-2qnns" Oct 31 00:08:54.076845 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:54.076818 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-vb9r5" Oct 31 00:08:54.100186 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:54.100152 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-xgpsz"] Oct 31 00:08:54.100894 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:08:54.100727 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ea8e8a0622bd064854d976900a471b591cf0ad747977a10109b496f5289b8df\": container with ID starting with 9ea8e8a0622bd064854d976900a471b591cf0ad747977a10109b496f5289b8df not found: ID does not exist" containerID="9ea8e8a0622bd064854d976900a471b591cf0ad747977a10109b496f5289b8df" Oct 31 00:08:54.101061 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:54.101037 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ea8e8a0622bd064854d976900a471b591cf0ad747977a10109b496f5289b8df"} err="failed to get container status \"9ea8e8a0622bd064854d976900a471b591cf0ad747977a10109b496f5289b8df\": rpc error: code = NotFound desc = could not find container \"9ea8e8a0622bd064854d976900a471b591cf0ad747977a10109b496f5289b8df\": container with ID starting with 9ea8e8a0622bd064854d976900a471b591cf0ad747977a10109b496f5289b8df not found: ID does not exist" Oct 31 00:08:54.101187 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:54.101173 4927 scope.go:122] "RemoveContainer" containerID="65e5c2072ab717203bf04484e90c092ed6756580261f380bc47fb601fcba23d1" Oct 31 00:08:54.104432 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:08:54.103551 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65e5c2072ab717203bf04484e90c092ed6756580261f380bc47fb601fcba23d1\": container with ID starting with 65e5c2072ab717203bf04484e90c092ed6756580261f380bc47fb601fcba23d1 not found: ID does not exist" containerID="65e5c2072ab717203bf04484e90c092ed6756580261f380bc47fb601fcba23d1" Oct 31 00:08:54.104432 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:54.103590 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65e5c2072ab717203bf04484e90c092ed6756580261f380bc47fb601fcba23d1"} err="failed to get container status \"65e5c2072ab717203bf04484e90c092ed6756580261f380bc47fb601fcba23d1\": rpc error: code = NotFound desc = could not find container \"65e5c2072ab717203bf04484e90c092ed6756580261f380bc47fb601fcba23d1\": container with ID starting with 65e5c2072ab717203bf04484e90c092ed6756580261f380bc47fb601fcba23d1 not found: ID does not exist" Oct 31 00:08:54.104432 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:54.103615 4927 scope.go:122] "RemoveContainer" containerID="dae44a8554e1269e47872205987bfe082bbc0e2df1fa6679ab5137a80ab6ad25" Oct 31 00:08:54.111281 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:54.104767 4927 generic.go:356] "Generic (PLEG): container finished" podID="334f41bf-6d81-474d-bddc-d38c3492d586" containerID="b2188dcfc96b95e4deeaf72da6a224c424e584af088e805d70914d27730aae5e" exitCode=0 Oct 31 00:08:54.111281 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:54.105039 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-vb9r5" event={"ID":"334f41bf-6d81-474d-bddc-d38c3492d586","Type":"ContainerDied","Data":"b2188dcfc96b95e4deeaf72da6a224c424e584af088e805d70914d27730aae5e"} Oct 31 00:08:54.111281 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:54.105151 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-7gm84" Oct 31 00:08:54.111281 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:54.111232 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:08:54.125967 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:54.125928 4927 scope.go:122] "RemoveContainer" containerID="dae44a8554e1269e47872205987bfe082bbc0e2df1fa6679ab5137a80ab6ad25" Oct 31 00:08:54.126456 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:08:54.126405 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dae44a8554e1269e47872205987bfe082bbc0e2df1fa6679ab5137a80ab6ad25\": container with ID starting with dae44a8554e1269e47872205987bfe082bbc0e2df1fa6679ab5137a80ab6ad25 not found: ID does not exist" containerID="dae44a8554e1269e47872205987bfe082bbc0e2df1fa6679ab5137a80ab6ad25" Oct 31 00:08:54.126569 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:54.126455 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dae44a8554e1269e47872205987bfe082bbc0e2df1fa6679ab5137a80ab6ad25"} err="failed to get container status \"dae44a8554e1269e47872205987bfe082bbc0e2df1fa6679ab5137a80ab6ad25\": rpc error: code = NotFound desc = could not find container \"dae44a8554e1269e47872205987bfe082bbc0e2df1fa6679ab5137a80ab6ad25\": container with ID starting with dae44a8554e1269e47872205987bfe082bbc0e2df1fa6679ab5137a80ab6ad25 not found: ID does not exist" Oct 31 00:08:54.126569 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:54.126484 4927 scope.go:122] "RemoveContainer" containerID="2cbee7072e17e752090d8a76ddbb931486f62ce02a86fb77ec088ce1899fa412" Oct 31 00:08:54.139283 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:54.139243 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b317f84-83ac-49aa-921e-0242889f2b4f-istio-token\" (UniqueName: \"kubernetes.io/projected/2b317f84-83ac-49aa-921e-0242889f2b4f-istio-token\") pod \"2b317f84-83ac-49aa-921e-0242889f2b4f\" (UID: \"2b317f84-83ac-49aa-921e-0242889f2b4f\") " Oct 31 00:08:54.139455 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:54.139295 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b7194267-cfef-4252-ae16-a194eccd2a1f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7194267-cfef-4252-ae16-a194eccd2a1f-istiod-ca-cert\") pod \"b7194267-cfef-4252-ae16-a194eccd2a1f\" (UID: \"b7194267-cfef-4252-ae16-a194eccd2a1f\") " Oct 31 00:08:54.139539 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:54.139509 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b317f84-83ac-49aa-921e-0242889f2b4f-kube-api-access-f7xcn\" (UniqueName: \"kubernetes.io/projected/2b317f84-83ac-49aa-921e-0242889f2b4f-kube-api-access-f7xcn\") pod \"2b317f84-83ac-49aa-921e-0242889f2b4f\" (UID: \"2b317f84-83ac-49aa-921e-0242889f2b4f\") " Oct 31 00:08:54.139741 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:54.139712 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7194267-cfef-4252-ae16-a194eccd2a1f-istiod-ca-cert" pod "b7194267-cfef-4252-ae16-a194eccd2a1f" (UID: "b7194267-cfef-4252-ae16-a194eccd2a1f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:08:54.140170 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:54.140134 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2b317f84-83ac-49aa-921e-0242889f2b4f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b317f84-83ac-49aa-921e-0242889f2b4f-istiod-ca-cert\") pod \"2b317f84-83ac-49aa-921e-0242889f2b4f\" (UID: \"2b317f84-83ac-49aa-921e-0242889f2b4f\") " Oct 31 00:08:54.140775 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:54.140739 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7194267-cfef-4252-ae16-a194eccd2a1f-credential-socket" pod "b7194267-cfef-4252-ae16-a194eccd2a1f" (UID: "b7194267-cfef-4252-ae16-a194eccd2a1f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:08:54.140775 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:54.140761 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b317f84-83ac-49aa-921e-0242889f2b4f-istiod-ca-cert" pod "2b317f84-83ac-49aa-921e-0242889f2b4f" (UID: "2b317f84-83ac-49aa-921e-0242889f2b4f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:08:54.141031 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:54.141012 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7194267-cfef-4252-ae16-a194eccd2a1f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7194267-cfef-4252-ae16-a194eccd2a1f-credential-socket\") pod \"b7194267-cfef-4252-ae16-a194eccd2a1f\" (UID: \"b7194267-cfef-4252-ae16-a194eccd2a1f\") " Oct 31 00:08:54.141250 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:54.141233 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b317f84-83ac-49aa-921e-0242889f2b4f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b317f84-83ac-49aa-921e-0242889f2b4f-credential-socket\") pod \"2b317f84-83ac-49aa-921e-0242889f2b4f\" (UID: \"2b317f84-83ac-49aa-921e-0242889f2b4f\") " Oct 31 00:08:54.143027 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:54.142250 4927 scope.go:122] "RemoveContainer" containerID="2cbee7072e17e752090d8a76ddbb931486f62ce02a86fb77ec088ce1899fa412" Oct 31 00:08:54.143154 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:54.142886 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b317f84-83ac-49aa-921e-0242889f2b4f-kube-api-access-f7xcn" pod "2b317f84-83ac-49aa-921e-0242889f2b4f" (UID: "2b317f84-83ac-49aa-921e-0242889f2b4f"). InnerVolumeSpecName "kube-api-access-f7xcn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:08:54.143154 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:54.143006 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2b317f84-83ac-49aa-921e-0242889f2b4f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b317f84-83ac-49aa-921e-0242889f2b4f-istio-podinfo\") pod \"2b317f84-83ac-49aa-921e-0242889f2b4f\" (UID: \"2b317f84-83ac-49aa-921e-0242889f2b4f\") " Oct 31 00:08:54.143268 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:54.142943 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b317f84-83ac-49aa-921e-0242889f2b4f-credential-socket" pod "2b317f84-83ac-49aa-921e-0242889f2b4f" (UID: "2b317f84-83ac-49aa-921e-0242889f2b4f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:08:54.143268 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:54.143210 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7194267-cfef-4252-ae16-a194eccd2a1f-istio-token\" (UniqueName: \"kubernetes.io/projected/b7194267-cfef-4252-ae16-a194eccd2a1f-istio-token\") pod \"b7194267-cfef-4252-ae16-a194eccd2a1f\" (UID: \"b7194267-cfef-4252-ae16-a194eccd2a1f\") " Oct 31 00:08:54.143268 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:54.143253 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b7194267-cfef-4252-ae16-a194eccd2a1f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7194267-cfef-4252-ae16-a194eccd2a1f-istio-podinfo\") pod \"b7194267-cfef-4252-ae16-a194eccd2a1f\" (UID: \"b7194267-cfef-4252-ae16-a194eccd2a1f\") " Oct 31 00:08:54.143430 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:54.143367 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7194267-cfef-4252-ae16-a194eccd2a1f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7194267-cfef-4252-ae16-a194eccd2a1f-istio-data\") pod \"b7194267-cfef-4252-ae16-a194eccd2a1f\" (UID: \"b7194267-cfef-4252-ae16-a194eccd2a1f\") " Oct 31 00:08:54.143430 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:54.143405 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b317f84-83ac-49aa-921e-0242889f2b4f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b317f84-83ac-49aa-921e-0242889f2b4f-workload-socket\") pod \"2b317f84-83ac-49aa-921e-0242889f2b4f\" (UID: \"2b317f84-83ac-49aa-921e-0242889f2b4f\") " Oct 31 00:08:54.143527 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:54.143464 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b317f84-83ac-49aa-921e-0242889f2b4f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b317f84-83ac-49aa-921e-0242889f2b4f-workload-certs\") pod \"2b317f84-83ac-49aa-921e-0242889f2b4f\" (UID: \"2b317f84-83ac-49aa-921e-0242889f2b4f\") " Oct 31 00:08:54.143527 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:54.143461 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b317f84-83ac-49aa-921e-0242889f2b4f-istio-token" pod "2b317f84-83ac-49aa-921e-0242889f2b4f" (UID: "2b317f84-83ac-49aa-921e-0242889f2b4f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:08:54.143527 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:54.143505 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7194267-cfef-4252-ae16-a194eccd2a1f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7194267-cfef-4252-ae16-a194eccd2a1f-istio-envoy\") pod \"b7194267-cfef-4252-ae16-a194eccd2a1f\" (UID: \"b7194267-cfef-4252-ae16-a194eccd2a1f\") " Oct 31 00:08:54.143527 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:08:54.143507 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2cbee7072e17e752090d8a76ddbb931486f62ce02a86fb77ec088ce1899fa412\": container with ID starting with 2cbee7072e17e752090d8a76ddbb931486f62ce02a86fb77ec088ce1899fa412 not found: ID does not exist" containerID="2cbee7072e17e752090d8a76ddbb931486f62ce02a86fb77ec088ce1899fa412" Oct 31 00:08:54.143775 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:54.143575 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b317f84-83ac-49aa-921e-0242889f2b4f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b317f84-83ac-49aa-921e-0242889f2b4f-istio-data\") pod \"2b317f84-83ac-49aa-921e-0242889f2b4f\" (UID: \"2b317f84-83ac-49aa-921e-0242889f2b4f\") " Oct 31 00:08:54.143775 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:54.143584 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2cbee7072e17e752090d8a76ddbb931486f62ce02a86fb77ec088ce1899fa412"} err="failed to get container status \"2cbee7072e17e752090d8a76ddbb931486f62ce02a86fb77ec088ce1899fa412\": rpc error: code = NotFound desc = could not find container \"2cbee7072e17e752090d8a76ddbb931486f62ce02a86fb77ec088ce1899fa412\": container with ID starting with 2cbee7072e17e752090d8a76ddbb931486f62ce02a86fb77ec088ce1899fa412 not found: ID does not exist" Oct 31 00:08:54.144000 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:54.143607 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7194267-cfef-4252-ae16-a194eccd2a1f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7194267-cfef-4252-ae16-a194eccd2a1f-workload-socket\") pod \"b7194267-cfef-4252-ae16-a194eccd2a1f\" (UID: \"b7194267-cfef-4252-ae16-a194eccd2a1f\") " Oct 31 00:08:54.144113 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:54.144078 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7194267-cfef-4252-ae16-a194eccd2a1f-kube-api-access-fp7p6\" (UniqueName: \"kubernetes.io/projected/b7194267-cfef-4252-ae16-a194eccd2a1f-kube-api-access-fp7p6\") pod \"b7194267-cfef-4252-ae16-a194eccd2a1f\" (UID: \"b7194267-cfef-4252-ae16-a194eccd2a1f\") " Oct 31 00:08:54.144113 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:54.143687 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7194267-cfef-4252-ae16-a194eccd2a1f-istio-data" pod "b7194267-cfef-4252-ae16-a194eccd2a1f" (UID: "b7194267-cfef-4252-ae16-a194eccd2a1f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:08:54.144418 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:54.143813 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b317f84-83ac-49aa-921e-0242889f2b4f-workload-certs" pod "2b317f84-83ac-49aa-921e-0242889f2b4f" (UID: "2b317f84-83ac-49aa-921e-0242889f2b4f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:08:54.144418 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:54.144082 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b317f84-83ac-49aa-921e-0242889f2b4f-workload-socket" pod "2b317f84-83ac-49aa-921e-0242889f2b4f" (UID: "2b317f84-83ac-49aa-921e-0242889f2b4f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:08:54.144418 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:54.144169 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7194267-cfef-4252-ae16-a194eccd2a1f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7194267-cfef-4252-ae16-a194eccd2a1f-workload-certs\") pod \"b7194267-cfef-4252-ae16-a194eccd2a1f\" (UID: \"b7194267-cfef-4252-ae16-a194eccd2a1f\") " Oct 31 00:08:54.144418 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:54.144200 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b317f84-83ac-49aa-921e-0242889f2b4f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b317f84-83ac-49aa-921e-0242889f2b4f-istio-envoy\") pod \"2b317f84-83ac-49aa-921e-0242889f2b4f\" (UID: \"2b317f84-83ac-49aa-921e-0242889f2b4f\") " Oct 31 00:08:54.144639 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:54.144504 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b317f84-83ac-49aa-921e-0242889f2b4f-istio-data" pod "2b317f84-83ac-49aa-921e-0242889f2b4f" (UID: "2b317f84-83ac-49aa-921e-0242889f2b4f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:08:54.144639 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:54.143614 4927 scope.go:122] "RemoveContainer" containerID="c4717a5f3b503ab1937f5d3e01e2b12f6a6927be6491842a06b68ade45ca38da" Oct 31 00:08:54.144893 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:54.144869 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7194267-cfef-4252-ae16-a194eccd2a1f-workload-certs" pod "b7194267-cfef-4252-ae16-a194eccd2a1f" (UID: "b7194267-cfef-4252-ae16-a194eccd2a1f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:08:54.145034 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:54.144970 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7194267-cfef-4252-ae16-a194eccd2a1f-workload-socket" pod "b7194267-cfef-4252-ae16-a194eccd2a1f" (UID: "b7194267-cfef-4252-ae16-a194eccd2a1f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:08:54.145276 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:54.145247 4927 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b317f84-83ac-49aa-921e-0242889f2b4f-istiod-ca-cert\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:08:54.145418 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:54.145281 4927 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7194267-cfef-4252-ae16-a194eccd2a1f-credential-socket\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:08:54.145418 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:54.145298 4927 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b317f84-83ac-49aa-921e-0242889f2b4f-credential-socket\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:08:54.145418 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:54.145313 4927 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7194267-cfef-4252-ae16-a194eccd2a1f-istio-data\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:08:54.145418 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:54.145343 4927 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b317f84-83ac-49aa-921e-0242889f2b4f-workload-socket\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:08:54.145418 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:54.145360 4927 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b317f84-83ac-49aa-921e-0242889f2b4f-workload-certs\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:08:54.145418 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:54.145374 4927 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b317f84-83ac-49aa-921e-0242889f2b4f-istio-data\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:08:54.145418 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:54.145391 4927 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7194267-cfef-4252-ae16-a194eccd2a1f-workload-socket\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:08:54.145418 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:54.145420 4927 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7194267-cfef-4252-ae16-a194eccd2a1f-workload-certs\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:08:54.145846 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:54.145436 4927 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b317f84-83ac-49aa-921e-0242889f2b4f-istio-token\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:08:54.145846 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:54.145450 4927 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7194267-cfef-4252-ae16-a194eccd2a1f-istiod-ca-cert\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:08:54.145846 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:54.145464 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f7xcn\" (UniqueName: \"kubernetes.io/projected/2b317f84-83ac-49aa-921e-0242889f2b4f-kube-api-access-f7xcn\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:08:54.147472 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:54.147446 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2b317f84-83ac-49aa-921e-0242889f2b4f-istio-podinfo" pod "2b317f84-83ac-49aa-921e-0242889f2b4f" (UID: "2b317f84-83ac-49aa-921e-0242889f2b4f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:08:54.147804 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:54.147780 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7194267-cfef-4252-ae16-a194eccd2a1f-istio-envoy" pod "b7194267-cfef-4252-ae16-a194eccd2a1f" (UID: "b7194267-cfef-4252-ae16-a194eccd2a1f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:08:54.151525 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:54.151497 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b7194267-cfef-4252-ae16-a194eccd2a1f-istio-podinfo" pod "b7194267-cfef-4252-ae16-a194eccd2a1f" (UID: "b7194267-cfef-4252-ae16-a194eccd2a1f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:08:54.153580 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:54.153557 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7194267-cfef-4252-ae16-a194eccd2a1f-kube-api-access-fp7p6" pod "b7194267-cfef-4252-ae16-a194eccd2a1f" (UID: "b7194267-cfef-4252-ae16-a194eccd2a1f"). InnerVolumeSpecName "kube-api-access-fp7p6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:08:54.154295 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:54.154262 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b317f84-83ac-49aa-921e-0242889f2b4f-istio-envoy" pod "2b317f84-83ac-49aa-921e-0242889f2b4f" (UID: "2b317f84-83ac-49aa-921e-0242889f2b4f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:08:54.155127 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:54.155099 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7194267-cfef-4252-ae16-a194eccd2a1f-istio-token" pod "b7194267-cfef-4252-ae16-a194eccd2a1f" (UID: "b7194267-cfef-4252-ae16-a194eccd2a1f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:08:54.164656 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:54.164629 4927 scope.go:122] "RemoveContainer" containerID="c4717a5f3b503ab1937f5d3e01e2b12f6a6927be6491842a06b68ade45ca38da" Oct 31 00:08:54.165036 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:08:54.165010 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4717a5f3b503ab1937f5d3e01e2b12f6a6927be6491842a06b68ade45ca38da\": container with ID starting with c4717a5f3b503ab1937f5d3e01e2b12f6a6927be6491842a06b68ade45ca38da not found: ID does not exist" containerID="c4717a5f3b503ab1937f5d3e01e2b12f6a6927be6491842a06b68ade45ca38da" Oct 31 00:08:54.165172 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:54.165049 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4717a5f3b503ab1937f5d3e01e2b12f6a6927be6491842a06b68ade45ca38da"} err="failed to get container status \"c4717a5f3b503ab1937f5d3e01e2b12f6a6927be6491842a06b68ade45ca38da\": rpc error: code = NotFound desc = could not find container \"c4717a5f3b503ab1937f5d3e01e2b12f6a6927be6491842a06b68ade45ca38da\": container with ID starting with c4717a5f3b503ab1937f5d3e01e2b12f6a6927be6491842a06b68ade45ca38da not found: ID does not exist" Oct 31 00:08:54.165172 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:54.165077 4927 scope.go:122] "RemoveContainer" containerID="b2188dcfc96b95e4deeaf72da6a224c424e584af088e805d70914d27730aae5e" Oct 31 00:08:54.215738 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:54.215706 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-94tkp"] Oct 31 00:08:54.223563 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:54.223529 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-94tkp"] Oct 31 00:08:54.246189 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:54.246161 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/45977fb0-7f5d-4f55-9530-112d60f269ab-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45977fb0-7f5d-4f55-9530-112d60f269ab-istio-podinfo\") pod \"45977fb0-7f5d-4f55-9530-112d60f269ab\" (UID: \"45977fb0-7f5d-4f55-9530-112d60f269ab\") " Oct 31 00:08:54.246368 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:54.246228 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/334f41bf-6d81-474d-bddc-d38c3492d586-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/334f41bf-6d81-474d-bddc-d38c3492d586-istio-podinfo\") pod \"334f41bf-6d81-474d-bddc-d38c3492d586\" (UID: \"334f41bf-6d81-474d-bddc-d38c3492d586\") " Oct 31 00:08:54.246368 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:54.246283 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45977fb0-7f5d-4f55-9530-112d60f269ab-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45977fb0-7f5d-4f55-9530-112d60f269ab-workload-socket\") pod \"45977fb0-7f5d-4f55-9530-112d60f269ab\" (UID: \"45977fb0-7f5d-4f55-9530-112d60f269ab\") " Oct 31 00:08:54.246368 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:54.246316 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45977fb0-7f5d-4f55-9530-112d60f269ab-istio-token\" (UniqueName: \"kubernetes.io/projected/45977fb0-7f5d-4f55-9530-112d60f269ab-istio-token\") pod \"45977fb0-7f5d-4f55-9530-112d60f269ab\" (UID: \"45977fb0-7f5d-4f55-9530-112d60f269ab\") " Oct 31 00:08:54.247053 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:54.246419 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45977fb0-7f5d-4f55-9530-112d60f269ab-kube-api-access-w56pf\" (UniqueName: \"kubernetes.io/projected/45977fb0-7f5d-4f55-9530-112d60f269ab-kube-api-access-w56pf\") pod \"45977fb0-7f5d-4f55-9530-112d60f269ab\" (UID: \"45977fb0-7f5d-4f55-9530-112d60f269ab\") " Oct 31 00:08:54.247053 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:54.246482 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45977fb0-7f5d-4f55-9530-112d60f269ab-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45977fb0-7f5d-4f55-9530-112d60f269ab-istio-data\") pod \"45977fb0-7f5d-4f55-9530-112d60f269ab\" (UID: \"45977fb0-7f5d-4f55-9530-112d60f269ab\") " Oct 31 00:08:54.247053 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:54.246526 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/334f41bf-6d81-474d-bddc-d38c3492d586-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/334f41bf-6d81-474d-bddc-d38c3492d586-istiod-ca-cert\") pod \"334f41bf-6d81-474d-bddc-d38c3492d586\" (UID: \"334f41bf-6d81-474d-bddc-d38c3492d586\") " Oct 31 00:08:54.247053 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:54.246611 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/334f41bf-6d81-474d-bddc-d38c3492d586-kube-api-access-x24zl\" (UniqueName: \"kubernetes.io/projected/334f41bf-6d81-474d-bddc-d38c3492d586-kube-api-access-x24zl\") pod \"334f41bf-6d81-474d-bddc-d38c3492d586\" (UID: \"334f41bf-6d81-474d-bddc-d38c3492d586\") " Oct 31 00:08:54.247053 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:54.246655 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45977fb0-7f5d-4f55-9530-112d60f269ab-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45977fb0-7f5d-4f55-9530-112d60f269ab-workload-certs\") pod \"45977fb0-7f5d-4f55-9530-112d60f269ab\" (UID: \"45977fb0-7f5d-4f55-9530-112d60f269ab\") " Oct 31 00:08:54.247053 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:54.246687 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/334f41bf-6d81-474d-bddc-d38c3492d586-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/334f41bf-6d81-474d-bddc-d38c3492d586-istio-envoy\") pod \"334f41bf-6d81-474d-bddc-d38c3492d586\" (UID: \"334f41bf-6d81-474d-bddc-d38c3492d586\") " Oct 31 00:08:54.247053 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:54.246751 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45977fb0-7f5d-4f55-9530-112d60f269ab-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45977fb0-7f5d-4f55-9530-112d60f269ab-istio-envoy\") pod \"45977fb0-7f5d-4f55-9530-112d60f269ab\" (UID: \"45977fb0-7f5d-4f55-9530-112d60f269ab\") " Oct 31 00:08:54.247053 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:54.246788 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/334f41bf-6d81-474d-bddc-d38c3492d586-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/334f41bf-6d81-474d-bddc-d38c3492d586-workload-certs\") pod \"334f41bf-6d81-474d-bddc-d38c3492d586\" (UID: \"334f41bf-6d81-474d-bddc-d38c3492d586\") " Oct 31 00:08:54.247053 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:54.246904 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/334f41bf-6d81-474d-bddc-d38c3492d586-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/334f41bf-6d81-474d-bddc-d38c3492d586-credential-socket\") pod \"334f41bf-6d81-474d-bddc-d38c3492d586\" (UID: \"334f41bf-6d81-474d-bddc-d38c3492d586\") " Oct 31 00:08:54.247053 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:54.246948 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45977fb0-7f5d-4f55-9530-112d60f269ab-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45977fb0-7f5d-4f55-9530-112d60f269ab-credential-socket\") pod \"45977fb0-7f5d-4f55-9530-112d60f269ab\" (UID: \"45977fb0-7f5d-4f55-9530-112d60f269ab\") " Oct 31 00:08:54.247053 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:54.247029 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/45977fb0-7f5d-4f55-9530-112d60f269ab-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45977fb0-7f5d-4f55-9530-112d60f269ab-istiod-ca-cert\") pod \"45977fb0-7f5d-4f55-9530-112d60f269ab\" (UID: \"45977fb0-7f5d-4f55-9530-112d60f269ab\") " Oct 31 00:08:54.247651 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:54.247120 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/334f41bf-6d81-474d-bddc-d38c3492d586-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/334f41bf-6d81-474d-bddc-d38c3492d586-workload-socket\") pod \"334f41bf-6d81-474d-bddc-d38c3492d586\" (UID: \"334f41bf-6d81-474d-bddc-d38c3492d586\") " Oct 31 00:08:54.247651 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:54.247173 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/334f41bf-6d81-474d-bddc-d38c3492d586-istio-token\" (UniqueName: \"kubernetes.io/projected/334f41bf-6d81-474d-bddc-d38c3492d586-istio-token\") pod \"334f41bf-6d81-474d-bddc-d38c3492d586\" (UID: \"334f41bf-6d81-474d-bddc-d38c3492d586\") " Oct 31 00:08:54.247651 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:54.247213 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/334f41bf-6d81-474d-bddc-d38c3492d586-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/334f41bf-6d81-474d-bddc-d38c3492d586-istio-data\") pod \"334f41bf-6d81-474d-bddc-d38c3492d586\" (UID: \"334f41bf-6d81-474d-bddc-d38c3492d586\") " Oct 31 00:08:54.250127 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:54.248074 4927 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b317f84-83ac-49aa-921e-0242889f2b4f-istio-podinfo\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:08:54.250127 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:54.248099 4927 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7194267-cfef-4252-ae16-a194eccd2a1f-istio-token\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:08:54.250127 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:54.248114 4927 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7194267-cfef-4252-ae16-a194eccd2a1f-istio-podinfo\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:08:54.250127 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:54.248130 4927 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7194267-cfef-4252-ae16-a194eccd2a1f-istio-envoy\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:08:54.250127 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:54.248145 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fp7p6\" (UniqueName: \"kubernetes.io/projected/b7194267-cfef-4252-ae16-a194eccd2a1f-kube-api-access-fp7p6\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:08:54.250127 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:54.248161 4927 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b317f84-83ac-49aa-921e-0242889f2b4f-istio-envoy\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:08:54.250127 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:54.248650 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45977fb0-7f5d-4f55-9530-112d60f269ab-istiod-ca-cert" pod "45977fb0-7f5d-4f55-9530-112d60f269ab" (UID: "45977fb0-7f5d-4f55-9530-112d60f269ab"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:08:54.250127 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:54.248765 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/334f41bf-6d81-474d-bddc-d38c3492d586-istio-data" pod "334f41bf-6d81-474d-bddc-d38c3492d586" (UID: "334f41bf-6d81-474d-bddc-d38c3492d586"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:08:54.250127 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:54.248778 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/334f41bf-6d81-474d-bddc-d38c3492d586-workload-socket" pod "334f41bf-6d81-474d-bddc-d38c3492d586" (UID: "334f41bf-6d81-474d-bddc-d38c3492d586"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:08:54.250127 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:54.249055 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45977fb0-7f5d-4f55-9530-112d60f269ab-istio-data" pod "45977fb0-7f5d-4f55-9530-112d60f269ab" (UID: "45977fb0-7f5d-4f55-9530-112d60f269ab"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:08:54.250127 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:54.249282 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45977fb0-7f5d-4f55-9530-112d60f269ab-workload-socket" pod "45977fb0-7f5d-4f55-9530-112d60f269ab" (UID: "45977fb0-7f5d-4f55-9530-112d60f269ab"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:08:54.254203 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:54.251229 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/334f41bf-6d81-474d-bddc-d38c3492d586-istiod-ca-cert" pod "334f41bf-6d81-474d-bddc-d38c3492d586" (UID: "334f41bf-6d81-474d-bddc-d38c3492d586"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:08:54.254203 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:54.251721 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45977fb0-7f5d-4f55-9530-112d60f269ab-workload-certs" pod "45977fb0-7f5d-4f55-9530-112d60f269ab" (UID: "45977fb0-7f5d-4f55-9530-112d60f269ab"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:08:54.254203 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:54.251871 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/334f41bf-6d81-474d-bddc-d38c3492d586-kube-api-access-x24zl" pod "334f41bf-6d81-474d-bddc-d38c3492d586" (UID: "334f41bf-6d81-474d-bddc-d38c3492d586"). InnerVolumeSpecName "kube-api-access-x24zl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:08:54.254203 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:54.252087 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/334f41bf-6d81-474d-bddc-d38c3492d586-credential-socket" pod "334f41bf-6d81-474d-bddc-d38c3492d586" (UID: "334f41bf-6d81-474d-bddc-d38c3492d586"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:08:54.254203 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:54.252248 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/334f41bf-6d81-474d-bddc-d38c3492d586-workload-certs" pod "334f41bf-6d81-474d-bddc-d38c3492d586" (UID: "334f41bf-6d81-474d-bddc-d38c3492d586"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:08:54.254203 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:54.252611 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45977fb0-7f5d-4f55-9530-112d60f269ab-credential-socket" pod "45977fb0-7f5d-4f55-9530-112d60f269ab" (UID: "45977fb0-7f5d-4f55-9530-112d60f269ab"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:08:54.254203 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:54.253451 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/334f41bf-6d81-474d-bddc-d38c3492d586-istio-envoy" pod "334f41bf-6d81-474d-bddc-d38c3492d586" (UID: "334f41bf-6d81-474d-bddc-d38c3492d586"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:08:54.254203 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:54.253556 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/45977fb0-7f5d-4f55-9530-112d60f269ab-istio-podinfo" pod "45977fb0-7f5d-4f55-9530-112d60f269ab" (UID: "45977fb0-7f5d-4f55-9530-112d60f269ab"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:08:54.254203 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:54.253676 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45977fb0-7f5d-4f55-9530-112d60f269ab-istio-token" pod "45977fb0-7f5d-4f55-9530-112d60f269ab" (UID: "45977fb0-7f5d-4f55-9530-112d60f269ab"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:08:54.254947 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:54.254585 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45977fb0-7f5d-4f55-9530-112d60f269ab-kube-api-access-w56pf" pod "45977fb0-7f5d-4f55-9530-112d60f269ab" (UID: "45977fb0-7f5d-4f55-9530-112d60f269ab"). InnerVolumeSpecName "kube-api-access-w56pf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:08:54.255139 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:54.255106 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/334f41bf-6d81-474d-bddc-d38c3492d586-istio-token" pod "334f41bf-6d81-474d-bddc-d38c3492d586" (UID: "334f41bf-6d81-474d-bddc-d38c3492d586"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:08:54.255872 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:54.255847 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45977fb0-7f5d-4f55-9530-112d60f269ab-istio-envoy" pod "45977fb0-7f5d-4f55-9530-112d60f269ab" (UID: "45977fb0-7f5d-4f55-9530-112d60f269ab"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:08:54.256018 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:54.255994 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/334f41bf-6d81-474d-bddc-d38c3492d586-istio-podinfo" pod "334f41bf-6d81-474d-bddc-d38c3492d586" (UID: "334f41bf-6d81-474d-bddc-d38c3492d586"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:08:54.257104 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:54.257024 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-jdcvx" Oct 31 00:08:54.263101 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:54.263076 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-kzqr4" Oct 31 00:08:54.275825 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:54.275761 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-tgc8q" Oct 31 00:08:54.349792 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:54.349182 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e8b64ac-0b2a-40aa-ab07-25ec13005517-kube-api-access-g4cqr\" (UniqueName: \"kubernetes.io/projected/3e8b64ac-0b2a-40aa-ab07-25ec13005517-kube-api-access-g4cqr\") pod \"3e8b64ac-0b2a-40aa-ab07-25ec13005517\" (UID: \"3e8b64ac-0b2a-40aa-ab07-25ec13005517\") " Oct 31 00:08:54.349792 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:54.349254 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a520a0b5-deab-4825-9b36-329c399754f6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a520a0b5-deab-4825-9b36-329c399754f6-workload-certs\") pod \"a520a0b5-deab-4825-9b36-329c399754f6\" (UID: \"a520a0b5-deab-4825-9b36-329c399754f6\") " Oct 31 00:08:54.349792 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:54.349306 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a520a0b5-deab-4825-9b36-329c399754f6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a520a0b5-deab-4825-9b36-329c399754f6-credential-socket\") pod \"a520a0b5-deab-4825-9b36-329c399754f6\" (UID: \"a520a0b5-deab-4825-9b36-329c399754f6\") " Oct 31 00:08:54.349792 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:54.349358 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3e8b64ac-0b2a-40aa-ab07-25ec13005517-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e8b64ac-0b2a-40aa-ab07-25ec13005517-istiod-ca-cert\") pod \"3e8b64ac-0b2a-40aa-ab07-25ec13005517\" (UID: \"3e8b64ac-0b2a-40aa-ab07-25ec13005517\") " Oct 31 00:08:54.349792 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:54.349388 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e8b64ac-0b2a-40aa-ab07-25ec13005517-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e8b64ac-0b2a-40aa-ab07-25ec13005517-credential-socket\") pod \"3e8b64ac-0b2a-40aa-ab07-25ec13005517\" (UID: \"3e8b64ac-0b2a-40aa-ab07-25ec13005517\") " Oct 31 00:08:54.349792 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:54.349423 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e8b64ac-0b2a-40aa-ab07-25ec13005517-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e8b64ac-0b2a-40aa-ab07-25ec13005517-istio-data\") pod \"3e8b64ac-0b2a-40aa-ab07-25ec13005517\" (UID: \"3e8b64ac-0b2a-40aa-ab07-25ec13005517\") " Oct 31 00:08:54.349792 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:54.349492 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e8b64ac-0b2a-40aa-ab07-25ec13005517-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e8b64ac-0b2a-40aa-ab07-25ec13005517-istio-envoy\") pod \"3e8b64ac-0b2a-40aa-ab07-25ec13005517\" (UID: \"3e8b64ac-0b2a-40aa-ab07-25ec13005517\") " Oct 31 00:08:54.349792 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:54.349518 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a550117e-ad41-4e7c-998d-910a8d13d617-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a550117e-ad41-4e7c-998d-910a8d13d617-credential-socket\") pod \"a550117e-ad41-4e7c-998d-910a8d13d617\" (UID: \"a550117e-ad41-4e7c-998d-910a8d13d617\") " Oct 31 00:08:54.349792 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:54.349587 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a550117e-ad41-4e7c-998d-910a8d13d617-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a550117e-ad41-4e7c-998d-910a8d13d617-istio-podinfo\") pod \"a550117e-ad41-4e7c-998d-910a8d13d617\" (UID: \"a550117e-ad41-4e7c-998d-910a8d13d617\") " Oct 31 00:08:54.349792 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:54.349632 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a520a0b5-deab-4825-9b36-329c399754f6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a520a0b5-deab-4825-9b36-329c399754f6-istiod-ca-cert\") pod \"a520a0b5-deab-4825-9b36-329c399754f6\" (UID: \"a520a0b5-deab-4825-9b36-329c399754f6\") " Oct 31 00:08:54.349792 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:54.349739 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a550117e-ad41-4e7c-998d-910a8d13d617-istio-token\" (UniqueName: \"kubernetes.io/projected/a550117e-ad41-4e7c-998d-910a8d13d617-istio-token\") pod \"a550117e-ad41-4e7c-998d-910a8d13d617\" (UID: \"a550117e-ad41-4e7c-998d-910a8d13d617\") " Oct 31 00:08:54.349792 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:54.349793 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a550117e-ad41-4e7c-998d-910a8d13d617-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a550117e-ad41-4e7c-998d-910a8d13d617-istiod-ca-cert\") pod \"a550117e-ad41-4e7c-998d-910a8d13d617\" (UID: \"a550117e-ad41-4e7c-998d-910a8d13d617\") " Oct 31 00:08:54.350569 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:54.349849 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a520a0b5-deab-4825-9b36-329c399754f6-istio-token\" (UniqueName: \"kubernetes.io/projected/a520a0b5-deab-4825-9b36-329c399754f6-istio-token\") pod \"a520a0b5-deab-4825-9b36-329c399754f6\" (UID: \"a520a0b5-deab-4825-9b36-329c399754f6\") " Oct 31 00:08:54.350569 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:54.349907 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a550117e-ad41-4e7c-998d-910a8d13d617-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a550117e-ad41-4e7c-998d-910a8d13d617-istio-envoy\") pod \"a550117e-ad41-4e7c-998d-910a8d13d617\" (UID: \"a550117e-ad41-4e7c-998d-910a8d13d617\") " Oct 31 00:08:54.350569 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:54.349951 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3e8b64ac-0b2a-40aa-ab07-25ec13005517-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e8b64ac-0b2a-40aa-ab07-25ec13005517-istio-podinfo\") pod \"3e8b64ac-0b2a-40aa-ab07-25ec13005517\" (UID: \"3e8b64ac-0b2a-40aa-ab07-25ec13005517\") " Oct 31 00:08:54.350569 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:54.350008 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e8b64ac-0b2a-40aa-ab07-25ec13005517-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e8b64ac-0b2a-40aa-ab07-25ec13005517-workload-socket\") pod \"3e8b64ac-0b2a-40aa-ab07-25ec13005517\" (UID: \"3e8b64ac-0b2a-40aa-ab07-25ec13005517\") " Oct 31 00:08:54.350569 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:54.350037 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e8b64ac-0b2a-40aa-ab07-25ec13005517-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e8b64ac-0b2a-40aa-ab07-25ec13005517-workload-certs\") pod \"3e8b64ac-0b2a-40aa-ab07-25ec13005517\" (UID: \"3e8b64ac-0b2a-40aa-ab07-25ec13005517\") " Oct 31 00:08:54.350569 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:54.350069 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a520a0b5-deab-4825-9b36-329c399754f6-kube-api-access-g4cgf\" (UniqueName: \"kubernetes.io/projected/a520a0b5-deab-4825-9b36-329c399754f6-kube-api-access-g4cgf\") pod \"a520a0b5-deab-4825-9b36-329c399754f6\" (UID: \"a520a0b5-deab-4825-9b36-329c399754f6\") " Oct 31 00:08:54.350569 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:54.350110 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a520a0b5-deab-4825-9b36-329c399754f6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a520a0b5-deab-4825-9b36-329c399754f6-istio-data\") pod \"a520a0b5-deab-4825-9b36-329c399754f6\" (UID: \"a520a0b5-deab-4825-9b36-329c399754f6\") " Oct 31 00:08:54.350946 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:54.350626 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e8b64ac-0b2a-40aa-ab07-25ec13005517-workload-socket" pod "3e8b64ac-0b2a-40aa-ab07-25ec13005517" (UID: "3e8b64ac-0b2a-40aa-ab07-25ec13005517"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:08:54.352105 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:54.351195 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e8b64ac-0b2a-40aa-ab07-25ec13005517-workload-certs" pod "3e8b64ac-0b2a-40aa-ab07-25ec13005517" (UID: "3e8b64ac-0b2a-40aa-ab07-25ec13005517"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:08:54.352105 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:54.351680 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e8b64ac-0b2a-40aa-ab07-25ec13005517-istiod-ca-cert" pod "3e8b64ac-0b2a-40aa-ab07-25ec13005517" (UID: "3e8b64ac-0b2a-40aa-ab07-25ec13005517"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:08:54.355555 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:54.354983 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e8b64ac-0b2a-40aa-ab07-25ec13005517-kube-api-access-g4cqr" pod "3e8b64ac-0b2a-40aa-ab07-25ec13005517" (UID: "3e8b64ac-0b2a-40aa-ab07-25ec13005517"). InnerVolumeSpecName "kube-api-access-g4cqr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:08:54.355555 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:54.355169 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a520a0b5-deab-4825-9b36-329c399754f6-workload-certs" pod "a520a0b5-deab-4825-9b36-329c399754f6" (UID: "a520a0b5-deab-4825-9b36-329c399754f6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:08:54.355555 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:54.355291 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a520a0b5-deab-4825-9b36-329c399754f6-credential-socket" pod "a520a0b5-deab-4825-9b36-329c399754f6" (UID: "a520a0b5-deab-4825-9b36-329c399754f6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:08:54.355555 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:54.355447 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e8b64ac-0b2a-40aa-ab07-25ec13005517-istio-data" pod "3e8b64ac-0b2a-40aa-ab07-25ec13005517" (UID: "3e8b64ac-0b2a-40aa-ab07-25ec13005517"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:08:54.355834 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:54.355583 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e8b64ac-0b2a-40aa-ab07-25ec13005517-credential-socket" pod "3e8b64ac-0b2a-40aa-ab07-25ec13005517" (UID: "3e8b64ac-0b2a-40aa-ab07-25ec13005517"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:08:54.355834 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:54.355584 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a520a0b5-deab-4825-9b36-329c399754f6-istio-data" pod "a520a0b5-deab-4825-9b36-329c399754f6" (UID: "a520a0b5-deab-4825-9b36-329c399754f6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:08:54.355834 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:54.355621 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a550117e-ad41-4e7c-998d-910a8d13d617-istiod-ca-cert" pod "a550117e-ad41-4e7c-998d-910a8d13d617" (UID: "a550117e-ad41-4e7c-998d-910a8d13d617"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:08:54.356019 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:54.355689 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a550117e-ad41-4e7c-998d-910a8d13d617-kube-api-access-kz484\" (UniqueName: \"kubernetes.io/projected/a550117e-ad41-4e7c-998d-910a8d13d617-kube-api-access-kz484\") pod \"a550117e-ad41-4e7c-998d-910a8d13d617\" (UID: \"a550117e-ad41-4e7c-998d-910a8d13d617\") " Oct 31 00:08:54.356082 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:54.356051 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a550117e-ad41-4e7c-998d-910a8d13d617-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a550117e-ad41-4e7c-998d-910a8d13d617-workload-certs\") pod \"a550117e-ad41-4e7c-998d-910a8d13d617\" (UID: \"a550117e-ad41-4e7c-998d-910a8d13d617\") " Oct 31 00:08:54.357146 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:54.356662 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a520a0b5-deab-4825-9b36-329c399754f6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a520a0b5-deab-4825-9b36-329c399754f6-istio-envoy\") pod \"a520a0b5-deab-4825-9b36-329c399754f6\" (UID: \"a520a0b5-deab-4825-9b36-329c399754f6\") " Oct 31 00:08:54.357146 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:54.356724 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a550117e-ad41-4e7c-998d-910a8d13d617-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a550117e-ad41-4e7c-998d-910a8d13d617-workload-socket\") pod \"a550117e-ad41-4e7c-998d-910a8d13d617\" (UID: \"a550117e-ad41-4e7c-998d-910a8d13d617\") " Oct 31 00:08:54.357146 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:54.356764 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a520a0b5-deab-4825-9b36-329c399754f6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a520a0b5-deab-4825-9b36-329c399754f6-istio-podinfo\") pod \"a520a0b5-deab-4825-9b36-329c399754f6\" (UID: \"a520a0b5-deab-4825-9b36-329c399754f6\") " Oct 31 00:08:54.357146 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:54.356791 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a520a0b5-deab-4825-9b36-329c399754f6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a520a0b5-deab-4825-9b36-329c399754f6-workload-socket\") pod \"a520a0b5-deab-4825-9b36-329c399754f6\" (UID: \"a520a0b5-deab-4825-9b36-329c399754f6\") " Oct 31 00:08:54.357146 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:54.356869 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a550117e-ad41-4e7c-998d-910a8d13d617-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a550117e-ad41-4e7c-998d-910a8d13d617-istio-data\") pod \"a550117e-ad41-4e7c-998d-910a8d13d617\" (UID: \"a550117e-ad41-4e7c-998d-910a8d13d617\") " Oct 31 00:08:54.357146 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:54.356921 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e8b64ac-0b2a-40aa-ab07-25ec13005517-istio-token\" (UniqueName: \"kubernetes.io/projected/3e8b64ac-0b2a-40aa-ab07-25ec13005517-istio-token\") pod \"3e8b64ac-0b2a-40aa-ab07-25ec13005517\" (UID: \"3e8b64ac-0b2a-40aa-ab07-25ec13005517\") " Oct 31 00:08:54.358611 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:54.357817 4927 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45977fb0-7f5d-4f55-9530-112d60f269ab-istio-envoy\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:08:54.358611 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:54.357849 4927 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/334f41bf-6d81-474d-bddc-d38c3492d586-workload-certs\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:08:54.358611 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:54.357865 4927 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a550117e-ad41-4e7c-998d-910a8d13d617-istiod-ca-cert\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:08:54.358611 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:54.357879 4927 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/334f41bf-6d81-474d-bddc-d38c3492d586-credential-socket\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:08:54.358611 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:54.357893 4927 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45977fb0-7f5d-4f55-9530-112d60f269ab-credential-socket\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:08:54.358611 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:54.357907 4927 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45977fb0-7f5d-4f55-9530-112d60f269ab-istiod-ca-cert\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:08:54.358611 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:54.357921 4927 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e8b64ac-0b2a-40aa-ab07-25ec13005517-workload-socket\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:08:54.358611 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:54.357933 4927 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e8b64ac-0b2a-40aa-ab07-25ec13005517-workload-certs\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:08:54.358611 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:54.357947 4927 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a520a0b5-deab-4825-9b36-329c399754f6-istio-data\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:08:54.358611 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:54.357960 4927 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/334f41bf-6d81-474d-bddc-d38c3492d586-workload-socket\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:08:54.358611 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:54.357979 4927 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/334f41bf-6d81-474d-bddc-d38c3492d586-istio-token\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:08:54.358611 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:54.357992 4927 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/334f41bf-6d81-474d-bddc-d38c3492d586-istio-data\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:08:54.358611 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:54.358006 4927 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45977fb0-7f5d-4f55-9530-112d60f269ab-istio-podinfo\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:08:54.358611 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:54.358020 4927 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/334f41bf-6d81-474d-bddc-d38c3492d586-istio-podinfo\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:08:54.358611 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:54.358035 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g4cqr\" (UniqueName: \"kubernetes.io/projected/3e8b64ac-0b2a-40aa-ab07-25ec13005517-kube-api-access-g4cqr\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:08:54.358611 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:54.358050 4927 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45977fb0-7f5d-4f55-9530-112d60f269ab-workload-socket\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:08:54.358611 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:54.358064 4927 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45977fb0-7f5d-4f55-9530-112d60f269ab-istio-token\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:08:54.358611 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:54.358078 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w56pf\" (UniqueName: \"kubernetes.io/projected/45977fb0-7f5d-4f55-9530-112d60f269ab-kube-api-access-w56pf\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:08:54.358611 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:54.358092 4927 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a520a0b5-deab-4825-9b36-329c399754f6-workload-certs\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:08:54.358611 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:54.358108 4927 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a520a0b5-deab-4825-9b36-329c399754f6-credential-socket\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:08:54.358611 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:54.358124 4927 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e8b64ac-0b2a-40aa-ab07-25ec13005517-istiod-ca-cert\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:08:54.358611 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:54.358138 4927 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e8b64ac-0b2a-40aa-ab07-25ec13005517-credential-socket\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:08:54.358611 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:54.358153 4927 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e8b64ac-0b2a-40aa-ab07-25ec13005517-istio-data\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:08:54.358611 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:54.358166 4927 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45977fb0-7f5d-4f55-9530-112d60f269ab-istio-data\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:08:54.358611 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:54.358181 4927 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/334f41bf-6d81-474d-bddc-d38c3492d586-istiod-ca-cert\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:08:54.358611 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:54.358198 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x24zl\" (UniqueName: \"kubernetes.io/projected/334f41bf-6d81-474d-bddc-d38c3492d586-kube-api-access-x24zl\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:08:54.358611 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:54.358212 4927 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45977fb0-7f5d-4f55-9530-112d60f269ab-workload-certs\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:08:54.358611 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:54.358229 4927 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/334f41bf-6d81-474d-bddc-d38c3492d586-istio-envoy\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:08:54.361073 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:54.355849 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a550117e-ad41-4e7c-998d-910a8d13d617-credential-socket" pod "a550117e-ad41-4e7c-998d-910a8d13d617" (UID: "a550117e-ad41-4e7c-998d-910a8d13d617"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:08:54.361073 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:54.359679 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e8b64ac-0b2a-40aa-ab07-25ec13005517-istio-envoy" pod "3e8b64ac-0b2a-40aa-ab07-25ec13005517" (UID: "3e8b64ac-0b2a-40aa-ab07-25ec13005517"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:08:54.361073 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:54.360401 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a550117e-ad41-4e7c-998d-910a8d13d617-istio-token" pod "a550117e-ad41-4e7c-998d-910a8d13d617" (UID: "a550117e-ad41-4e7c-998d-910a8d13d617"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:08:54.361073 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:54.360721 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a520a0b5-deab-4825-9b36-329c399754f6-istiod-ca-cert" pod "a520a0b5-deab-4825-9b36-329c399754f6" (UID: "a520a0b5-deab-4825-9b36-329c399754f6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:08:54.361073 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:54.360778 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3e8b64ac-0b2a-40aa-ab07-25ec13005517-istio-podinfo" pod "3e8b64ac-0b2a-40aa-ab07-25ec13005517" (UID: "3e8b64ac-0b2a-40aa-ab07-25ec13005517"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:08:54.362671 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:54.362382 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a550117e-ad41-4e7c-998d-910a8d13d617-istio-podinfo" pod "a550117e-ad41-4e7c-998d-910a8d13d617" (UID: "a550117e-ad41-4e7c-998d-910a8d13d617"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:08:54.363133 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:54.362914 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a520a0b5-deab-4825-9b36-329c399754f6-istio-token" pod "a520a0b5-deab-4825-9b36-329c399754f6" (UID: "a520a0b5-deab-4825-9b36-329c399754f6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:08:54.363220 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:54.363164 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a550117e-ad41-4e7c-998d-910a8d13d617-workload-socket" pod "a550117e-ad41-4e7c-998d-910a8d13d617" (UID: "a550117e-ad41-4e7c-998d-910a8d13d617"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:08:54.363861 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:54.363299 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a550117e-ad41-4e7c-998d-910a8d13d617-workload-certs" pod "a550117e-ad41-4e7c-998d-910a8d13d617" (UID: "a550117e-ad41-4e7c-998d-910a8d13d617"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:08:54.363861 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:54.363447 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a520a0b5-deab-4825-9b36-329c399754f6-workload-socket" pod "a520a0b5-deab-4825-9b36-329c399754f6" (UID: "a520a0b5-deab-4825-9b36-329c399754f6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:08:54.363861 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:54.363565 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a550117e-ad41-4e7c-998d-910a8d13d617-istio-data" pod "a550117e-ad41-4e7c-998d-910a8d13d617" (UID: "a550117e-ad41-4e7c-998d-910a8d13d617"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:08:54.364206 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:54.364184 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a520a0b5-deab-4825-9b36-329c399754f6-kube-api-access-g4cgf" pod "a520a0b5-deab-4825-9b36-329c399754f6" (UID: "a520a0b5-deab-4825-9b36-329c399754f6"). InnerVolumeSpecName "kube-api-access-g4cgf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:08:54.364286 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:54.364245 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a520a0b5-deab-4825-9b36-329c399754f6-istio-envoy" pod "a520a0b5-deab-4825-9b36-329c399754f6" (UID: "a520a0b5-deab-4825-9b36-329c399754f6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:08:54.364607 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:54.364572 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a550117e-ad41-4e7c-998d-910a8d13d617-istio-envoy" pod "a550117e-ad41-4e7c-998d-910a8d13d617" (UID: "a550117e-ad41-4e7c-998d-910a8d13d617"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:08:54.365699 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:54.365673 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e8b64ac-0b2a-40aa-ab07-25ec13005517-istio-token" pod "3e8b64ac-0b2a-40aa-ab07-25ec13005517" (UID: "3e8b64ac-0b2a-40aa-ab07-25ec13005517"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:08:54.366402 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:54.366053 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a550117e-ad41-4e7c-998d-910a8d13d617-kube-api-access-kz484" pod "a550117e-ad41-4e7c-998d-910a8d13d617" (UID: "a550117e-ad41-4e7c-998d-910a8d13d617"). InnerVolumeSpecName "kube-api-access-kz484". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:08:54.367458 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:54.367435 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a520a0b5-deab-4825-9b36-329c399754f6-istio-podinfo" pod "a520a0b5-deab-4825-9b36-329c399754f6" (UID: "a520a0b5-deab-4825-9b36-329c399754f6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:08:54.415293 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:54.415264 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-ccdzj" Oct 31 00:08:54.460158 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:54.460122 4927 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e8b64ac-0b2a-40aa-ab07-25ec13005517-istio-envoy\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:08:54.460312 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:54.460163 4927 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a550117e-ad41-4e7c-998d-910a8d13d617-credential-socket\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:08:54.460312 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:54.460181 4927 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a550117e-ad41-4e7c-998d-910a8d13d617-istio-podinfo\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:08:54.460312 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:54.460195 4927 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a520a0b5-deab-4825-9b36-329c399754f6-istiod-ca-cert\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:08:54.460312 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:54.460210 4927 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a550117e-ad41-4e7c-998d-910a8d13d617-istio-token\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:08:54.460312 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:54.460223 4927 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a520a0b5-deab-4825-9b36-329c399754f6-istio-token\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:08:54.460312 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:54.460236 4927 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a550117e-ad41-4e7c-998d-910a8d13d617-istio-envoy\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:08:54.460312 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:54.460251 4927 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e8b64ac-0b2a-40aa-ab07-25ec13005517-istio-podinfo\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:08:54.460312 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:54.460266 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g4cgf\" (UniqueName: \"kubernetes.io/projected/a520a0b5-deab-4825-9b36-329c399754f6-kube-api-access-g4cgf\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:08:54.460312 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:54.460280 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kz484\" (UniqueName: \"kubernetes.io/projected/a550117e-ad41-4e7c-998d-910a8d13d617-kube-api-access-kz484\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:08:54.460312 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:54.460296 4927 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a550117e-ad41-4e7c-998d-910a8d13d617-workload-certs\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:08:54.460312 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:54.460308 4927 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a520a0b5-deab-4825-9b36-329c399754f6-istio-envoy\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:08:54.460709 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:54.460322 4927 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a550117e-ad41-4e7c-998d-910a8d13d617-workload-socket\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:08:54.460709 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:54.460353 4927 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a520a0b5-deab-4825-9b36-329c399754f6-istio-podinfo\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:08:54.460709 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:54.460368 4927 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a520a0b5-deab-4825-9b36-329c399754f6-workload-socket\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:08:54.460709 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:54.460382 4927 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a550117e-ad41-4e7c-998d-910a8d13d617-istio-data\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:08:54.460709 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:54.460396 4927 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e8b64ac-0b2a-40aa-ab07-25ec13005517-istio-token\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:08:54.561021 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:54.560990 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/afc963ee-cfd4-441b-9185-a6ae83276e19-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/afc963ee-cfd4-441b-9185-a6ae83276e19-istiod-ca-cert\") pod \"afc963ee-cfd4-441b-9185-a6ae83276e19\" (UID: \"afc963ee-cfd4-441b-9185-a6ae83276e19\") " Oct 31 00:08:54.561208 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:54.561036 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/afc963ee-cfd4-441b-9185-a6ae83276e19-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/afc963ee-cfd4-441b-9185-a6ae83276e19-workload-certs\") pod \"afc963ee-cfd4-441b-9185-a6ae83276e19\" (UID: \"afc963ee-cfd4-441b-9185-a6ae83276e19\") " Oct 31 00:08:54.561208 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:54.561076 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/afc963ee-cfd4-441b-9185-a6ae83276e19-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/afc963ee-cfd4-441b-9185-a6ae83276e19-credential-socket\") pod \"afc963ee-cfd4-441b-9185-a6ae83276e19\" (UID: \"afc963ee-cfd4-441b-9185-a6ae83276e19\") " Oct 31 00:08:54.561208 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:54.561107 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/afc963ee-cfd4-441b-9185-a6ae83276e19-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/afc963ee-cfd4-441b-9185-a6ae83276e19-workload-socket\") pod \"afc963ee-cfd4-441b-9185-a6ae83276e19\" (UID: \"afc963ee-cfd4-441b-9185-a6ae83276e19\") " Oct 31 00:08:54.561208 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:54.561194 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/afc963ee-cfd4-441b-9185-a6ae83276e19-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/afc963ee-cfd4-441b-9185-a6ae83276e19-istio-podinfo\") pod \"afc963ee-cfd4-441b-9185-a6ae83276e19\" (UID: \"afc963ee-cfd4-441b-9185-a6ae83276e19\") " Oct 31 00:08:54.561496 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:54.561385 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afc963ee-cfd4-441b-9185-a6ae83276e19-workload-certs" pod "afc963ee-cfd4-441b-9185-a6ae83276e19" (UID: "afc963ee-cfd4-441b-9185-a6ae83276e19"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:08:54.561496 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:54.561410 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/afc963ee-cfd4-441b-9185-a6ae83276e19-istiod-ca-cert" pod "afc963ee-cfd4-441b-9185-a6ae83276e19" (UID: "afc963ee-cfd4-441b-9185-a6ae83276e19"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:08:54.561605 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:54.561579 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afc963ee-cfd4-441b-9185-a6ae83276e19-workload-socket" pod "afc963ee-cfd4-441b-9185-a6ae83276e19" (UID: "afc963ee-cfd4-441b-9185-a6ae83276e19"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:08:54.561665 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:54.561604 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afc963ee-cfd4-441b-9185-a6ae83276e19-credential-socket" pod "afc963ee-cfd4-441b-9185-a6ae83276e19" (UID: "afc963ee-cfd4-441b-9185-a6ae83276e19"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:08:54.561745 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:54.561718 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/afc963ee-cfd4-441b-9185-a6ae83276e19-istio-token\" (UniqueName: \"kubernetes.io/projected/afc963ee-cfd4-441b-9185-a6ae83276e19-istio-token\") pod \"afc963ee-cfd4-441b-9185-a6ae83276e19\" (UID: \"afc963ee-cfd4-441b-9185-a6ae83276e19\") " Oct 31 00:08:54.561909 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:54.561891 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/afc963ee-cfd4-441b-9185-a6ae83276e19-kube-api-access-zr7sv\" (UniqueName: \"kubernetes.io/projected/afc963ee-cfd4-441b-9185-a6ae83276e19-kube-api-access-zr7sv\") pod \"afc963ee-cfd4-441b-9185-a6ae83276e19\" (UID: \"afc963ee-cfd4-441b-9185-a6ae83276e19\") " Oct 31 00:08:54.561975 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:54.561963 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/afc963ee-cfd4-441b-9185-a6ae83276e19-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/afc963ee-cfd4-441b-9185-a6ae83276e19-istio-envoy\") pod \"afc963ee-cfd4-441b-9185-a6ae83276e19\" (UID: \"afc963ee-cfd4-441b-9185-a6ae83276e19\") " Oct 31 00:08:54.562084 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:54.562062 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/afc963ee-cfd4-441b-9185-a6ae83276e19-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/afc963ee-cfd4-441b-9185-a6ae83276e19-istio-data\") pod \"afc963ee-cfd4-441b-9185-a6ae83276e19\" (UID: \"afc963ee-cfd4-441b-9185-a6ae83276e19\") " Oct 31 00:08:54.562434 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:54.562405 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afc963ee-cfd4-441b-9185-a6ae83276e19-istio-data" pod "afc963ee-cfd4-441b-9185-a6ae83276e19" (UID: "afc963ee-cfd4-441b-9185-a6ae83276e19"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:08:54.562797 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:54.562774 4927 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/afc963ee-cfd4-441b-9185-a6ae83276e19-istio-data\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:08:54.562881 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:54.562804 4927 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/afc963ee-cfd4-441b-9185-a6ae83276e19-istiod-ca-cert\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:08:54.562881 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:54.562822 4927 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/afc963ee-cfd4-441b-9185-a6ae83276e19-workload-certs\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:08:54.562881 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:54.562836 4927 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/afc963ee-cfd4-441b-9185-a6ae83276e19-credential-socket\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:08:54.563054 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:54.562868 4927 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/afc963ee-cfd4-441b-9185-a6ae83276e19-workload-socket\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:08:54.563698 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:54.563675 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/afc963ee-cfd4-441b-9185-a6ae83276e19-istio-podinfo" pod "afc963ee-cfd4-441b-9185-a6ae83276e19" (UID: "afc963ee-cfd4-441b-9185-a6ae83276e19"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:08:54.563800 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:54.563726 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/afc963ee-cfd4-441b-9185-a6ae83276e19-istio-token" pod "afc963ee-cfd4-441b-9185-a6ae83276e19" (UID: "afc963ee-cfd4-441b-9185-a6ae83276e19"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:08:54.564068 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:54.564051 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afc963ee-cfd4-441b-9185-a6ae83276e19-istio-envoy" pod "afc963ee-cfd4-441b-9185-a6ae83276e19" (UID: "afc963ee-cfd4-441b-9185-a6ae83276e19"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:08:54.564561 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:54.564533 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/afc963ee-cfd4-441b-9185-a6ae83276e19-kube-api-access-zr7sv" pod "afc963ee-cfd4-441b-9185-a6ae83276e19" (UID: "afc963ee-cfd4-441b-9185-a6ae83276e19"). InnerVolumeSpecName "kube-api-access-zr7sv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:08:54.693360 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:54.671795 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zr7sv\" (UniqueName: \"kubernetes.io/projected/afc963ee-cfd4-441b-9185-a6ae83276e19-kube-api-access-zr7sv\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:08:54.693360 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:54.671833 4927 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/afc963ee-cfd4-441b-9185-a6ae83276e19-istio-envoy\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:08:54.693360 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:54.671848 4927 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/afc963ee-cfd4-441b-9185-a6ae83276e19-istio-podinfo\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:08:54.693360 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:54.671864 4927 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/afc963ee-cfd4-441b-9185-a6ae83276e19-istio-token\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:08:54.934366 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:54.919870 4927 patch_prober.go:29] interesting pod/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-vb9r5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:08:54.934366 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:54.919955 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-vb9r5" podUID="334f41bf-6d81-474d-bddc-d38c3492d586" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:08:55.036879 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:54.953314 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-nr4qz"] Oct 31 00:08:55.037162 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:55.037130 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-nr4qz"] Oct 31 00:08:55.037347 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:55.037321 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w6-0d75be-openshift-default-57f99d9cbc-smttn"] Oct 31 00:08:55.038711 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:55.038668 4927 cpu_manager.go:414] "containerMap: removing container" podUID="45977fb0-7f5d-4f55-9530-112d60f269ab" containerName="istio-proxy" Oct 31 00:08:55.038837 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:55.038824 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45977fb0-7f5d-4f55-9530-112d60f269ab" containerName="istio-proxy" Oct 31 00:08:55.038949 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:55.038939 4927 cpu_manager.go:414] "containerMap: removing container" podUID="b7194267-cfef-4252-ae16-a194eccd2a1f" containerName="istio-proxy" Oct 31 00:08:55.039022 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:55.039012 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7194267-cfef-4252-ae16-a194eccd2a1f" containerName="istio-proxy" Oct 31 00:08:55.039094 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:55.039086 4927 cpu_manager.go:414] "containerMap: removing container" podUID="e6b778b5-4861-4a3f-85cb-5524098583f1" containerName="istio-proxy" Oct 31 00:08:55.039163 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:55.039153 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6b778b5-4861-4a3f-85cb-5524098583f1" containerName="istio-proxy" Oct 31 00:08:55.039241 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:55.039233 4927 cpu_manager.go:414] "containerMap: removing container" podUID="9f348596-394c-4bfb-ba79-4788da8fc6de" containerName="istio-proxy" Oct 31 00:08:55.039307 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:55.039297 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f348596-394c-4bfb-ba79-4788da8fc6de" containerName="istio-proxy" Oct 31 00:08:55.039394 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:55.039384 4927 cpu_manager.go:414] "containerMap: removing container" podUID="afc963ee-cfd4-441b-9185-a6ae83276e19" containerName="istio-proxy" Oct 31 00:08:55.039456 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:55.039447 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="afc963ee-cfd4-441b-9185-a6ae83276e19" containerName="istio-proxy" Oct 31 00:08:55.039532 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:55.039524 4927 cpu_manager.go:414] "containerMap: removing container" podUID="a520a0b5-deab-4825-9b36-329c399754f6" containerName="istio-proxy" Oct 31 00:08:55.039595 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:55.039585 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a520a0b5-deab-4825-9b36-329c399754f6" containerName="istio-proxy" Oct 31 00:08:55.039687 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:55.039679 4927 cpu_manager.go:414] "containerMap: removing container" podUID="333d9301-4a9b-4867-b3d9-88d3766d1acb" containerName="istio-proxy" Oct 31 00:08:55.039752 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:55.039742 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="333d9301-4a9b-4867-b3d9-88d3766d1acb" containerName="istio-proxy" Oct 31 00:08:55.039829 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:55.039816 4927 cpu_manager.go:414] "containerMap: removing container" podUID="3e8b64ac-0b2a-40aa-ab07-25ec13005517" containerName="istio-proxy" Oct 31 00:08:55.039829 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:55.039826 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e8b64ac-0b2a-40aa-ab07-25ec13005517" containerName="istio-proxy" Oct 31 00:08:55.039973 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:55.039845 4927 cpu_manager.go:414] "containerMap: removing container" podUID="334f41bf-6d81-474d-bddc-d38c3492d586" containerName="istio-proxy" Oct 31 00:08:55.039973 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:55.039852 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="334f41bf-6d81-474d-bddc-d38c3492d586" containerName="istio-proxy" Oct 31 00:08:55.039973 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:55.039871 4927 cpu_manager.go:414] "containerMap: removing container" podUID="a550117e-ad41-4e7c-998d-910a8d13d617" containerName="istio-proxy" Oct 31 00:08:55.039973 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:55.039878 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a550117e-ad41-4e7c-998d-910a8d13d617" containerName="istio-proxy" Oct 31 00:08:55.039973 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:55.039894 4927 cpu_manager.go:414] "containerMap: removing container" podUID="b7eb6748-2997-4527-91b3-86900ea70eed" containerName="istio-proxy" Oct 31 00:08:55.039973 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:55.039901 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7eb6748-2997-4527-91b3-86900ea70eed" containerName="istio-proxy" Oct 31 00:08:55.039973 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:55.039926 4927 cpu_manager.go:414] "containerMap: removing container" podUID="2b317f84-83ac-49aa-921e-0242889f2b4f" containerName="istio-proxy" Oct 31 00:08:55.039973 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:55.039933 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b317f84-83ac-49aa-921e-0242889f2b4f" containerName="istio-proxy" Oct 31 00:08:55.040393 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:55.040130 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="a550117e-ad41-4e7c-998d-910a8d13d617" containerName="istio-proxy" Oct 31 00:08:55.040393 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:55.040146 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7eb6748-2997-4527-91b3-86900ea70eed" containerName="istio-proxy" Oct 31 00:08:55.040393 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:55.040158 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b317f84-83ac-49aa-921e-0242889f2b4f" containerName="istio-proxy" Oct 31 00:08:55.040393 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:55.040172 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f348596-394c-4bfb-ba79-4788da8fc6de" containerName="istio-proxy" Oct 31 00:08:55.040393 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:55.040182 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="45977fb0-7f5d-4f55-9530-112d60f269ab" containerName="istio-proxy" Oct 31 00:08:55.040393 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:55.040192 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7194267-cfef-4252-ae16-a194eccd2a1f" containerName="istio-proxy" Oct 31 00:08:55.040393 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:55.040200 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6b778b5-4861-4a3f-85cb-5524098583f1" containerName="istio-proxy" Oct 31 00:08:55.040393 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:55.040209 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f348596-394c-4bfb-ba79-4788da8fc6de" containerName="istio-proxy" Oct 31 00:08:55.040393 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:55.040219 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="afc963ee-cfd4-441b-9185-a6ae83276e19" containerName="istio-proxy" Oct 31 00:08:55.040393 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:55.040229 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="a520a0b5-deab-4825-9b36-329c399754f6" containerName="istio-proxy" Oct 31 00:08:55.040393 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:55.040242 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="333d9301-4a9b-4867-b3d9-88d3766d1acb" containerName="istio-proxy" Oct 31 00:08:55.040393 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:55.040252 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e8b64ac-0b2a-40aa-ab07-25ec13005517" containerName="istio-proxy" Oct 31 00:08:55.040393 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:55.040261 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="334f41bf-6d81-474d-bddc-d38c3492d586" containerName="istio-proxy" Oct 31 00:08:55.045115 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:55.044918 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-0d75be-openshift-default-57f99d9cbc-smttn"] Oct 31 00:08:55.045115 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:55.045020 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-0d75be-openshift-default-57f99d9cbc-smttn" Oct 31 00:08:55.045292 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:55.017280 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:08:55.100732 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:55.100703 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w6-0d75be-openshift-default-dockercfg-c6frw\"" Oct 31 00:08:55.103869 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:55.103838 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="333d9301-4a9b-4867-b3d9-88d3766d1acb" path="/var/lib/kubelet/pods/333d9301-4a9b-4867-b3d9-88d3766d1acb/volumes" Oct 31 00:08:55.104577 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:55.104553 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f348596-394c-4bfb-ba79-4788da8fc6de" path="/var/lib/kubelet/pods/9f348596-394c-4bfb-ba79-4788da8fc6de/volumes" Oct 31 00:08:55.105202 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:55.105175 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e6b778b5-4861-4a3f-85cb-5524098583f1" path="/var/lib/kubelet/pods/e6b778b5-4861-4a3f-85cb-5524098583f1/volumes" Oct 31 00:08:55.126777 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:55.126747 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61bb8f6c-7dd6-4e87-ac44-5374dd11c0dc-istio-envoy\") pod \"e2e-shared-b-w6-0d75be-openshift-default-57f99d9cbc-smttn\" (UID: \"61bb8f6c-7dd6-4e87-ac44-5374dd11c0dc\") " pod="openshift-ingress/e2e-shared-b-w6-0d75be-openshift-default-57f99d9cbc-smttn" Oct 31 00:08:55.126981 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:55.126800 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61bb8f6c-7dd6-4e87-ac44-5374dd11c0dc-workload-certs\") pod \"e2e-shared-b-w6-0d75be-openshift-default-57f99d9cbc-smttn\" (UID: \"61bb8f6c-7dd6-4e87-ac44-5374dd11c0dc\") " pod="openshift-ingress/e2e-shared-b-w6-0d75be-openshift-default-57f99d9cbc-smttn" Oct 31 00:08:55.126981 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:55.126909 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61bb8f6c-7dd6-4e87-ac44-5374dd11c0dc-istio-podinfo\") pod \"e2e-shared-b-w6-0d75be-openshift-default-57f99d9cbc-smttn\" (UID: \"61bb8f6c-7dd6-4e87-ac44-5374dd11c0dc\") " pod="openshift-ingress/e2e-shared-b-w6-0d75be-openshift-default-57f99d9cbc-smttn" Oct 31 00:08:55.126981 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:55.126938 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ld68\" (UniqueName: \"kubernetes.io/projected/61bb8f6c-7dd6-4e87-ac44-5374dd11c0dc-kube-api-access-9ld68\") pod \"e2e-shared-b-w6-0d75be-openshift-default-57f99d9cbc-smttn\" (UID: \"61bb8f6c-7dd6-4e87-ac44-5374dd11c0dc\") " pod="openshift-ingress/e2e-shared-b-w6-0d75be-openshift-default-57f99d9cbc-smttn" Oct 31 00:08:55.127155 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:55.127045 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61bb8f6c-7dd6-4e87-ac44-5374dd11c0dc-istiod-ca-cert\") pod \"e2e-shared-b-w6-0d75be-openshift-default-57f99d9cbc-smttn\" (UID: \"61bb8f6c-7dd6-4e87-ac44-5374dd11c0dc\") " pod="openshift-ingress/e2e-shared-b-w6-0d75be-openshift-default-57f99d9cbc-smttn" Oct 31 00:08:55.127155 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:55.127125 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61bb8f6c-7dd6-4e87-ac44-5374dd11c0dc-credential-socket\") pod \"e2e-shared-b-w6-0d75be-openshift-default-57f99d9cbc-smttn\" (UID: \"61bb8f6c-7dd6-4e87-ac44-5374dd11c0dc\") " pod="openshift-ingress/e2e-shared-b-w6-0d75be-openshift-default-57f99d9cbc-smttn" Oct 31 00:08:55.127265 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:55.127151 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61bb8f6c-7dd6-4e87-ac44-5374dd11c0dc-istio-token\") pod \"e2e-shared-b-w6-0d75be-openshift-default-57f99d9cbc-smttn\" (UID: \"61bb8f6c-7dd6-4e87-ac44-5374dd11c0dc\") " pod="openshift-ingress/e2e-shared-b-w6-0d75be-openshift-default-57f99d9cbc-smttn" Oct 31 00:08:55.127265 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:55.127188 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61bb8f6c-7dd6-4e87-ac44-5374dd11c0dc-istio-data\") pod \"e2e-shared-b-w6-0d75be-openshift-default-57f99d9cbc-smttn\" (UID: \"61bb8f6c-7dd6-4e87-ac44-5374dd11c0dc\") " pod="openshift-ingress/e2e-shared-b-w6-0d75be-openshift-default-57f99d9cbc-smttn" Oct 31 00:08:55.127265 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:55.127243 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61bb8f6c-7dd6-4e87-ac44-5374dd11c0dc-workload-socket\") pod \"e2e-shared-b-w6-0d75be-openshift-default-57f99d9cbc-smttn\" (UID: \"61bb8f6c-7dd6-4e87-ac44-5374dd11c0dc\") " pod="openshift-ingress/e2e-shared-b-w6-0d75be-openshift-default-57f99d9cbc-smttn" Oct 31 00:08:55.137427 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:55.137401 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-vb9r5" Oct 31 00:08:55.137518 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:55.137399 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-vb9r5" event={"ID":"334f41bf-6d81-474d-bddc-d38c3492d586","Type":"ContainerDied","Data":"a8faa7e12e482d7a0f93946238c3a4bea11ea9a8989aca687e18df250bd5420c"} Oct 31 00:08:55.155625 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:55.155539 4927 generic.go:356] "Generic (PLEG): container finished" podID="a550117e-ad41-4e7c-998d-910a8d13d617" containerID="8351cbcebc39fc6a3f1bd858ac5bebc698fad85fb20361588f689aae67eaefa9" exitCode=0 Oct 31 00:08:55.157216 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:55.155720 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-kzqr4" event={"ID":"a550117e-ad41-4e7c-998d-910a8d13d617","Type":"ContainerDied","Data":"8351cbcebc39fc6a3f1bd858ac5bebc698fad85fb20361588f689aae67eaefa9"} Oct 31 00:08:55.157525 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:55.157500 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-kzqr4" event={"ID":"a550117e-ad41-4e7c-998d-910a8d13d617","Type":"ContainerDied","Data":"bee5c50d0ae3c67110da57bfdf45b6cd79ee2b556401181d0c987e2eae13d0b7"} Oct 31 00:08:55.157622 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:55.157584 4927 scope.go:122] "RemoveContainer" containerID="8351cbcebc39fc6a3f1bd858ac5bebc698fad85fb20361588f689aae67eaefa9" Oct 31 00:08:55.157752 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:55.155912 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-kzqr4" Oct 31 00:08:55.169066 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:55.169037 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-7gm84" event={"ID":"45977fb0-7f5d-4f55-9530-112d60f269ab","Type":"ContainerDied","Data":"c8f56af937bf936720bb0359806ee16cec1b02c5fe9d9345f78b93a58d757268"} Oct 31 00:08:55.169347 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:55.169198 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-7gm84" Oct 31 00:08:55.173599 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:55.173569 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-2qnns" event={"ID":"b7194267-cfef-4252-ae16-a194eccd2a1f","Type":"ContainerDied","Data":"606def46e4d828e7bc4e8fa5db08c5ea941805999bf5b50f32be59dcc6dc51c4"} Oct 31 00:08:55.173739 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:55.173723 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-2qnns" Oct 31 00:08:55.175840 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:55.175815 4927 generic.go:356] "Generic (PLEG): container finished" podID="a520a0b5-deab-4825-9b36-329c399754f6" containerID="7176a65eae8ecdf19042d0452aac23fa54b6cb2c488a78b8fd12c4e09bc31744" exitCode=0 Oct 31 00:08:55.176485 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:55.175896 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-jdcvx" event={"ID":"a520a0b5-deab-4825-9b36-329c399754f6","Type":"ContainerDied","Data":"7176a65eae8ecdf19042d0452aac23fa54b6cb2c488a78b8fd12c4e09bc31744"} Oct 31 00:08:55.176596 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:55.176530 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-jdcvx" event={"ID":"a520a0b5-deab-4825-9b36-329c399754f6","Type":"ContainerDied","Data":"fd9f345acb4b480c88d6746c033f65a00a195c33416f0dbcc07a463f8c7de337"} Oct 31 00:08:55.176596 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:55.175959 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-jdcvx" Oct 31 00:08:55.181965 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:55.181696 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-khcqd" Oct 31 00:08:55.182172 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:55.181695 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-khcqd" event={"ID":"2b317f84-83ac-49aa-921e-0242889f2b4f","Type":"ContainerDied","Data":"186e18ef3a62a47297bb18293b16591f3c30e78e12fea1793ac95b1e7883e0da"} Oct 31 00:08:55.183368 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:55.183322 4927 generic.go:356] "Generic (PLEG): container finished" podID="afc963ee-cfd4-441b-9185-a6ae83276e19" containerID="51b115498114d0ec225e6a94c5934f19833fa1d6d10f859b7f49762a5b8d07df" exitCode=0 Oct 31 00:08:55.183565 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:55.183433 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-ccdzj" event={"ID":"afc963ee-cfd4-441b-9185-a6ae83276e19","Type":"ContainerDied","Data":"51b115498114d0ec225e6a94c5934f19833fa1d6d10f859b7f49762a5b8d07df"} Oct 31 00:08:55.183648 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:55.183590 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-ccdzj" event={"ID":"afc963ee-cfd4-441b-9185-a6ae83276e19","Type":"ContainerDied","Data":"a310c24e096de9af436960158880c9c0cba8034fb08aa825387af6a8df5be438"} Oct 31 00:08:55.183648 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:55.183456 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-ccdzj" Oct 31 00:08:55.187243 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:55.187190 4927 generic.go:356] "Generic (PLEG): container finished" podID="3e8b64ac-0b2a-40aa-ab07-25ec13005517" containerID="d67f2bd16f7e33e47e54489c1a91fb2008dda3cfc24949e624a22d15d0ebf8e1" exitCode=0 Oct 31 00:08:55.187318 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:55.187267 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-tgc8q" event={"ID":"3e8b64ac-0b2a-40aa-ab07-25ec13005517","Type":"ContainerDied","Data":"d67f2bd16f7e33e47e54489c1a91fb2008dda3cfc24949e624a22d15d0ebf8e1"} Oct 31 00:08:55.187318 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:55.187303 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-tgc8q" event={"ID":"3e8b64ac-0b2a-40aa-ab07-25ec13005517","Type":"ContainerDied","Data":"eb0f9e5dd4bc9c042b076161a3da97fcb688899ae0642c52b637291ac5256b6d"} Oct 31 00:08:55.187431 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:55.187415 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-tgc8q" Oct 31 00:08:55.192506 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:55.192020 4927 scope.go:122] "RemoveContainer" containerID="8351cbcebc39fc6a3f1bd858ac5bebc698fad85fb20361588f689aae67eaefa9" Oct 31 00:08:55.192506 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:08:55.192268 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8351cbcebc39fc6a3f1bd858ac5bebc698fad85fb20361588f689aae67eaefa9\": container with ID starting with 8351cbcebc39fc6a3f1bd858ac5bebc698fad85fb20361588f689aae67eaefa9 not found: ID does not exist" containerID="8351cbcebc39fc6a3f1bd858ac5bebc698fad85fb20361588f689aae67eaefa9" Oct 31 00:08:55.192506 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:55.192292 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8351cbcebc39fc6a3f1bd858ac5bebc698fad85fb20361588f689aae67eaefa9"} err="failed to get container status \"8351cbcebc39fc6a3f1bd858ac5bebc698fad85fb20361588f689aae67eaefa9\": rpc error: code = NotFound desc = could not find container \"8351cbcebc39fc6a3f1bd858ac5bebc698fad85fb20361588f689aae67eaefa9\": container with ID starting with 8351cbcebc39fc6a3f1bd858ac5bebc698fad85fb20361588f689aae67eaefa9 not found: ID does not exist" Oct 31 00:08:55.192506 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:55.192308 4927 scope.go:122] "RemoveContainer" containerID="646c05c17da8a4f530f45bf9bb4e8348afb3fbe87e9040d1ee2cea4e92dccb62" Oct 31 00:08:55.277360 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:55.232541 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61bb8f6c-7dd6-4e87-ac44-5374dd11c0dc-workload-certs\") pod \"e2e-shared-b-w6-0d75be-openshift-default-57f99d9cbc-smttn\" (UID: \"61bb8f6c-7dd6-4e87-ac44-5374dd11c0dc\") " pod="openshift-ingress/e2e-shared-b-w6-0d75be-openshift-default-57f99d9cbc-smttn" Oct 31 00:08:55.310163 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:55.308979 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61bb8f6c-7dd6-4e87-ac44-5374dd11c0dc-istio-podinfo\") pod \"e2e-shared-b-w6-0d75be-openshift-default-57f99d9cbc-smttn\" (UID: \"61bb8f6c-7dd6-4e87-ac44-5374dd11c0dc\") " pod="openshift-ingress/e2e-shared-b-w6-0d75be-openshift-default-57f99d9cbc-smttn" Oct 31 00:08:55.347777 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:55.347735 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ld68\" (UniqueName: \"kubernetes.io/projected/61bb8f6c-7dd6-4e87-ac44-5374dd11c0dc-kube-api-access-9ld68\") pod \"e2e-shared-b-w6-0d75be-openshift-default-57f99d9cbc-smttn\" (UID: \"61bb8f6c-7dd6-4e87-ac44-5374dd11c0dc\") " pod="openshift-ingress/e2e-shared-b-w6-0d75be-openshift-default-57f99d9cbc-smttn" Oct 31 00:08:55.348350 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:55.348307 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61bb8f6c-7dd6-4e87-ac44-5374dd11c0dc-istiod-ca-cert\") pod \"e2e-shared-b-w6-0d75be-openshift-default-57f99d9cbc-smttn\" (UID: \"61bb8f6c-7dd6-4e87-ac44-5374dd11c0dc\") " pod="openshift-ingress/e2e-shared-b-w6-0d75be-openshift-default-57f99d9cbc-smttn" Oct 31 00:08:55.415452 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:55.415413 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61bb8f6c-7dd6-4e87-ac44-5374dd11c0dc-credential-socket\") pod \"e2e-shared-b-w6-0d75be-openshift-default-57f99d9cbc-smttn\" (UID: \"61bb8f6c-7dd6-4e87-ac44-5374dd11c0dc\") " pod="openshift-ingress/e2e-shared-b-w6-0d75be-openshift-default-57f99d9cbc-smttn" Oct 31 00:08:55.418904 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:55.418872 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61bb8f6c-7dd6-4e87-ac44-5374dd11c0dc-istio-token\") pod \"e2e-shared-b-w6-0d75be-openshift-default-57f99d9cbc-smttn\" (UID: \"61bb8f6c-7dd6-4e87-ac44-5374dd11c0dc\") " pod="openshift-ingress/e2e-shared-b-w6-0d75be-openshift-default-57f99d9cbc-smttn" Oct 31 00:08:55.420019 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:55.419993 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61bb8f6c-7dd6-4e87-ac44-5374dd11c0dc-istio-data\") pod \"e2e-shared-b-w6-0d75be-openshift-default-57f99d9cbc-smttn\" (UID: \"61bb8f6c-7dd6-4e87-ac44-5374dd11c0dc\") " pod="openshift-ingress/e2e-shared-b-w6-0d75be-openshift-default-57f99d9cbc-smttn" Oct 31 00:08:55.420230 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:55.420213 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61bb8f6c-7dd6-4e87-ac44-5374dd11c0dc-workload-socket\") pod \"e2e-shared-b-w6-0d75be-openshift-default-57f99d9cbc-smttn\" (UID: \"61bb8f6c-7dd6-4e87-ac44-5374dd11c0dc\") " pod="openshift-ingress/e2e-shared-b-w6-0d75be-openshift-default-57f99d9cbc-smttn" Oct 31 00:08:55.420693 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:55.272631 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61bb8f6c-7dd6-4e87-ac44-5374dd11c0dc-workload-certs\") pod \"e2e-shared-b-w6-0d75be-openshift-default-57f99d9cbc-smttn\" (UID: \"61bb8f6c-7dd6-4e87-ac44-5374dd11c0dc\") " pod="openshift-ingress/e2e-shared-b-w6-0d75be-openshift-default-57f99d9cbc-smttn" Oct 31 00:08:55.420833 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:55.338674 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:08:55.420833 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:55.419225 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61bb8f6c-7dd6-4e87-ac44-5374dd11c0dc-credential-socket\") pod \"e2e-shared-b-w6-0d75be-openshift-default-57f99d9cbc-smttn\" (UID: \"61bb8f6c-7dd6-4e87-ac44-5374dd11c0dc\") " pod="openshift-ingress/e2e-shared-b-w6-0d75be-openshift-default-57f99d9cbc-smttn" Oct 31 00:08:55.420833 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:55.355078 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61bb8f6c-7dd6-4e87-ac44-5374dd11c0dc-istiod-ca-cert\") pod \"e2e-shared-b-w6-0d75be-openshift-default-57f99d9cbc-smttn\" (UID: \"61bb8f6c-7dd6-4e87-ac44-5374dd11c0dc\") " pod="openshift-ingress/e2e-shared-b-w6-0d75be-openshift-default-57f99d9cbc-smttn" Oct 31 00:08:55.421466 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:55.421113 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61bb8f6c-7dd6-4e87-ac44-5374dd11c0dc-workload-socket\") pod \"e2e-shared-b-w6-0d75be-openshift-default-57f99d9cbc-smttn\" (UID: \"61bb8f6c-7dd6-4e87-ac44-5374dd11c0dc\") " pod="openshift-ingress/e2e-shared-b-w6-0d75be-openshift-default-57f99d9cbc-smttn" Oct 31 00:08:55.421466 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:55.308760 4927 scope.go:122] "RemoveContainer" containerID="6616bfd1ea4a480ce02466ce388380eb31ea8e8030af93e453c8d0c7a09224b2" Oct 31 00:08:55.421862 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:55.386646 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61bb8f6c-7dd6-4e87-ac44-5374dd11c0dc-istio-podinfo\") pod \"e2e-shared-b-w6-0d75be-openshift-default-57f99d9cbc-smttn\" (UID: \"61bb8f6c-7dd6-4e87-ac44-5374dd11c0dc\") " pod="openshift-ingress/e2e-shared-b-w6-0d75be-openshift-default-57f99d9cbc-smttn" Oct 31 00:08:55.422279 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:55.422242 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61bb8f6c-7dd6-4e87-ac44-5374dd11c0dc-istio-data\") pod \"e2e-shared-b-w6-0d75be-openshift-default-57f99d9cbc-smttn\" (UID: \"61bb8f6c-7dd6-4e87-ac44-5374dd11c0dc\") " pod="openshift-ingress/e2e-shared-b-w6-0d75be-openshift-default-57f99d9cbc-smttn" Oct 31 00:08:55.424723 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:55.424701 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61bb8f6c-7dd6-4e87-ac44-5374dd11c0dc-istio-envoy\") pod \"e2e-shared-b-w6-0d75be-openshift-default-57f99d9cbc-smttn\" (UID: \"61bb8f6c-7dd6-4e87-ac44-5374dd11c0dc\") " pod="openshift-ingress/e2e-shared-b-w6-0d75be-openshift-default-57f99d9cbc-smttn" Oct 31 00:08:55.434406 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:55.434368 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ld68\" (UniqueName: \"kubernetes.io/projected/61bb8f6c-7dd6-4e87-ac44-5374dd11c0dc-kube-api-access-9ld68\") pod \"e2e-shared-b-w6-0d75be-openshift-default-57f99d9cbc-smttn\" (UID: \"61bb8f6c-7dd6-4e87-ac44-5374dd11c0dc\") " pod="openshift-ingress/e2e-shared-b-w6-0d75be-openshift-default-57f99d9cbc-smttn" Oct 31 00:08:55.475559 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:55.475490 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61bb8f6c-7dd6-4e87-ac44-5374dd11c0dc-istio-envoy\") pod \"e2e-shared-b-w6-0d75be-openshift-default-57f99d9cbc-smttn\" (UID: \"61bb8f6c-7dd6-4e87-ac44-5374dd11c0dc\") " pod="openshift-ingress/e2e-shared-b-w6-0d75be-openshift-default-57f99d9cbc-smttn" Oct 31 00:08:55.485708 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:55.485673 4927 scope.go:122] "RemoveContainer" containerID="7176a65eae8ecdf19042d0452aac23fa54b6cb2c488a78b8fd12c4e09bc31744" Oct 31 00:08:55.508057 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:55.507988 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61bb8f6c-7dd6-4e87-ac44-5374dd11c0dc-istio-token\") pod \"e2e-shared-b-w6-0d75be-openshift-default-57f99d9cbc-smttn\" (UID: \"61bb8f6c-7dd6-4e87-ac44-5374dd11c0dc\") " pod="openshift-ingress/e2e-shared-b-w6-0d75be-openshift-default-57f99d9cbc-smttn" Oct 31 00:08:55.528687 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:55.528664 4927 scope.go:122] "RemoveContainer" containerID="7176a65eae8ecdf19042d0452aac23fa54b6cb2c488a78b8fd12c4e09bc31744" Oct 31 00:08:55.529104 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:08:55.529078 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7176a65eae8ecdf19042d0452aac23fa54b6cb2c488a78b8fd12c4e09bc31744\": container with ID starting with 7176a65eae8ecdf19042d0452aac23fa54b6cb2c488a78b8fd12c4e09bc31744 not found: ID does not exist" containerID="7176a65eae8ecdf19042d0452aac23fa54b6cb2c488a78b8fd12c4e09bc31744" Oct 31 00:08:55.529242 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:55.529119 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7176a65eae8ecdf19042d0452aac23fa54b6cb2c488a78b8fd12c4e09bc31744"} err="failed to get container status \"7176a65eae8ecdf19042d0452aac23fa54b6cb2c488a78b8fd12c4e09bc31744\": rpc error: code = NotFound desc = could not find container \"7176a65eae8ecdf19042d0452aac23fa54b6cb2c488a78b8fd12c4e09bc31744\": container with ID starting with 7176a65eae8ecdf19042d0452aac23fa54b6cb2c488a78b8fd12c4e09bc31744 not found: ID does not exist" Oct 31 00:08:55.529242 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:55.529144 4927 scope.go:122] "RemoveContainer" containerID="f2edac246d0fdf2378197ddfb3fb9eab46f7ea47415499c914edaee90412f22b" Oct 31 00:08:55.540675 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:55.540651 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-klh6t"] Oct 31 00:08:55.544060 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:55.544031 4927 scope.go:122] "RemoveContainer" containerID="51b115498114d0ec225e6a94c5934f19833fa1d6d10f859b7f49762a5b8d07df" Oct 31 00:08:55.546285 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:55.546264 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-klh6t"] Oct 31 00:08:55.555533 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:55.555509 4927 scope.go:122] "RemoveContainer" containerID="51b115498114d0ec225e6a94c5934f19833fa1d6d10f859b7f49762a5b8d07df" Oct 31 00:08:55.555849 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:08:55.555813 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"51b115498114d0ec225e6a94c5934f19833fa1d6d10f859b7f49762a5b8d07df\": container with ID starting with 51b115498114d0ec225e6a94c5934f19833fa1d6d10f859b7f49762a5b8d07df not found: ID does not exist" containerID="51b115498114d0ec225e6a94c5934f19833fa1d6d10f859b7f49762a5b8d07df" Oct 31 00:08:55.555921 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:55.555858 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51b115498114d0ec225e6a94c5934f19833fa1d6d10f859b7f49762a5b8d07df"} err="failed to get container status \"51b115498114d0ec225e6a94c5934f19833fa1d6d10f859b7f49762a5b8d07df\": rpc error: code = NotFound desc = could not find container \"51b115498114d0ec225e6a94c5934f19833fa1d6d10f859b7f49762a5b8d07df\": container with ID starting with 51b115498114d0ec225e6a94c5934f19833fa1d6d10f859b7f49762a5b8d07df not found: ID does not exist" Oct 31 00:08:55.555921 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:55.555878 4927 scope.go:122] "RemoveContainer" containerID="d67f2bd16f7e33e47e54489c1a91fb2008dda3cfc24949e624a22d15d0ebf8e1" Oct 31 00:08:55.567855 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:55.567834 4927 scope.go:122] "RemoveContainer" containerID="d67f2bd16f7e33e47e54489c1a91fb2008dda3cfc24949e624a22d15d0ebf8e1" Oct 31 00:08:55.568227 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:08:55.568200 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d67f2bd16f7e33e47e54489c1a91fb2008dda3cfc24949e624a22d15d0ebf8e1\": container with ID starting with d67f2bd16f7e33e47e54489c1a91fb2008dda3cfc24949e624a22d15d0ebf8e1 not found: ID does not exist" containerID="d67f2bd16f7e33e47e54489c1a91fb2008dda3cfc24949e624a22d15d0ebf8e1" Oct 31 00:08:55.568324 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:55.568231 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d67f2bd16f7e33e47e54489c1a91fb2008dda3cfc24949e624a22d15d0ebf8e1"} err="failed to get container status \"d67f2bd16f7e33e47e54489c1a91fb2008dda3cfc24949e624a22d15d0ebf8e1\": rpc error: code = NotFound desc = could not find container \"d67f2bd16f7e33e47e54489c1a91fb2008dda3cfc24949e624a22d15d0ebf8e1\": container with ID starting with d67f2bd16f7e33e47e54489c1a91fb2008dda3cfc24949e624a22d15d0ebf8e1 not found: ID does not exist" Oct 31 00:08:55.686512 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:55.686470 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-0d75be-openshift-default-57f99d9cbc-smttn" Oct 31 00:08:55.941907 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:55.941870 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-jdcvx"] Oct 31 00:08:55.962183 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:55.962112 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-jdcvx"] Oct 31 00:08:56.201034 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:56.201003 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-tgc8q"] Oct 31 00:08:56.207937 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:56.207912 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-tgc8q"] Oct 31 00:08:56.227636 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:56.227606 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-0d75be-openshift-default-57f99d9cbc-smttn"] Oct 31 00:08:56.227861 ip-10-0-141-180 kubenswrapper[4927]: W1031 00:08:56.227839 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61bb8f6c_7dd6_4e87_ac44_5374dd11c0dc.slice/crio-e53c66abfd5c9cbb206a1f340d71f045df7db48f695aa9448d089ba541d0e7f2 WatchSource:0}: Error finding container e53c66abfd5c9cbb206a1f340d71f045df7db48f695aa9448d089ba541d0e7f2: Status 404 returned error can't find the container with id e53c66abfd5c9cbb206a1f340d71f045df7db48f695aa9448d089ba541d0e7f2 Oct 31 00:08:56.230521 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:56.230490 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:08:56.230623 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:56.230551 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:08:56.230623 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:56.230584 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:08:56.317660 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:56.317635 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-ccdzj"] Oct 31 00:08:56.325020 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:56.324995 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-ccdzj"] Oct 31 00:08:56.437353 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:56.436523 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-khcqd"] Oct 31 00:08:56.441506 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:56.441479 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-khcqd"] Oct 31 00:08:56.563809 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:56.563711 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-vb9r5"] Oct 31 00:08:56.569704 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:56.569673 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-vb9r5"] Oct 31 00:08:56.681246 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:56.681194 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-kzqr4"] Oct 31 00:08:56.685842 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:56.685816 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-kzqr4"] Oct 31 00:08:56.801571 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:56.801533 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-2qnns"] Oct 31 00:08:56.806805 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:56.806773 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-2qnns"] Oct 31 00:08:56.917290 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:56.917248 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-7gm84"] Oct 31 00:08:56.922111 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:56.922083 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-7gm84"] Oct 31 00:08:57.102718 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:57.102682 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b317f84-83ac-49aa-921e-0242889f2b4f" path="/var/lib/kubelet/pods/2b317f84-83ac-49aa-921e-0242889f2b4f/volumes" Oct 31 00:08:57.103234 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:57.103220 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="334f41bf-6d81-474d-bddc-d38c3492d586" path="/var/lib/kubelet/pods/334f41bf-6d81-474d-bddc-d38c3492d586/volumes" Oct 31 00:08:57.103705 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:57.103691 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3e8b64ac-0b2a-40aa-ab07-25ec13005517" path="/var/lib/kubelet/pods/3e8b64ac-0b2a-40aa-ab07-25ec13005517/volumes" Oct 31 00:08:57.104562 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:57.104531 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="45977fb0-7f5d-4f55-9530-112d60f269ab" path="/var/lib/kubelet/pods/45977fb0-7f5d-4f55-9530-112d60f269ab/volumes" Oct 31 00:08:57.105233 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:57.105213 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a520a0b5-deab-4825-9b36-329c399754f6" path="/var/lib/kubelet/pods/a520a0b5-deab-4825-9b36-329c399754f6/volumes" Oct 31 00:08:57.105894 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:57.105875 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a550117e-ad41-4e7c-998d-910a8d13d617" path="/var/lib/kubelet/pods/a550117e-ad41-4e7c-998d-910a8d13d617/volumes" Oct 31 00:08:57.106787 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:57.106770 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="afc963ee-cfd4-441b-9185-a6ae83276e19" path="/var/lib/kubelet/pods/afc963ee-cfd4-441b-9185-a6ae83276e19/volumes" Oct 31 00:08:57.107226 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:57.107213 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b7194267-cfef-4252-ae16-a194eccd2a1f" path="/var/lib/kubelet/pods/b7194267-cfef-4252-ae16-a194eccd2a1f/volumes" Oct 31 00:08:57.108933 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:57.108914 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b7eb6748-2997-4527-91b3-86900ea70eed" path="/var/lib/kubelet/pods/b7eb6748-2997-4527-91b3-86900ea70eed/volumes" Oct 31 00:08:57.218693 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:57.218594 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-0d75be-openshift-default-57f99d9cbc-smttn" event={"ID":"61bb8f6c-7dd6-4e87-ac44-5374dd11c0dc","Type":"ContainerStarted","Data":"294e83eb65d8d48cedf6847168059337f49522ad8337e6d31732cabd4f76172e"} Oct 31 00:08:57.218693 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:57.218659 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-0d75be-openshift-default-57f99d9cbc-smttn" event={"ID":"61bb8f6c-7dd6-4e87-ac44-5374dd11c0dc","Type":"ContainerStarted","Data":"e53c66abfd5c9cbb206a1f340d71f045df7db48f695aa9448d089ba541d0e7f2"} Oct 31 00:08:57.349169 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:57.349092 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w6-0d75be-openshift-default-57f99d9cbc-smttn" podStartSLOduration=3.349071973 podStartE2EDuration="3.349071973s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:08:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:08:57.34151941 +0000 UTC m=+1650.377850233" watchObservedRunningTime="2026-10-31 00:08:57.349071973 +0000 UTC m=+1650.385402797" Oct 31 00:08:57.506292 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:57.506200 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:08:57.696357 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:57.692673 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w6-0d75be-openshift-default-57f99d9cbc-smttn" Oct 31 00:08:57.730358 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:57.724755 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w6-0d75be-openshift-default-57f99d9cbc-smttn" Oct 31 00:08:58.230534 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:58.230498 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w6-0d75be-openshift-default-57f99d9cbc-smttn" Oct 31 00:08:58.231597 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:58.231575 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w6-0d75be-openshift-default-57f99d9cbc-smttn" Oct 31 00:08:58.975233 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:58.975197 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:08:59.232866 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:59.232767 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:08:59.397367 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:59.391036 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:08:59.441395 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:59.441362 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-09efcb6d-lt68p"] Oct 31 00:08:59.494481 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:59.493346 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-09efcb6d-lt68p"] Oct 31 00:08:59.509393 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:59.509321 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-09efcb6d-openshift-default-7c46fd677c-6wtzg"] Oct 31 00:08:59.509627 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:59.509602 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-09efcb6d-openshift-default-7c46fd677c-6wtzg" podUID="d403f39c-fdf7-4a18-b6c6-60425475307d" containerName="istio-proxy" containerID="cri-o://3592cc00707b3971a162844ca12c8950340c36d51e3906f5d9316c542b0cfff4" gracePeriod=30 Oct 31 00:08:59.555124 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:08:59.555088 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:09:00.366287 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:00.366253 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:09:01.028065 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:01.028028 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:09:01.672925 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:01.672878 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:09:01.677733 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:01.677701 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="96d362b8-d811-4ce5-abee-6b47f1482045" path="/var/lib/kubelet/pods/96d362b8-d811-4ce5-abee-6b47f1482045/volumes" Oct 31 00:09:01.678409 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:01.678388 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-d5360e-openshift-default-fbccfc549-d828w"] Oct 31 00:09:01.679916 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:01.679896 4927 cpu_manager.go:414] "containerMap: removing container" podUID="9f348596-394c-4bfb-ba79-4788da8fc6de" containerName="istio-proxy" Oct 31 00:09:01.680039 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:01.680028 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f348596-394c-4bfb-ba79-4788da8fc6de" containerName="istio-proxy" Oct 31 00:09:01.695867 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:01.692576 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-d5360e-openshift-default-fbccfc549-d828w"] Oct 31 00:09:01.695867 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:01.692714 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-d5360e-openshift-default-fbccfc549-d828w" Oct 31 00:09:01.797981 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:01.797946 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w0-d5360e-openshift-default-dockercfg-9ch2s\"" Oct 31 00:09:01.832174 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:01.832137 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:09:01.846305 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:01.846266 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37108b54-01b5-4e2c-add9-4e3aa1f3cd94-istiod-ca-cert\") pod \"e2e-shared-a-w0-d5360e-openshift-default-fbccfc549-d828w\" (UID: \"37108b54-01b5-4e2c-add9-4e3aa1f3cd94\") " pod="openshift-ingress/e2e-shared-a-w0-d5360e-openshift-default-fbccfc549-d828w" Oct 31 00:09:01.846491 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:01.846349 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37108b54-01b5-4e2c-add9-4e3aa1f3cd94-istio-envoy\") pod \"e2e-shared-a-w0-d5360e-openshift-default-fbccfc549-d828w\" (UID: \"37108b54-01b5-4e2c-add9-4e3aa1f3cd94\") " pod="openshift-ingress/e2e-shared-a-w0-d5360e-openshift-default-fbccfc549-d828w" Oct 31 00:09:01.846491 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:01.846449 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37108b54-01b5-4e2c-add9-4e3aa1f3cd94-istio-data\") pod \"e2e-shared-a-w0-d5360e-openshift-default-fbccfc549-d828w\" (UID: \"37108b54-01b5-4e2c-add9-4e3aa1f3cd94\") " pod="openshift-ingress/e2e-shared-a-w0-d5360e-openshift-default-fbccfc549-d828w" Oct 31 00:09:01.846574 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:01.846509 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qdvx\" (UniqueName: \"kubernetes.io/projected/37108b54-01b5-4e2c-add9-4e3aa1f3cd94-kube-api-access-2qdvx\") pod \"e2e-shared-a-w0-d5360e-openshift-default-fbccfc549-d828w\" (UID: \"37108b54-01b5-4e2c-add9-4e3aa1f3cd94\") " pod="openshift-ingress/e2e-shared-a-w0-d5360e-openshift-default-fbccfc549-d828w" Oct 31 00:09:01.846655 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:01.846627 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37108b54-01b5-4e2c-add9-4e3aa1f3cd94-workload-certs\") pod \"e2e-shared-a-w0-d5360e-openshift-default-fbccfc549-d828w\" (UID: \"37108b54-01b5-4e2c-add9-4e3aa1f3cd94\") " pod="openshift-ingress/e2e-shared-a-w0-d5360e-openshift-default-fbccfc549-d828w" Oct 31 00:09:01.846796 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:01.846779 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37108b54-01b5-4e2c-add9-4e3aa1f3cd94-istio-token\") pod \"e2e-shared-a-w0-d5360e-openshift-default-fbccfc549-d828w\" (UID: \"37108b54-01b5-4e2c-add9-4e3aa1f3cd94\") " pod="openshift-ingress/e2e-shared-a-w0-d5360e-openshift-default-fbccfc549-d828w" Oct 31 00:09:01.846837 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:01.846815 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37108b54-01b5-4e2c-add9-4e3aa1f3cd94-workload-socket\") pod \"e2e-shared-a-w0-d5360e-openshift-default-fbccfc549-d828w\" (UID: \"37108b54-01b5-4e2c-add9-4e3aa1f3cd94\") " pod="openshift-ingress/e2e-shared-a-w0-d5360e-openshift-default-fbccfc549-d828w" Oct 31 00:09:01.846943 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:01.846927 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37108b54-01b5-4e2c-add9-4e3aa1f3cd94-istio-podinfo\") pod \"e2e-shared-a-w0-d5360e-openshift-default-fbccfc549-d828w\" (UID: \"37108b54-01b5-4e2c-add9-4e3aa1f3cd94\") " pod="openshift-ingress/e2e-shared-a-w0-d5360e-openshift-default-fbccfc549-d828w" Oct 31 00:09:01.846982 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:01.846960 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37108b54-01b5-4e2c-add9-4e3aa1f3cd94-credential-socket\") pod \"e2e-shared-a-w0-d5360e-openshift-default-fbccfc549-d828w\" (UID: \"37108b54-01b5-4e2c-add9-4e3aa1f3cd94\") " pod="openshift-ingress/e2e-shared-a-w0-d5360e-openshift-default-fbccfc549-d828w" Oct 31 00:09:01.948051 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:01.947954 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37108b54-01b5-4e2c-add9-4e3aa1f3cd94-istio-token\") pod \"e2e-shared-a-w0-d5360e-openshift-default-fbccfc549-d828w\" (UID: \"37108b54-01b5-4e2c-add9-4e3aa1f3cd94\") " pod="openshift-ingress/e2e-shared-a-w0-d5360e-openshift-default-fbccfc549-d828w" Oct 31 00:09:01.948051 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:01.947998 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37108b54-01b5-4e2c-add9-4e3aa1f3cd94-workload-socket\") pod \"e2e-shared-a-w0-d5360e-openshift-default-fbccfc549-d828w\" (UID: \"37108b54-01b5-4e2c-add9-4e3aa1f3cd94\") " pod="openshift-ingress/e2e-shared-a-w0-d5360e-openshift-default-fbccfc549-d828w" Oct 31 00:09:01.948051 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:01.948040 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37108b54-01b5-4e2c-add9-4e3aa1f3cd94-istio-podinfo\") pod \"e2e-shared-a-w0-d5360e-openshift-default-fbccfc549-d828w\" (UID: \"37108b54-01b5-4e2c-add9-4e3aa1f3cd94\") " pod="openshift-ingress/e2e-shared-a-w0-d5360e-openshift-default-fbccfc549-d828w" Oct 31 00:09:01.948421 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:01.948069 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37108b54-01b5-4e2c-add9-4e3aa1f3cd94-credential-socket\") pod \"e2e-shared-a-w0-d5360e-openshift-default-fbccfc549-d828w\" (UID: \"37108b54-01b5-4e2c-add9-4e3aa1f3cd94\") " pod="openshift-ingress/e2e-shared-a-w0-d5360e-openshift-default-fbccfc549-d828w" Oct 31 00:09:01.948421 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:01.948141 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37108b54-01b5-4e2c-add9-4e3aa1f3cd94-istiod-ca-cert\") pod \"e2e-shared-a-w0-d5360e-openshift-default-fbccfc549-d828w\" (UID: \"37108b54-01b5-4e2c-add9-4e3aa1f3cd94\") " pod="openshift-ingress/e2e-shared-a-w0-d5360e-openshift-default-fbccfc549-d828w" Oct 31 00:09:01.948421 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:01.948187 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37108b54-01b5-4e2c-add9-4e3aa1f3cd94-istio-envoy\") pod \"e2e-shared-a-w0-d5360e-openshift-default-fbccfc549-d828w\" (UID: \"37108b54-01b5-4e2c-add9-4e3aa1f3cd94\") " pod="openshift-ingress/e2e-shared-a-w0-d5360e-openshift-default-fbccfc549-d828w" Oct 31 00:09:01.948421 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:01.948249 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37108b54-01b5-4e2c-add9-4e3aa1f3cd94-istio-data\") pod \"e2e-shared-a-w0-d5360e-openshift-default-fbccfc549-d828w\" (UID: \"37108b54-01b5-4e2c-add9-4e3aa1f3cd94\") " pod="openshift-ingress/e2e-shared-a-w0-d5360e-openshift-default-fbccfc549-d828w" Oct 31 00:09:01.948421 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:01.948313 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qdvx\" (UniqueName: \"kubernetes.io/projected/37108b54-01b5-4e2c-add9-4e3aa1f3cd94-kube-api-access-2qdvx\") pod \"e2e-shared-a-w0-d5360e-openshift-default-fbccfc549-d828w\" (UID: \"37108b54-01b5-4e2c-add9-4e3aa1f3cd94\") " pod="openshift-ingress/e2e-shared-a-w0-d5360e-openshift-default-fbccfc549-d828w" Oct 31 00:09:01.948421 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:01.948389 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37108b54-01b5-4e2c-add9-4e3aa1f3cd94-workload-certs\") pod \"e2e-shared-a-w0-d5360e-openshift-default-fbccfc549-d828w\" (UID: \"37108b54-01b5-4e2c-add9-4e3aa1f3cd94\") " pod="openshift-ingress/e2e-shared-a-w0-d5360e-openshift-default-fbccfc549-d828w" Oct 31 00:09:01.948737 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:01.948562 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37108b54-01b5-4e2c-add9-4e3aa1f3cd94-workload-socket\") pod \"e2e-shared-a-w0-d5360e-openshift-default-fbccfc549-d828w\" (UID: \"37108b54-01b5-4e2c-add9-4e3aa1f3cd94\") " pod="openshift-ingress/e2e-shared-a-w0-d5360e-openshift-default-fbccfc549-d828w" Oct 31 00:09:01.948737 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:01.948598 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37108b54-01b5-4e2c-add9-4e3aa1f3cd94-credential-socket\") pod \"e2e-shared-a-w0-d5360e-openshift-default-fbccfc549-d828w\" (UID: \"37108b54-01b5-4e2c-add9-4e3aa1f3cd94\") " pod="openshift-ingress/e2e-shared-a-w0-d5360e-openshift-default-fbccfc549-d828w" Oct 31 00:09:01.948737 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:01.948734 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37108b54-01b5-4e2c-add9-4e3aa1f3cd94-workload-certs\") pod \"e2e-shared-a-w0-d5360e-openshift-default-fbccfc549-d828w\" (UID: \"37108b54-01b5-4e2c-add9-4e3aa1f3cd94\") " pod="openshift-ingress/e2e-shared-a-w0-d5360e-openshift-default-fbccfc549-d828w" Oct 31 00:09:01.948894 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:01.948816 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37108b54-01b5-4e2c-add9-4e3aa1f3cd94-istio-data\") pod \"e2e-shared-a-w0-d5360e-openshift-default-fbccfc549-d828w\" (UID: \"37108b54-01b5-4e2c-add9-4e3aa1f3cd94\") " pod="openshift-ingress/e2e-shared-a-w0-d5360e-openshift-default-fbccfc549-d828w" Oct 31 00:09:01.949283 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:01.949265 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37108b54-01b5-4e2c-add9-4e3aa1f3cd94-istiod-ca-cert\") pod \"e2e-shared-a-w0-d5360e-openshift-default-fbccfc549-d828w\" (UID: \"37108b54-01b5-4e2c-add9-4e3aa1f3cd94\") " pod="openshift-ingress/e2e-shared-a-w0-d5360e-openshift-default-fbccfc549-d828w" Oct 31 00:09:01.951170 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:01.951143 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37108b54-01b5-4e2c-add9-4e3aa1f3cd94-istio-envoy\") pod \"e2e-shared-a-w0-d5360e-openshift-default-fbccfc549-d828w\" (UID: \"37108b54-01b5-4e2c-add9-4e3aa1f3cd94\") " pod="openshift-ingress/e2e-shared-a-w0-d5360e-openshift-default-fbccfc549-d828w" Oct 31 00:09:01.951321 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:01.951305 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37108b54-01b5-4e2c-add9-4e3aa1f3cd94-istio-podinfo\") pod \"e2e-shared-a-w0-d5360e-openshift-default-fbccfc549-d828w\" (UID: \"37108b54-01b5-4e2c-add9-4e3aa1f3cd94\") " pod="openshift-ingress/e2e-shared-a-w0-d5360e-openshift-default-fbccfc549-d828w" Oct 31 00:09:02.011247 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:02.011211 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37108b54-01b5-4e2c-add9-4e3aa1f3cd94-istio-token\") pod \"e2e-shared-a-w0-d5360e-openshift-default-fbccfc549-d828w\" (UID: \"37108b54-01b5-4e2c-add9-4e3aa1f3cd94\") " pod="openshift-ingress/e2e-shared-a-w0-d5360e-openshift-default-fbccfc549-d828w" Oct 31 00:09:02.011503 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:02.011478 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qdvx\" (UniqueName: \"kubernetes.io/projected/37108b54-01b5-4e2c-add9-4e3aa1f3cd94-kube-api-access-2qdvx\") pod \"e2e-shared-a-w0-d5360e-openshift-default-fbccfc549-d828w\" (UID: \"37108b54-01b5-4e2c-add9-4e3aa1f3cd94\") " pod="openshift-ingress/e2e-shared-a-w0-d5360e-openshift-default-fbccfc549-d828w" Oct 31 00:09:02.059750 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:02.059710 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-d5360e-openshift-default-fbccfc549-d828w" Oct 31 00:09:02.513392 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:02.513314 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-d5360e-openshift-default-fbccfc549-d828w"] Oct 31 00:09:02.513734 ip-10-0-141-180 kubenswrapper[4927]: W1031 00:09:02.513711 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37108b54_01b5_4e2c_add9_4e3aa1f3cd94.slice/crio-8dffbad23077f74dd5e2da8f2cab7363f373041631cc3b9a1dcb0919c4a07efc WatchSource:0}: Error finding container 8dffbad23077f74dd5e2da8f2cab7363f373041631cc3b9a1dcb0919c4a07efc: Status 404 returned error can't find the container with id 8dffbad23077f74dd5e2da8f2cab7363f373041631cc3b9a1dcb0919c4a07efc Oct 31 00:09:02.517318 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:02.517275 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:09:02.517418 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:02.517381 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:09:02.517460 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:02.517423 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:09:02.688136 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:02.688091 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-d5360e-openshift-default-fbccfc549-d828w" event={"ID":"37108b54-01b5-4e2c-add9-4e3aa1f3cd94","Type":"ContainerStarted","Data":"1afa8b2c79b4cf4621ad2544e1b2e946d465ab8c3fd31dbf19c27ef7e50497fb"} Oct 31 00:09:02.688671 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:02.688169 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-d5360e-openshift-default-fbccfc549-d828w" event={"ID":"37108b54-01b5-4e2c-add9-4e3aa1f3cd94","Type":"ContainerStarted","Data":"8dffbad23077f74dd5e2da8f2cab7363f373041631cc3b9a1dcb0919c4a07efc"} Oct 31 00:09:02.820646 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:02.820532 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-d5360e-openshift-default-fbccfc549-d828w" podStartSLOduration=2.820509307 podStartE2EDuration="2.820509307s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:09:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:09:02.814044871 +0000 UTC m=+1655.850375719" watchObservedRunningTime="2026-10-31 00:09:02.820509307 +0000 UTC m=+1655.856840129" Oct 31 00:09:03.068322 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:03.067670 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-d5360e-openshift-default-fbccfc549-d828w" Oct 31 00:09:03.068322 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:03.067812 4927 patch_prober.go:29] interesting pod/e2e-shared-a-w0-d5360e-openshift-default-fbccfc549-d828w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" start-of-body= Oct 31 00:09:03.068322 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:03.067857 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-d5360e-openshift-default-fbccfc549-d828w" podUID="37108b54-01b5-4e2c-add9-4e3aa1f3cd94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" Oct 31 00:09:03.650355 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:03.650183 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:09:03.911356 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:03.909816 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:09:04.074746 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:04.074674 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-d5360e-openshift-default-fbccfc549-d828w" Oct 31 00:09:04.109394 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:04.108437 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-71b76a-64f77d669f-hm2nn" Oct 31 00:09:04.515352 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:04.489178 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:09:04.717381 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:04.717351 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:09:04.830409 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:04.812050 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-0d75be-75f9855f5-kftq4"] Oct 31 00:09:04.903356 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:04.895351 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-0d75be-75f9855f5-kftq4" Oct 31 00:09:05.013381 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:05.013055 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w6-0d75be-dockercfg-vlhqb\"" Oct 31 00:09:05.013381 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:05.013165 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w6-0d75be\"" Oct 31 00:09:05.015842 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:05.014746 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b774910-1e71-40dd-9d93-026df8b5f9d2-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w6-0d75be-75f9855f5-kftq4\" (UID: \"0b774910-1e71-40dd-9d93-026df8b5f9d2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-0d75be-75f9855f5-kftq4" Oct 31 00:09:05.015842 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:05.015047 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5h5r\" (UniqueName: \"kubernetes.io/projected/0b774910-1e71-40dd-9d93-026df8b5f9d2-kube-api-access-r5h5r\") pod \"payload-pre-processing-e2e-shared-b-w6-0d75be-75f9855f5-kftq4\" (UID: \"0b774910-1e71-40dd-9d93-026df8b5f9d2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-0d75be-75f9855f5-kftq4" Oct 31 00:09:05.015842 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:05.015209 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-0d75be-75f9855f5-kftq4"] Oct 31 00:09:05.111623 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:05.111508 4927 generic.go:356] "Generic (PLEG): container finished" podID="d403f39c-fdf7-4a18-b6c6-60425475307d" containerID="3592cc00707b3971a162844ca12c8950340c36d51e3906f5d9316c542b0cfff4" exitCode=0 Oct 31 00:09:05.113156 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:05.113103 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-09efcb6d-openshift-default-7c46fd677c-6wtzg" event={"ID":"d403f39c-fdf7-4a18-b6c6-60425475307d","Type":"ContainerDied","Data":"3592cc00707b3971a162844ca12c8950340c36d51e3906f5d9316c542b0cfff4"} Oct 31 00:09:05.113311 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:05.113174 4927 scope.go:122] "RemoveContainer" containerID="813b90da108fa46f8396f51017fb233e30c03277f9b21df73a24237ba27c8b95" Oct 31 00:09:05.114782 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:05.114622 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-d5360e-openshift-default-fbccfc549-d828w" Oct 31 00:09:05.117633 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:05.117607 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b774910-1e71-40dd-9d93-026df8b5f9d2-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w6-0d75be-75f9855f5-kftq4\" (UID: \"0b774910-1e71-40dd-9d93-026df8b5f9d2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-0d75be-75f9855f5-kftq4" Oct 31 00:09:05.118051 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:05.118011 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5h5r\" (UniqueName: \"kubernetes.io/projected/0b774910-1e71-40dd-9d93-026df8b5f9d2-kube-api-access-r5h5r\") pod \"payload-pre-processing-e2e-shared-b-w6-0d75be-75f9855f5-kftq4\" (UID: \"0b774910-1e71-40dd-9d93-026df8b5f9d2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-0d75be-75f9855f5-kftq4" Oct 31 00:09:05.119893 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:05.119875 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-d5360e-openshift-default-fbccfc549-d828w" Oct 31 00:09:05.202887 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:05.202847 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b774910-1e71-40dd-9d93-026df8b5f9d2-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w6-0d75be-75f9855f5-kftq4\" (UID: \"0b774910-1e71-40dd-9d93-026df8b5f9d2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-0d75be-75f9855f5-kftq4" Oct 31 00:09:05.332404 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:05.328436 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-0d75be-7b66b84856-j28w4"] Oct 31 00:09:05.362077 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:05.361745 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5h5r\" (UniqueName: \"kubernetes.io/projected/0b774910-1e71-40dd-9d93-026df8b5f9d2-kube-api-access-r5h5r\") pod \"payload-pre-processing-e2e-shared-b-w6-0d75be-75f9855f5-kftq4\" (UID: \"0b774910-1e71-40dd-9d93-026df8b5f9d2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-0d75be-75f9855f5-kftq4" Oct 31 00:09:05.390230 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:05.390188 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-0d75be-7b66b84856-j28w4"] Oct 31 00:09:05.390435 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:05.390314 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-0d75be-7b66b84856-j28w4" Oct 31 00:09:05.435924 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:05.435894 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-0d75be-75f9855f5-kftq4" Oct 31 00:09:05.440998 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:05.440968 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nv8nj\" (UniqueName: \"kubernetes.io/projected/9fd4cd19-a1e6-455a-b36a-c9f58692b081-kube-api-access-nv8nj\") pod \"payload-processing-e2e-shared-b-w6-0d75be-7b66b84856-j28w4\" (UID: \"9fd4cd19-a1e6-455a-b36a-c9f58692b081\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-0d75be-7b66b84856-j28w4" Oct 31 00:09:05.441177 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:05.441155 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9fd4cd19-a1e6-455a-b36a-c9f58692b081-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w6-0d75be-7b66b84856-j28w4\" (UID: \"9fd4cd19-a1e6-455a-b36a-c9f58692b081\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-0d75be-7b66b84856-j28w4" Oct 31 00:09:05.523354 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:05.522123 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:09:05.542656 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:05.542559 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9fd4cd19-a1e6-455a-b36a-c9f58692b081-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w6-0d75be-7b66b84856-j28w4\" (UID: \"9fd4cd19-a1e6-455a-b36a-c9f58692b081\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-0d75be-7b66b84856-j28w4" Oct 31 00:09:05.544735 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:05.542884 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nv8nj\" (UniqueName: \"kubernetes.io/projected/9fd4cd19-a1e6-455a-b36a-c9f58692b081-kube-api-access-nv8nj\") pod \"payload-processing-e2e-shared-b-w6-0d75be-7b66b84856-j28w4\" (UID: \"9fd4cd19-a1e6-455a-b36a-c9f58692b081\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-0d75be-7b66b84856-j28w4" Oct 31 00:09:05.544735 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:05.544686 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9fd4cd19-a1e6-455a-b36a-c9f58692b081-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w6-0d75be-7b66b84856-j28w4\" (UID: \"9fd4cd19-a1e6-455a-b36a-c9f58692b081\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-0d75be-7b66b84856-j28w4" Oct 31 00:09:05.603223 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:05.603194 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-09efcb6d-openshift-default-7c46fd677c-6wtzg" Oct 31 00:09:05.606722 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:05.606695 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nv8nj\" (UniqueName: \"kubernetes.io/projected/9fd4cd19-a1e6-455a-b36a-c9f58692b081-kube-api-access-nv8nj\") pod \"payload-processing-e2e-shared-b-w6-0d75be-7b66b84856-j28w4\" (UID: \"9fd4cd19-a1e6-455a-b36a-c9f58692b081\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-0d75be-7b66b84856-j28w4" Oct 31 00:09:05.646751 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:05.644881 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d403f39c-fdf7-4a18-b6c6-60425475307d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d403f39c-fdf7-4a18-b6c6-60425475307d-credential-socket\") pod \"d403f39c-fdf7-4a18-b6c6-60425475307d\" (UID: \"d403f39c-fdf7-4a18-b6c6-60425475307d\") " Oct 31 00:09:05.646751 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:05.645013 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d403f39c-fdf7-4a18-b6c6-60425475307d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d403f39c-fdf7-4a18-b6c6-60425475307d-workload-certs\") pod \"d403f39c-fdf7-4a18-b6c6-60425475307d\" (UID: \"d403f39c-fdf7-4a18-b6c6-60425475307d\") " Oct 31 00:09:05.646751 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:05.645049 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d403f39c-fdf7-4a18-b6c6-60425475307d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d403f39c-fdf7-4a18-b6c6-60425475307d-istio-podinfo\") pod \"d403f39c-fdf7-4a18-b6c6-60425475307d\" (UID: \"d403f39c-fdf7-4a18-b6c6-60425475307d\") " Oct 31 00:09:05.646751 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:05.645081 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d403f39c-fdf7-4a18-b6c6-60425475307d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d403f39c-fdf7-4a18-b6c6-60425475307d-istio-data\") pod \"d403f39c-fdf7-4a18-b6c6-60425475307d\" (UID: \"d403f39c-fdf7-4a18-b6c6-60425475307d\") " Oct 31 00:09:05.646751 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:05.645216 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d403f39c-fdf7-4a18-b6c6-60425475307d-kube-api-access-2vv2n\" (UniqueName: \"kubernetes.io/projected/d403f39c-fdf7-4a18-b6c6-60425475307d-kube-api-access-2vv2n\") pod \"d403f39c-fdf7-4a18-b6c6-60425475307d\" (UID: \"d403f39c-fdf7-4a18-b6c6-60425475307d\") " Oct 31 00:09:05.646751 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:05.645364 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d403f39c-fdf7-4a18-b6c6-60425475307d-istio-token\" (UniqueName: \"kubernetes.io/projected/d403f39c-fdf7-4a18-b6c6-60425475307d-istio-token\") pod \"d403f39c-fdf7-4a18-b6c6-60425475307d\" (UID: \"d403f39c-fdf7-4a18-b6c6-60425475307d\") " Oct 31 00:09:05.646751 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:05.645499 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d403f39c-fdf7-4a18-b6c6-60425475307d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d403f39c-fdf7-4a18-b6c6-60425475307d-istiod-ca-cert\") pod \"d403f39c-fdf7-4a18-b6c6-60425475307d\" (UID: \"d403f39c-fdf7-4a18-b6c6-60425475307d\") " Oct 31 00:09:05.646751 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:05.645532 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d403f39c-fdf7-4a18-b6c6-60425475307d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d403f39c-fdf7-4a18-b6c6-60425475307d-workload-socket\") pod \"d403f39c-fdf7-4a18-b6c6-60425475307d\" (UID: \"d403f39c-fdf7-4a18-b6c6-60425475307d\") " Oct 31 00:09:05.646751 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:05.645631 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d403f39c-fdf7-4a18-b6c6-60425475307d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d403f39c-fdf7-4a18-b6c6-60425475307d-istio-envoy\") pod \"d403f39c-fdf7-4a18-b6c6-60425475307d\" (UID: \"d403f39c-fdf7-4a18-b6c6-60425475307d\") " Oct 31 00:09:05.647968 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:05.647651 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d403f39c-fdf7-4a18-b6c6-60425475307d-istio-data" pod "d403f39c-fdf7-4a18-b6c6-60425475307d" (UID: "d403f39c-fdf7-4a18-b6c6-60425475307d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:09:05.654728 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:05.649077 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d403f39c-fdf7-4a18-b6c6-60425475307d-istiod-ca-cert" pod "d403f39c-fdf7-4a18-b6c6-60425475307d" (UID: "d403f39c-fdf7-4a18-b6c6-60425475307d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:09:05.654728 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:05.654409 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d403f39c-fdf7-4a18-b6c6-60425475307d-workload-certs" pod "d403f39c-fdf7-4a18-b6c6-60425475307d" (UID: "d403f39c-fdf7-4a18-b6c6-60425475307d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:09:05.655184 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:05.654973 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d403f39c-fdf7-4a18-b6c6-60425475307d-credential-socket" pod "d403f39c-fdf7-4a18-b6c6-60425475307d" (UID: "d403f39c-fdf7-4a18-b6c6-60425475307d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:09:05.656914 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:05.655326 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d403f39c-fdf7-4a18-b6c6-60425475307d-workload-socket" pod "d403f39c-fdf7-4a18-b6c6-60425475307d" (UID: "d403f39c-fdf7-4a18-b6c6-60425475307d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:09:05.664990 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:05.664943 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d403f39c-fdf7-4a18-b6c6-60425475307d-istio-podinfo" pod "d403f39c-fdf7-4a18-b6c6-60425475307d" (UID: "d403f39c-fdf7-4a18-b6c6-60425475307d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:09:05.666705 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:05.666663 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d403f39c-fdf7-4a18-b6c6-60425475307d-istio-envoy" pod "d403f39c-fdf7-4a18-b6c6-60425475307d" (UID: "d403f39c-fdf7-4a18-b6c6-60425475307d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:09:05.667087 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:05.667042 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d403f39c-fdf7-4a18-b6c6-60425475307d-kube-api-access-2vv2n" pod "d403f39c-fdf7-4a18-b6c6-60425475307d" (UID: "d403f39c-fdf7-4a18-b6c6-60425475307d"). InnerVolumeSpecName "kube-api-access-2vv2n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:09:05.667087 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:05.667072 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d403f39c-fdf7-4a18-b6c6-60425475307d-istio-token" pod "d403f39c-fdf7-4a18-b6c6-60425475307d" (UID: "d403f39c-fdf7-4a18-b6c6-60425475307d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:09:05.731206 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:05.730853 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-0d75be-7b66b84856-j28w4" Oct 31 00:09:05.732296 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:05.732195 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:09:05.749826 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:05.749646 4927 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d403f39c-fdf7-4a18-b6c6-60425475307d-istio-token\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:09:05.749826 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:05.749679 4927 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d403f39c-fdf7-4a18-b6c6-60425475307d-istiod-ca-cert\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:09:05.749826 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:05.749695 4927 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d403f39c-fdf7-4a18-b6c6-60425475307d-workload-socket\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:09:05.749826 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:05.749708 4927 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d403f39c-fdf7-4a18-b6c6-60425475307d-istio-envoy\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:09:05.749826 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:05.749723 4927 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d403f39c-fdf7-4a18-b6c6-60425475307d-credential-socket\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:09:05.749826 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:05.749736 4927 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d403f39c-fdf7-4a18-b6c6-60425475307d-workload-certs\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:09:05.749826 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:05.749750 4927 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d403f39c-fdf7-4a18-b6c6-60425475307d-istio-podinfo\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:09:05.749826 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:05.749763 4927 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d403f39c-fdf7-4a18-b6c6-60425475307d-istio-data\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:09:05.749826 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:05.749777 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2vv2n\" (UniqueName: \"kubernetes.io/projected/d403f39c-fdf7-4a18-b6c6-60425475307d-kube-api-access-2vv2n\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:09:05.901233 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:05.895233 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-0d75be-75f9855f5-kftq4"] Oct 31 00:09:05.946057 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:05.937972 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:09:05.951939 ip-10-0-141-180 kubenswrapper[4927]: W1031 00:09:05.951456 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b774910_1e71_40dd_9d93_026df8b5f9d2.slice/crio-4d6b38331795f6656fae91b16a0fa9ef17425b60c1c2aff55963229136e766ad WatchSource:0}: Error finding container 4d6b38331795f6656fae91b16a0fa9ef17425b60c1c2aff55963229136e766ad: Status 404 returned error can't find the container with id 4d6b38331795f6656fae91b16a0fa9ef17425b60c1c2aff55963229136e766ad Oct 31 00:09:06.034314 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:06.034281 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7c8802f8-vtdwf"] Oct 31 00:09:06.041669 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:06.041640 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7c8802f8-vtdwf"] Oct 31 00:09:06.125781 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:06.125751 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-09efcb6d-openshift-default-7c46fd677c-6wtzg" Oct 31 00:09:06.125974 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:06.125779 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-09efcb6d-openshift-default-7c46fd677c-6wtzg" event={"ID":"d403f39c-fdf7-4a18-b6c6-60425475307d","Type":"ContainerDied","Data":"d005f23958edc1970c4d32bfd6721a0b6376a106b71858f2a2d7e622d8168655"} Oct 31 00:09:06.125974 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:06.125855 4927 scope.go:122] "RemoveContainer" containerID="3592cc00707b3971a162844ca12c8950340c36d51e3906f5d9316c542b0cfff4" Oct 31 00:09:06.128769 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:06.128743 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-0d75be-75f9855f5-kftq4" event={"ID":"0b774910-1e71-40dd-9d93-026df8b5f9d2","Type":"ContainerStarted","Data":"fccf0c779f70085c9950db7547d74987e208d9579ea0ba23f9ad709ba3e8ae09"} Oct 31 00:09:06.128906 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:06.128804 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-0d75be-75f9855f5-kftq4" event={"ID":"0b774910-1e71-40dd-9d93-026df8b5f9d2","Type":"ContainerStarted","Data":"4d6b38331795f6656fae91b16a0fa9ef17425b60c1c2aff55963229136e766ad"} Oct 31 00:09:06.129035 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:06.129021 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-0d75be-75f9855f5-kftq4" Oct 31 00:09:06.183837 ip-10-0-141-180 kubenswrapper[4927]: W1031 00:09:06.183801 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fd4cd19_a1e6_455a_b36a_c9f58692b081.slice/crio-d753420c050efb812f8d81e84f4a5c5d0084d2adf1946eabe46f1cede1245bcc WatchSource:0}: Error finding container d753420c050efb812f8d81e84f4a5c5d0084d2adf1946eabe46f1cede1245bcc: Status 404 returned error can't find the container with id d753420c050efb812f8d81e84f4a5c5d0084d2adf1946eabe46f1cede1245bcc Oct 31 00:09:06.184686 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:06.184659 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-0d75be-7b66b84856-j28w4"] Oct 31 00:09:06.298248 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:06.298214 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-09efcb6d-openshift-default-7c46fd677c-6wtzg"] Oct 31 00:09:06.306955 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:06.306927 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-09efcb6d-openshift-default-7c46fd677c-6wtzg"] Oct 31 00:09:06.423620 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:06.423558 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-0d75be-75f9855f5-kftq4" podStartSLOduration=2.423541074 podStartE2EDuration="2.423541074s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:09:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:09:06.422761638 +0000 UTC m=+1659.459092460" watchObservedRunningTime="2026-10-31 00:09:06.423541074 +0000 UTC m=+1659.459871894" Oct 31 00:09:06.562850 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:06.562758 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-7c8802f8-openshift-default-74499c95cc-v5qrl"] Oct 31 00:09:06.563080 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:06.563048 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-7c8802f8-openshift-default-74499c95cc-v5qrl" podUID="85533e83-d620-4bd3-8290-408d0c41164c" containerName="istio-proxy" containerID="cri-o://dbe5de481eae525d6ef94eb186fd51421928cd6be4e48593e55c0da8a239f7f1" gracePeriod=30 Oct 31 00:09:06.587711 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:06.587673 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:09:07.089717 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:07.089684 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e5c0632-30c2-4d21-a27b-eb252afe5747" path="/var/lib/kubelet/pods/7e5c0632-30c2-4d21-a27b-eb252afe5747/volumes" Oct 31 00:09:07.090258 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:07.090241 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d403f39c-fdf7-4a18-b6c6-60425475307d" path="/var/lib/kubelet/pods/d403f39c-fdf7-4a18-b6c6-60425475307d/volumes" Oct 31 00:09:07.140420 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:07.140386 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-0d75be-7b66b84856-j28w4" event={"ID":"9fd4cd19-a1e6-455a-b36a-c9f58692b081","Type":"ContainerStarted","Data":"da5ec8cc378a8c46a065d5f4772e22cf96a174e9c8b7517d5a74777de2b4de94"} Oct 31 00:09:07.140623 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:07.140448 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-0d75be-7b66b84856-j28w4" event={"ID":"9fd4cd19-a1e6-455a-b36a-c9f58692b081","Type":"ContainerStarted","Data":"d753420c050efb812f8d81e84f4a5c5d0084d2adf1946eabe46f1cede1245bcc"} Oct 31 00:09:07.140623 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:07.140505 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-0d75be-7b66b84856-j28w4" Oct 31 00:09:07.261940 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:07.261884 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-0d75be-7b66b84856-j28w4" podStartSLOduration=3.261867342 podStartE2EDuration="3.261867342s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:09:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:09:07.261376989 +0000 UTC m=+1660.297707834" watchObservedRunningTime="2026-10-31 00:09:07.261867342 +0000 UTC m=+1660.298198166" Oct 31 00:09:07.265640 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:07.265614 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:09:07.433394 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:07.425959 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:09:07.597449 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:07.597421 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:09:07.782656 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:07.782574 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:09:08.093119 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:08.093027 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-54ad8c-openshift-default-575c545fdf-gvfbr"] Oct 31 00:09:08.094774 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:08.094744 4927 cpu_manager.go:414] "containerMap: removing container" podUID="d403f39c-fdf7-4a18-b6c6-60425475307d" containerName="istio-proxy" Oct 31 00:09:08.110274 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:08.110237 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d403f39c-fdf7-4a18-b6c6-60425475307d" containerName="istio-proxy" Oct 31 00:09:08.129502 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:08.110573 4927 cpu_manager.go:414] "containerMap: removing container" podUID="d403f39c-fdf7-4a18-b6c6-60425475307d" containerName="istio-proxy" Oct 31 00:09:08.129756 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:08.129726 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d403f39c-fdf7-4a18-b6c6-60425475307d" containerName="istio-proxy" Oct 31 00:09:08.152358 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:08.130114 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="d403f39c-fdf7-4a18-b6c6-60425475307d" containerName="istio-proxy" Oct 31 00:09:08.169366 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:08.158606 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-54ad8c-openshift-default-575c545fdf-gvfbr"] Oct 31 00:09:08.183357 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:08.173523 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-54ad8c-openshift-default-575c545fdf-gvfbr" Oct 31 00:09:08.275226 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:08.211959 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:09:08.275226 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:08.263201 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85098bb1-0c3e-4ecb-a771-3f57503a7cd8-istio-envoy\") pod \"e2e-shared-a-w4-54ad8c-openshift-default-575c545fdf-gvfbr\" (UID: \"85098bb1-0c3e-4ecb-a771-3f57503a7cd8\") " pod="openshift-ingress/e2e-shared-a-w4-54ad8c-openshift-default-575c545fdf-gvfbr" Oct 31 00:09:08.275226 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:08.263253 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85098bb1-0c3e-4ecb-a771-3f57503a7cd8-istiod-ca-cert\") pod \"e2e-shared-a-w4-54ad8c-openshift-default-575c545fdf-gvfbr\" (UID: \"85098bb1-0c3e-4ecb-a771-3f57503a7cd8\") " pod="openshift-ingress/e2e-shared-a-w4-54ad8c-openshift-default-575c545fdf-gvfbr" Oct 31 00:09:08.275226 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:08.263306 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85098bb1-0c3e-4ecb-a771-3f57503a7cd8-istio-data\") pod \"e2e-shared-a-w4-54ad8c-openshift-default-575c545fdf-gvfbr\" (UID: \"85098bb1-0c3e-4ecb-a771-3f57503a7cd8\") " pod="openshift-ingress/e2e-shared-a-w4-54ad8c-openshift-default-575c545fdf-gvfbr" Oct 31 00:09:08.275226 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:08.263421 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85098bb1-0c3e-4ecb-a771-3f57503a7cd8-istio-token\") pod \"e2e-shared-a-w4-54ad8c-openshift-default-575c545fdf-gvfbr\" (UID: \"85098bb1-0c3e-4ecb-a771-3f57503a7cd8\") " pod="openshift-ingress/e2e-shared-a-w4-54ad8c-openshift-default-575c545fdf-gvfbr" Oct 31 00:09:08.275226 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:08.263479 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85098bb1-0c3e-4ecb-a771-3f57503a7cd8-workload-certs\") pod \"e2e-shared-a-w4-54ad8c-openshift-default-575c545fdf-gvfbr\" (UID: \"85098bb1-0c3e-4ecb-a771-3f57503a7cd8\") " pod="openshift-ingress/e2e-shared-a-w4-54ad8c-openshift-default-575c545fdf-gvfbr" Oct 31 00:09:08.275226 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:08.263637 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85098bb1-0c3e-4ecb-a771-3f57503a7cd8-istio-podinfo\") pod \"e2e-shared-a-w4-54ad8c-openshift-default-575c545fdf-gvfbr\" (UID: \"85098bb1-0c3e-4ecb-a771-3f57503a7cd8\") " pod="openshift-ingress/e2e-shared-a-w4-54ad8c-openshift-default-575c545fdf-gvfbr" Oct 31 00:09:08.275226 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:08.263678 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85098bb1-0c3e-4ecb-a771-3f57503a7cd8-workload-socket\") pod \"e2e-shared-a-w4-54ad8c-openshift-default-575c545fdf-gvfbr\" (UID: \"85098bb1-0c3e-4ecb-a771-3f57503a7cd8\") " pod="openshift-ingress/e2e-shared-a-w4-54ad8c-openshift-default-575c545fdf-gvfbr" Oct 31 00:09:08.275226 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:08.263717 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drxp5\" (UniqueName: \"kubernetes.io/projected/85098bb1-0c3e-4ecb-a771-3f57503a7cd8-kube-api-access-drxp5\") pod \"e2e-shared-a-w4-54ad8c-openshift-default-575c545fdf-gvfbr\" (UID: \"85098bb1-0c3e-4ecb-a771-3f57503a7cd8\") " pod="openshift-ingress/e2e-shared-a-w4-54ad8c-openshift-default-575c545fdf-gvfbr" Oct 31 00:09:08.275226 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:08.263802 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85098bb1-0c3e-4ecb-a771-3f57503a7cd8-credential-socket\") pod \"e2e-shared-a-w4-54ad8c-openshift-default-575c545fdf-gvfbr\" (UID: \"85098bb1-0c3e-4ecb-a771-3f57503a7cd8\") " pod="openshift-ingress/e2e-shared-a-w4-54ad8c-openshift-default-575c545fdf-gvfbr" Oct 31 00:09:08.350416 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:08.339818 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w4-54ad8c-openshift-default-dockercfg-ktf2z\"" Oct 31 00:09:08.373358 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:08.366857 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85098bb1-0c3e-4ecb-a771-3f57503a7cd8-credential-socket\") pod \"e2e-shared-a-w4-54ad8c-openshift-default-575c545fdf-gvfbr\" (UID: \"85098bb1-0c3e-4ecb-a771-3f57503a7cd8\") " pod="openshift-ingress/e2e-shared-a-w4-54ad8c-openshift-default-575c545fdf-gvfbr" Oct 31 00:09:08.373358 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:08.367082 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85098bb1-0c3e-4ecb-a771-3f57503a7cd8-istio-envoy\") pod \"e2e-shared-a-w4-54ad8c-openshift-default-575c545fdf-gvfbr\" (UID: \"85098bb1-0c3e-4ecb-a771-3f57503a7cd8\") " pod="openshift-ingress/e2e-shared-a-w4-54ad8c-openshift-default-575c545fdf-gvfbr" Oct 31 00:09:08.373358 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:08.367122 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85098bb1-0c3e-4ecb-a771-3f57503a7cd8-istiod-ca-cert\") pod \"e2e-shared-a-w4-54ad8c-openshift-default-575c545fdf-gvfbr\" (UID: \"85098bb1-0c3e-4ecb-a771-3f57503a7cd8\") " pod="openshift-ingress/e2e-shared-a-w4-54ad8c-openshift-default-575c545fdf-gvfbr" Oct 31 00:09:08.373358 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:08.367168 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85098bb1-0c3e-4ecb-a771-3f57503a7cd8-istio-data\") pod \"e2e-shared-a-w4-54ad8c-openshift-default-575c545fdf-gvfbr\" (UID: \"85098bb1-0c3e-4ecb-a771-3f57503a7cd8\") " pod="openshift-ingress/e2e-shared-a-w4-54ad8c-openshift-default-575c545fdf-gvfbr" Oct 31 00:09:08.373358 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:08.367221 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85098bb1-0c3e-4ecb-a771-3f57503a7cd8-istio-token\") pod \"e2e-shared-a-w4-54ad8c-openshift-default-575c545fdf-gvfbr\" (UID: \"85098bb1-0c3e-4ecb-a771-3f57503a7cd8\") " pod="openshift-ingress/e2e-shared-a-w4-54ad8c-openshift-default-575c545fdf-gvfbr" Oct 31 00:09:08.373358 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:08.367529 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85098bb1-0c3e-4ecb-a771-3f57503a7cd8-workload-certs\") pod \"e2e-shared-a-w4-54ad8c-openshift-default-575c545fdf-gvfbr\" (UID: \"85098bb1-0c3e-4ecb-a771-3f57503a7cd8\") " pod="openshift-ingress/e2e-shared-a-w4-54ad8c-openshift-default-575c545fdf-gvfbr" Oct 31 00:09:08.373358 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:08.367708 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85098bb1-0c3e-4ecb-a771-3f57503a7cd8-istio-podinfo\") pod \"e2e-shared-a-w4-54ad8c-openshift-default-575c545fdf-gvfbr\" (UID: \"85098bb1-0c3e-4ecb-a771-3f57503a7cd8\") " pod="openshift-ingress/e2e-shared-a-w4-54ad8c-openshift-default-575c545fdf-gvfbr" Oct 31 00:09:08.373358 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:08.367751 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85098bb1-0c3e-4ecb-a771-3f57503a7cd8-workload-socket\") pod \"e2e-shared-a-w4-54ad8c-openshift-default-575c545fdf-gvfbr\" (UID: \"85098bb1-0c3e-4ecb-a771-3f57503a7cd8\") " pod="openshift-ingress/e2e-shared-a-w4-54ad8c-openshift-default-575c545fdf-gvfbr" Oct 31 00:09:08.373358 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:08.367790 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drxp5\" (UniqueName: \"kubernetes.io/projected/85098bb1-0c3e-4ecb-a771-3f57503a7cd8-kube-api-access-drxp5\") pod \"e2e-shared-a-w4-54ad8c-openshift-default-575c545fdf-gvfbr\" (UID: \"85098bb1-0c3e-4ecb-a771-3f57503a7cd8\") " pod="openshift-ingress/e2e-shared-a-w4-54ad8c-openshift-default-575c545fdf-gvfbr" Oct 31 00:09:08.373358 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:08.367873 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85098bb1-0c3e-4ecb-a771-3f57503a7cd8-workload-certs\") pod \"e2e-shared-a-w4-54ad8c-openshift-default-575c545fdf-gvfbr\" (UID: \"85098bb1-0c3e-4ecb-a771-3f57503a7cd8\") " pod="openshift-ingress/e2e-shared-a-w4-54ad8c-openshift-default-575c545fdf-gvfbr" Oct 31 00:09:08.373358 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:08.368157 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85098bb1-0c3e-4ecb-a771-3f57503a7cd8-workload-socket\") pod \"e2e-shared-a-w4-54ad8c-openshift-default-575c545fdf-gvfbr\" (UID: \"85098bb1-0c3e-4ecb-a771-3f57503a7cd8\") " pod="openshift-ingress/e2e-shared-a-w4-54ad8c-openshift-default-575c545fdf-gvfbr" Oct 31 00:09:08.373358 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:08.368703 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85098bb1-0c3e-4ecb-a771-3f57503a7cd8-credential-socket\") pod \"e2e-shared-a-w4-54ad8c-openshift-default-575c545fdf-gvfbr\" (UID: \"85098bb1-0c3e-4ecb-a771-3f57503a7cd8\") " pod="openshift-ingress/e2e-shared-a-w4-54ad8c-openshift-default-575c545fdf-gvfbr" Oct 31 00:09:08.373358 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:08.370165 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85098bb1-0c3e-4ecb-a771-3f57503a7cd8-istiod-ca-cert\") pod \"e2e-shared-a-w4-54ad8c-openshift-default-575c545fdf-gvfbr\" (UID: \"85098bb1-0c3e-4ecb-a771-3f57503a7cd8\") " pod="openshift-ingress/e2e-shared-a-w4-54ad8c-openshift-default-575c545fdf-gvfbr" Oct 31 00:09:08.381404 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:08.380360 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85098bb1-0c3e-4ecb-a771-3f57503a7cd8-istio-data\") pod \"e2e-shared-a-w4-54ad8c-openshift-default-575c545fdf-gvfbr\" (UID: \"85098bb1-0c3e-4ecb-a771-3f57503a7cd8\") " pod="openshift-ingress/e2e-shared-a-w4-54ad8c-openshift-default-575c545fdf-gvfbr" Oct 31 00:09:08.409472 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:08.392921 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85098bb1-0c3e-4ecb-a771-3f57503a7cd8-istio-podinfo\") pod \"e2e-shared-a-w4-54ad8c-openshift-default-575c545fdf-gvfbr\" (UID: \"85098bb1-0c3e-4ecb-a771-3f57503a7cd8\") " pod="openshift-ingress/e2e-shared-a-w4-54ad8c-openshift-default-575c545fdf-gvfbr" Oct 31 00:09:08.409472 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:08.400071 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:09:08.444349 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:08.444294 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85098bb1-0c3e-4ecb-a771-3f57503a7cd8-istio-envoy\") pod \"e2e-shared-a-w4-54ad8c-openshift-default-575c545fdf-gvfbr\" (UID: \"85098bb1-0c3e-4ecb-a771-3f57503a7cd8\") " pod="openshift-ingress/e2e-shared-a-w4-54ad8c-openshift-default-575c545fdf-gvfbr" Oct 31 00:09:08.448266 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:08.448187 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85098bb1-0c3e-4ecb-a771-3f57503a7cd8-istio-token\") pod \"e2e-shared-a-w4-54ad8c-openshift-default-575c545fdf-gvfbr\" (UID: \"85098bb1-0c3e-4ecb-a771-3f57503a7cd8\") " pod="openshift-ingress/e2e-shared-a-w4-54ad8c-openshift-default-575c545fdf-gvfbr" Oct 31 00:09:08.454554 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:08.454142 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-drxp5\" (UniqueName: \"kubernetes.io/projected/85098bb1-0c3e-4ecb-a771-3f57503a7cd8-kube-api-access-drxp5\") pod \"e2e-shared-a-w4-54ad8c-openshift-default-575c545fdf-gvfbr\" (UID: \"85098bb1-0c3e-4ecb-a771-3f57503a7cd8\") " pod="openshift-ingress/e2e-shared-a-w4-54ad8c-openshift-default-575c545fdf-gvfbr" Oct 31 00:09:08.740672 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:08.740633 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-54ad8c-openshift-default-575c545fdf-gvfbr" Oct 31 00:09:09.198292 ip-10-0-141-180 kubenswrapper[4927]: W1031 00:09:09.198255 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85098bb1_0c3e_4ecb_a771_3f57503a7cd8.slice/crio-0a4eba3644d482523efed24cb40a74b3ac226d24591a0d958a79e2ce7513b4ab WatchSource:0}: Error finding container 0a4eba3644d482523efed24cb40a74b3ac226d24591a0d958a79e2ce7513b4ab: Status 404 returned error can't find the container with id 0a4eba3644d482523efed24cb40a74b3ac226d24591a0d958a79e2ce7513b4ab Oct 31 00:09:09.198764 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:09.198478 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-54ad8c-openshift-default-575c545fdf-gvfbr"] Oct 31 00:09:09.203159 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:09.202207 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:09:09.203159 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:09.202406 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:09:09.203159 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:09.202533 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:09:09.554623 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:09.554536 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-54ad8c-openshift-default-575c545fdf-gvfbr" event={"ID":"85098bb1-0c3e-4ecb-a771-3f57503a7cd8","Type":"ContainerStarted","Data":"7b5c45f7b944ca39203a0d1ceca2181c2fe8985765f8f818ff71500e3dbcfc04"} Oct 31 00:09:09.554772 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:09.554632 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-54ad8c-openshift-default-575c545fdf-gvfbr" event={"ID":"85098bb1-0c3e-4ecb-a771-3f57503a7cd8","Type":"ContainerStarted","Data":"0a4eba3644d482523efed24cb40a74b3ac226d24591a0d958a79e2ce7513b4ab"} Oct 31 00:09:09.682158 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:09.682102 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-54ad8c-openshift-default-575c545fdf-gvfbr" podStartSLOduration=2.682085101 podStartE2EDuration="2.682085101s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:09:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:09:09.678821056 +0000 UTC m=+1662.715151889" watchObservedRunningTime="2026-10-31 00:09:09.682085101 +0000 UTC m=+1662.718415922" Oct 31 00:09:09.741182 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:09.741151 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-54ad8c-openshift-default-575c545fdf-gvfbr" Oct 31 00:09:10.747226 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:10.747197 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-54ad8c-openshift-default-575c545fdf-gvfbr" Oct 31 00:09:10.830783 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:10.830746 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:09:11.309971 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:11.309924 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:09:11.545404 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:11.545361 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:09:11.727943 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:11.727868 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-54ad8c-openshift-default-575c545fdf-gvfbr" Oct 31 00:09:11.733633 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:11.732072 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-54ad8c-openshift-default-575c545fdf-gvfbr" Oct 31 00:09:11.738627 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:11.738600 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-d5360e-b6fb568dc-htsps"] Oct 31 00:09:11.740838 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:11.740810 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="d403f39c-fdf7-4a18-b6c6-60425475307d" containerName="istio-proxy" Oct 31 00:09:11.767363 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:11.767035 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-d5360e-b6fb568dc-htsps" Oct 31 00:09:11.829356 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:11.827742 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w0-d5360e\"" Oct 31 00:09:11.844811 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:11.834552 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w0-d5360e-dockercfg-c84bm\"" Oct 31 00:09:11.844811 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:11.835515 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-d5360e-b6fb568dc-htsps"] Oct 31 00:09:11.844811 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:11.835884 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64551325-0270-49ec-bbc7-748cd5c3611e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-d5360e-b6fb568dc-htsps\" (UID: \"64551325-0270-49ec-bbc7-748cd5c3611e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-d5360e-b6fb568dc-htsps" Oct 31 00:09:11.844811 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:11.835996 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhvfx\" (UniqueName: \"kubernetes.io/projected/64551325-0270-49ec-bbc7-748cd5c3611e-kube-api-access-dhvfx\") pod \"payload-pre-processing-e2e-shared-a-w0-d5360e-b6fb568dc-htsps\" (UID: \"64551325-0270-49ec-bbc7-748cd5c3611e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-d5360e-b6fb568dc-htsps" Oct 31 00:09:12.009355 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:11.998447 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64551325-0270-49ec-bbc7-748cd5c3611e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-d5360e-b6fb568dc-htsps\" (UID: \"64551325-0270-49ec-bbc7-748cd5c3611e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-d5360e-b6fb568dc-htsps" Oct 31 00:09:12.009355 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:11.998552 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhvfx\" (UniqueName: \"kubernetes.io/projected/64551325-0270-49ec-bbc7-748cd5c3611e-kube-api-access-dhvfx\") pod \"payload-pre-processing-e2e-shared-a-w0-d5360e-b6fb568dc-htsps\" (UID: \"64551325-0270-49ec-bbc7-748cd5c3611e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-d5360e-b6fb568dc-htsps" Oct 31 00:09:12.009355 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:11.999819 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64551325-0270-49ec-bbc7-748cd5c3611e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-d5360e-b6fb568dc-htsps\" (UID: \"64551325-0270-49ec-bbc7-748cd5c3611e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-d5360e-b6fb568dc-htsps" Oct 31 00:09:12.133720 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:12.120231 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhvfx\" (UniqueName: \"kubernetes.io/projected/64551325-0270-49ec-bbc7-748cd5c3611e-kube-api-access-dhvfx\") pod \"payload-pre-processing-e2e-shared-a-w0-d5360e-b6fb568dc-htsps\" (UID: \"64551325-0270-49ec-bbc7-748cd5c3611e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-d5360e-b6fb568dc-htsps" Oct 31 00:09:12.133720 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:12.126371 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-d5360e-854fc9bc5d-mc76v"] Oct 31 00:09:12.155687 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:12.143093 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-d5360e-854fc9bc5d-mc76v" Oct 31 00:09:12.155687 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:12.148421 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:09:12.328354 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:12.326556 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lhrht\" (UniqueName: \"kubernetes.io/projected/e27553d8-0637-46a1-bd84-186e69ba3f15-kube-api-access-lhrht\") pod \"payload-processing-e2e-shared-a-w0-d5360e-854fc9bc5d-mc76v\" (UID: \"e27553d8-0637-46a1-bd84-186e69ba3f15\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-d5360e-854fc9bc5d-mc76v" Oct 31 00:09:12.328354 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:12.326896 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e27553d8-0637-46a1-bd84-186e69ba3f15-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-d5360e-854fc9bc5d-mc76v\" (UID: \"e27553d8-0637-46a1-bd84-186e69ba3f15\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-d5360e-854fc9bc5d-mc76v" Oct 31 00:09:12.476535 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:12.420237 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-d5360e-854fc9bc5d-mc76v"] Oct 31 00:09:12.476535 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:12.431263 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e27553d8-0637-46a1-bd84-186e69ba3f15-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-d5360e-854fc9bc5d-mc76v\" (UID: \"e27553d8-0637-46a1-bd84-186e69ba3f15\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-d5360e-854fc9bc5d-mc76v" Oct 31 00:09:12.476535 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:12.431362 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhrht\" (UniqueName: \"kubernetes.io/projected/e27553d8-0637-46a1-bd84-186e69ba3f15-kube-api-access-lhrht\") pod \"payload-processing-e2e-shared-a-w0-d5360e-854fc9bc5d-mc76v\" (UID: \"e27553d8-0637-46a1-bd84-186e69ba3f15\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-d5360e-854fc9bc5d-mc76v" Oct 31 00:09:12.476535 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:12.436276 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e27553d8-0637-46a1-bd84-186e69ba3f15-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-d5360e-854fc9bc5d-mc76v\" (UID: \"e27553d8-0637-46a1-bd84-186e69ba3f15\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-d5360e-854fc9bc5d-mc76v" Oct 31 00:09:12.476535 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:12.437749 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-a251fdfb/e2e-lifecycle-model-a251fdfb-kserve-5966998657-7wkbl"] Oct 31 00:09:12.593884 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:12.592113 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-a251fdfb/e2e-lifecycle-model-a251fdfb-kserve-5966998657-7wkbl" Oct 31 00:09:12.596523 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:12.591979 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-a251fdfb/e2e-lifecycle-model-a251fdfb-kserve-5966998657-7wkbl"] Oct 31 00:09:12.639412 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:12.614484 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhrht\" (UniqueName: \"kubernetes.io/projected/e27553d8-0637-46a1-bd84-186e69ba3f15-kube-api-access-lhrht\") pod \"payload-processing-e2e-shared-a-w0-d5360e-854fc9bc5d-mc76v\" (UID: \"e27553d8-0637-46a1-bd84-186e69ba3f15\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-d5360e-854fc9bc5d-mc76v" Oct 31 00:09:12.650252 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:12.649893 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9c18450-ac2a-4e6a-a712-00c2a755535d-dshm\") pod \"e2e-lifecycle-model-a251fdfb-kserve-5966998657-7wkbl\" (UID: \"e9c18450-ac2a-4e6a-a712-00c2a755535d\") " pod="ai-tenant-e2e-mt-a251fdfb/e2e-lifecycle-model-a251fdfb-kserve-5966998657-7wkbl" Oct 31 00:09:12.650252 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:12.649972 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9c18450-ac2a-4e6a-a712-00c2a755535d-model-cache\") pod \"e2e-lifecycle-model-a251fdfb-kserve-5966998657-7wkbl\" (UID: \"e9c18450-ac2a-4e6a-a712-00c2a755535d\") " pod="ai-tenant-e2e-mt-a251fdfb/e2e-lifecycle-model-a251fdfb-kserve-5966998657-7wkbl" Oct 31 00:09:12.650252 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:12.650009 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9c18450-ac2a-4e6a-a712-00c2a755535d-tmp-dir\") pod \"e2e-lifecycle-model-a251fdfb-kserve-5966998657-7wkbl\" (UID: \"e9c18450-ac2a-4e6a-a712-00c2a755535d\") " pod="ai-tenant-e2e-mt-a251fdfb/e2e-lifecycle-model-a251fdfb-kserve-5966998657-7wkbl" Oct 31 00:09:12.650252 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:12.650102 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9c18450-ac2a-4e6a-a712-00c2a755535d-tls-certs\") pod \"e2e-lifecycle-model-a251fdfb-kserve-5966998657-7wkbl\" (UID: \"e9c18450-ac2a-4e6a-a712-00c2a755535d\") " pod="ai-tenant-e2e-mt-a251fdfb/e2e-lifecycle-model-a251fdfb-kserve-5966998657-7wkbl" Oct 31 00:09:12.667254 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:12.651867 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-a251fdfb\"/\"openshift-service-ca.crt\"" Oct 31 00:09:12.667254 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:12.652127 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-a251fdfb\"/\"default-dockercfg-wnnfz\"" Oct 31 00:09:12.667254 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:12.652267 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-a251fdfb\"/\"e2e-lifecycle-model-a251fdfb-kserve-self-signed-certs\"" Oct 31 00:09:12.667254 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:12.652386 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-a251fdfb\"/\"kube-root-ca.crt\"" Oct 31 00:09:12.669241 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:12.668701 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e9c18450-ac2a-4e6a-a712-00c2a755535d-home\") pod \"e2e-lifecycle-model-a251fdfb-kserve-5966998657-7wkbl\" (UID: \"e9c18450-ac2a-4e6a-a712-00c2a755535d\") " pod="ai-tenant-e2e-mt-a251fdfb/e2e-lifecycle-model-a251fdfb-kserve-5966998657-7wkbl" Oct 31 00:09:12.669241 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:12.668927 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2p984\" (UniqueName: \"kubernetes.io/projected/e9c18450-ac2a-4e6a-a712-00c2a755535d-kube-api-access-2p984\") pod \"e2e-lifecycle-model-a251fdfb-kserve-5966998657-7wkbl\" (UID: \"e9c18450-ac2a-4e6a-a712-00c2a755535d\") " pod="ai-tenant-e2e-mt-a251fdfb/e2e-lifecycle-model-a251fdfb-kserve-5966998657-7wkbl" Oct 31 00:09:12.714387 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:12.701082 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-d5360e-b6fb568dc-htsps" Oct 31 00:09:12.722675 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:12.718557 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-d5360e-854fc9bc5d-mc76v" Oct 31 00:09:12.745097 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:12.745066 4927 generic.go:356] "Generic (PLEG): container finished" podID="85533e83-d620-4bd3-8290-408d0c41164c" containerID="dbe5de481eae525d6ef94eb186fd51421928cd6be4e48593e55c0da8a239f7f1" exitCode=0 Oct 31 00:09:12.745202 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:12.745150 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-7c8802f8-openshift-default-74499c95cc-v5qrl" event={"ID":"85533e83-d620-4bd3-8290-408d0c41164c","Type":"ContainerDied","Data":"dbe5de481eae525d6ef94eb186fd51421928cd6be4e48593e55c0da8a239f7f1"} Oct 31 00:09:12.745264 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:12.745242 4927 scope.go:122] "RemoveContainer" containerID="6459ff1422fee57db06f23be1010e336bf13a79c0d217e76dcff62c9d8f7296c" Oct 31 00:09:12.770944 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:12.770850 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2p984\" (UniqueName: \"kubernetes.io/projected/e9c18450-ac2a-4e6a-a712-00c2a755535d-kube-api-access-2p984\") pod \"e2e-lifecycle-model-a251fdfb-kserve-5966998657-7wkbl\" (UID: \"e9c18450-ac2a-4e6a-a712-00c2a755535d\") " pod="ai-tenant-e2e-mt-a251fdfb/e2e-lifecycle-model-a251fdfb-kserve-5966998657-7wkbl" Oct 31 00:09:12.771298 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:12.771141 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9c18450-ac2a-4e6a-a712-00c2a755535d-dshm\") pod \"e2e-lifecycle-model-a251fdfb-kserve-5966998657-7wkbl\" (UID: \"e9c18450-ac2a-4e6a-a712-00c2a755535d\") " pod="ai-tenant-e2e-mt-a251fdfb/e2e-lifecycle-model-a251fdfb-kserve-5966998657-7wkbl" Oct 31 00:09:12.771298 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:12.771193 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9c18450-ac2a-4e6a-a712-00c2a755535d-model-cache\") pod \"e2e-lifecycle-model-a251fdfb-kserve-5966998657-7wkbl\" (UID: \"e9c18450-ac2a-4e6a-a712-00c2a755535d\") " pod="ai-tenant-e2e-mt-a251fdfb/e2e-lifecycle-model-a251fdfb-kserve-5966998657-7wkbl" Oct 31 00:09:12.771298 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:12.771223 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9c18450-ac2a-4e6a-a712-00c2a755535d-tmp-dir\") pod \"e2e-lifecycle-model-a251fdfb-kserve-5966998657-7wkbl\" (UID: \"e9c18450-ac2a-4e6a-a712-00c2a755535d\") " pod="ai-tenant-e2e-mt-a251fdfb/e2e-lifecycle-model-a251fdfb-kserve-5966998657-7wkbl" Oct 31 00:09:12.771934 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:12.771367 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9c18450-ac2a-4e6a-a712-00c2a755535d-tls-certs\") pod \"e2e-lifecycle-model-a251fdfb-kserve-5966998657-7wkbl\" (UID: \"e9c18450-ac2a-4e6a-a712-00c2a755535d\") " pod="ai-tenant-e2e-mt-a251fdfb/e2e-lifecycle-model-a251fdfb-kserve-5966998657-7wkbl" Oct 31 00:09:12.771934 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:12.771486 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e9c18450-ac2a-4e6a-a712-00c2a755535d-home\") pod \"e2e-lifecycle-model-a251fdfb-kserve-5966998657-7wkbl\" (UID: \"e9c18450-ac2a-4e6a-a712-00c2a755535d\") " pod="ai-tenant-e2e-mt-a251fdfb/e2e-lifecycle-model-a251fdfb-kserve-5966998657-7wkbl" Oct 31 00:09:12.771934 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:12.771530 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9c18450-ac2a-4e6a-a712-00c2a755535d-tmp-dir\") pod \"e2e-lifecycle-model-a251fdfb-kserve-5966998657-7wkbl\" (UID: \"e9c18450-ac2a-4e6a-a712-00c2a755535d\") " pod="ai-tenant-e2e-mt-a251fdfb/e2e-lifecycle-model-a251fdfb-kserve-5966998657-7wkbl" Oct 31 00:09:12.772084 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:12.771936 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e9c18450-ac2a-4e6a-a712-00c2a755535d-home\") pod \"e2e-lifecycle-model-a251fdfb-kserve-5966998657-7wkbl\" (UID: \"e9c18450-ac2a-4e6a-a712-00c2a755535d\") " pod="ai-tenant-e2e-mt-a251fdfb/e2e-lifecycle-model-a251fdfb-kserve-5966998657-7wkbl" Oct 31 00:09:12.772160 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:12.772137 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9c18450-ac2a-4e6a-a712-00c2a755535d-model-cache\") pod \"e2e-lifecycle-model-a251fdfb-kserve-5966998657-7wkbl\" (UID: \"e9c18450-ac2a-4e6a-a712-00c2a755535d\") " pod="ai-tenant-e2e-mt-a251fdfb/e2e-lifecycle-model-a251fdfb-kserve-5966998657-7wkbl" Oct 31 00:09:12.774855 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:12.774796 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9c18450-ac2a-4e6a-a712-00c2a755535d-dshm\") pod \"e2e-lifecycle-model-a251fdfb-kserve-5966998657-7wkbl\" (UID: \"e9c18450-ac2a-4e6a-a712-00c2a755535d\") " pod="ai-tenant-e2e-mt-a251fdfb/e2e-lifecycle-model-a251fdfb-kserve-5966998657-7wkbl" Oct 31 00:09:12.776458 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:12.776435 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9c18450-ac2a-4e6a-a712-00c2a755535d-tls-certs\") pod \"e2e-lifecycle-model-a251fdfb-kserve-5966998657-7wkbl\" (UID: \"e9c18450-ac2a-4e6a-a712-00c2a755535d\") " pod="ai-tenant-e2e-mt-a251fdfb/e2e-lifecycle-model-a251fdfb-kserve-5966998657-7wkbl" Oct 31 00:09:12.781748 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:12.781675 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:09:12.847720 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:12.847685 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-7c8802f8-openshift-default-74499c95cc-v5qrl" Oct 31 00:09:12.853065 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:12.852900 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2p984\" (UniqueName: \"kubernetes.io/projected/e9c18450-ac2a-4e6a-a712-00c2a755535d-kube-api-access-2p984\") pod \"e2e-lifecycle-model-a251fdfb-kserve-5966998657-7wkbl\" (UID: \"e9c18450-ac2a-4e6a-a712-00c2a755535d\") " pod="ai-tenant-e2e-mt-a251fdfb/e2e-lifecycle-model-a251fdfb-kserve-5966998657-7wkbl" Oct 31 00:09:12.946241 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:12.946208 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:09:12.977420 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:12.977386 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85533e83-d620-4bd3-8290-408d0c41164c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85533e83-d620-4bd3-8290-408d0c41164c-workload-certs\") pod \"85533e83-d620-4bd3-8290-408d0c41164c\" (UID: \"85533e83-d620-4bd3-8290-408d0c41164c\") " Oct 31 00:09:12.977552 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:12.977490 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/85533e83-d620-4bd3-8290-408d0c41164c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85533e83-d620-4bd3-8290-408d0c41164c-istiod-ca-cert\") pod \"85533e83-d620-4bd3-8290-408d0c41164c\" (UID: \"85533e83-d620-4bd3-8290-408d0c41164c\") " Oct 31 00:09:12.977552 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:12.977519 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85533e83-d620-4bd3-8290-408d0c41164c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85533e83-d620-4bd3-8290-408d0c41164c-workload-socket\") pod \"85533e83-d620-4bd3-8290-408d0c41164c\" (UID: \"85533e83-d620-4bd3-8290-408d0c41164c\") " Oct 31 00:09:12.977654 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:12.977609 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85533e83-d620-4bd3-8290-408d0c41164c-istio-token\" (UniqueName: \"kubernetes.io/projected/85533e83-d620-4bd3-8290-408d0c41164c-istio-token\") pod \"85533e83-d620-4bd3-8290-408d0c41164c\" (UID: \"85533e83-d620-4bd3-8290-408d0c41164c\") " Oct 31 00:09:12.977654 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:12.977635 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85533e83-d620-4bd3-8290-408d0c41164c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85533e83-d620-4bd3-8290-408d0c41164c-istio-envoy\") pod \"85533e83-d620-4bd3-8290-408d0c41164c\" (UID: \"85533e83-d620-4bd3-8290-408d0c41164c\") " Oct 31 00:09:12.977785 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:12.977723 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85533e83-d620-4bd3-8290-408d0c41164c-workload-certs" pod "85533e83-d620-4bd3-8290-408d0c41164c" (UID: "85533e83-d620-4bd3-8290-408d0c41164c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:09:12.977785 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:12.977733 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85533e83-d620-4bd3-8290-408d0c41164c-kube-api-access-t78vf\" (UniqueName: \"kubernetes.io/projected/85533e83-d620-4bd3-8290-408d0c41164c-kube-api-access-t78vf\") pod \"85533e83-d620-4bd3-8290-408d0c41164c\" (UID: \"85533e83-d620-4bd3-8290-408d0c41164c\") " Oct 31 00:09:12.978006 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:12.977950 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85533e83-d620-4bd3-8290-408d0c41164c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85533e83-d620-4bd3-8290-408d0c41164c-credential-socket\") pod \"85533e83-d620-4bd3-8290-408d0c41164c\" (UID: \"85533e83-d620-4bd3-8290-408d0c41164c\") " Oct 31 00:09:12.978187 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:12.978118 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/85533e83-d620-4bd3-8290-408d0c41164c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85533e83-d620-4bd3-8290-408d0c41164c-istio-podinfo\") pod \"85533e83-d620-4bd3-8290-408d0c41164c\" (UID: \"85533e83-d620-4bd3-8290-408d0c41164c\") " Oct 31 00:09:12.978187 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:12.978163 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85533e83-d620-4bd3-8290-408d0c41164c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85533e83-d620-4bd3-8290-408d0c41164c-istio-data\") pod \"85533e83-d620-4bd3-8290-408d0c41164c\" (UID: \"85533e83-d620-4bd3-8290-408d0c41164c\") " Oct 31 00:09:12.979257 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:12.979067 4927 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85533e83-d620-4bd3-8290-408d0c41164c-workload-certs\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:09:12.979454 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:12.979416 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85533e83-d620-4bd3-8290-408d0c41164c-istio-data" pod "85533e83-d620-4bd3-8290-408d0c41164c" (UID: "85533e83-d620-4bd3-8290-408d0c41164c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:09:12.979809 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:12.979776 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85533e83-d620-4bd3-8290-408d0c41164c-workload-socket" pod "85533e83-d620-4bd3-8290-408d0c41164c" (UID: "85533e83-d620-4bd3-8290-408d0c41164c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:09:12.979947 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:12.979918 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85533e83-d620-4bd3-8290-408d0c41164c-istiod-ca-cert" pod "85533e83-d620-4bd3-8290-408d0c41164c" (UID: "85533e83-d620-4bd3-8290-408d0c41164c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:09:12.981315 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:12.979186 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85533e83-d620-4bd3-8290-408d0c41164c-credential-socket" pod "85533e83-d620-4bd3-8290-408d0c41164c" (UID: "85533e83-d620-4bd3-8290-408d0c41164c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:09:12.982444 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:12.982409 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/85533e83-d620-4bd3-8290-408d0c41164c-istio-podinfo" pod "85533e83-d620-4bd3-8290-408d0c41164c" (UID: "85533e83-d620-4bd3-8290-408d0c41164c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:09:12.982696 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:12.982657 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85533e83-d620-4bd3-8290-408d0c41164c-kube-api-access-t78vf" pod "85533e83-d620-4bd3-8290-408d0c41164c" (UID: "85533e83-d620-4bd3-8290-408d0c41164c"). InnerVolumeSpecName "kube-api-access-t78vf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:09:12.982817 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:12.982721 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85533e83-d620-4bd3-8290-408d0c41164c-istio-envoy" pod "85533e83-d620-4bd3-8290-408d0c41164c" (UID: "85533e83-d620-4bd3-8290-408d0c41164c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:09:12.983532 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:12.983506 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85533e83-d620-4bd3-8290-408d0c41164c-istio-token" pod "85533e83-d620-4bd3-8290-408d0c41164c" (UID: "85533e83-d620-4bd3-8290-408d0c41164c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:09:13.023131 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:13.023047 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-a251fdfb/e2e-lifecycle-model-a251fdfb-kserve-5966998657-7wkbl" Oct 31 00:09:13.080173 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:13.080142 4927 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85533e83-d620-4bd3-8290-408d0c41164c-credential-socket\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:09:13.080173 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:13.080169 4927 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85533e83-d620-4bd3-8290-408d0c41164c-istio-podinfo\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:09:13.080173 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:13.080180 4927 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85533e83-d620-4bd3-8290-408d0c41164c-istio-data\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:09:13.080446 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:13.080189 4927 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85533e83-d620-4bd3-8290-408d0c41164c-istiod-ca-cert\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:09:13.080446 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:13.080201 4927 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85533e83-d620-4bd3-8290-408d0c41164c-workload-socket\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:09:13.080446 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:13.080213 4927 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85533e83-d620-4bd3-8290-408d0c41164c-istio-token\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:09:13.080446 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:13.080250 4927 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85533e83-d620-4bd3-8290-408d0c41164c-istio-envoy\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:09:13.080446 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:13.080273 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t78vf\" (UniqueName: \"kubernetes.io/projected/85533e83-d620-4bd3-8290-408d0c41164c-kube-api-access-t78vf\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:09:13.146620 ip-10-0-141-180 kubenswrapper[4927]: W1031 00:09:13.146584 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode27553d8_0637_46a1_bd84_186e69ba3f15.slice/crio-ab8aeb82b05e46a9685b9c9b057826962a1e280dee9e6334bf4c97fe3e13fd9a WatchSource:0}: Error finding container ab8aeb82b05e46a9685b9c9b057826962a1e280dee9e6334bf4c97fe3e13fd9a: Status 404 returned error can't find the container with id ab8aeb82b05e46a9685b9c9b057826962a1e280dee9e6334bf4c97fe3e13fd9a Oct 31 00:09:13.148182 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:13.148153 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-d5360e-854fc9bc5d-mc76v"] Oct 31 00:09:13.184478 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:13.184451 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:09:13.185845 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:13.185823 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-d5360e-b6fb568dc-htsps"] Oct 31 00:09:13.195767 ip-10-0-141-180 kubenswrapper[4927]: W1031 00:09:13.195737 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64551325_0270_49ec_bbc7_748cd5c3611e.slice/crio-1afd116633824e8e387efbb61d4466aff7d4aa353069c8c3f36e62e4df3e7775 WatchSource:0}: Error finding container 1afd116633824e8e387efbb61d4466aff7d4aa353069c8c3f36e62e4df3e7775: Status 404 returned error can't find the container with id 1afd116633824e8e387efbb61d4466aff7d4aa353069c8c3f36e62e4df3e7775 Oct 31 00:09:13.420731 ip-10-0-141-180 kubenswrapper[4927]: W1031 00:09:13.420696 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode9c18450_ac2a_4e6a_a712_00c2a755535d.slice/crio-a20c82547afb2207318e618562bdf8715907b9104bf786d494d5f1e7dc78a8b7 WatchSource:0}: Error finding container a20c82547afb2207318e618562bdf8715907b9104bf786d494d5f1e7dc78a8b7: Status 404 returned error can't find the container with id a20c82547afb2207318e618562bdf8715907b9104bf786d494d5f1e7dc78a8b7 Oct 31 00:09:13.421145 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:13.421113 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-a251fdfb/e2e-lifecycle-model-a251fdfb-kserve-5966998657-7wkbl"] Oct 31 00:09:13.769504 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:13.769458 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-d5360e-854fc9bc5d-mc76v" event={"ID":"e27553d8-0637-46a1-bd84-186e69ba3f15","Type":"ContainerStarted","Data":"91f4ab37f571dda42eb43047b701d99c5460a0a6931d51411f117c8fd6314e5b"} Oct 31 00:09:13.769660 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:13.769549 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-d5360e-854fc9bc5d-mc76v" event={"ID":"e27553d8-0637-46a1-bd84-186e69ba3f15","Type":"ContainerStarted","Data":"ab8aeb82b05e46a9685b9c9b057826962a1e280dee9e6334bf4c97fe3e13fd9a"} Oct 31 00:09:13.769660 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:13.769578 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-d5360e-854fc9bc5d-mc76v" Oct 31 00:09:13.771432 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:13.771390 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-d5360e-b6fb568dc-htsps" event={"ID":"64551325-0270-49ec-bbc7-748cd5c3611e","Type":"ContainerStarted","Data":"ba6a08f7b995d7314ecd91ce1bd1015e3420e269e81e81380e4573b45f77b353"} Oct 31 00:09:13.771957 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:13.771470 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-d5360e-b6fb568dc-htsps" Oct 31 00:09:13.771957 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:13.771495 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-d5360e-b6fb568dc-htsps" event={"ID":"64551325-0270-49ec-bbc7-748cd5c3611e","Type":"ContainerStarted","Data":"1afd116633824e8e387efbb61d4466aff7d4aa353069c8c3f36e62e4df3e7775"} Oct 31 00:09:13.772887 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:13.772864 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-7c8802f8-openshift-default-74499c95cc-v5qrl" Oct 31 00:09:13.773015 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:13.772873 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-7c8802f8-openshift-default-74499c95cc-v5qrl" event={"ID":"85533e83-d620-4bd3-8290-408d0c41164c","Type":"ContainerDied","Data":"d9c09cea150c30ffc7688aa7f177d0f9cfa9936a4f8af8604d7e360c6d087dfe"} Oct 31 00:09:13.773015 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:13.772989 4927 scope.go:122] "RemoveContainer" containerID="dbe5de481eae525d6ef94eb186fd51421928cd6be4e48593e55c0da8a239f7f1" Oct 31 00:09:13.775795 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:13.775771 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-a251fdfb/e2e-lifecycle-model-a251fdfb-kserve-5966998657-7wkbl" event={"ID":"e9c18450-ac2a-4e6a-a712-00c2a755535d","Type":"ContainerStarted","Data":"e0443ccad024469a8e53ffc04186b0f362c0dab29e418fb70032ba59372e2435"} Oct 31 00:09:13.775882 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:13.775802 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-a251fdfb/e2e-lifecycle-model-a251fdfb-kserve-5966998657-7wkbl" event={"ID":"e9c18450-ac2a-4e6a-a712-00c2a755535d","Type":"ContainerStarted","Data":"a20c82547afb2207318e618562bdf8715907b9104bf786d494d5f1e7dc78a8b7"} Oct 31 00:09:13.891040 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:13.890986 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-d5360e-854fc9bc5d-mc76v" podStartSLOduration=2.89097378 podStartE2EDuration="2.89097378s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:09:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:09:13.887635477 +0000 UTC m=+1666.923966346" watchObservedRunningTime="2026-10-31 00:09:13.89097378 +0000 UTC m=+1666.927304602" Oct 31 00:09:14.044409 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:14.038450 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-a251fdfb/e2e-lifecycle-model-a251fdfb-kserve-5966998657-7wkbl" Oct 31 00:09:14.044409 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:14.043693 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-mt-a251fdfb/e2e-lifecycle-model-a251fdfb-kserve-5966998657-7wkbl" podStartSLOduration=3.043672366 podStartE2EDuration="3.043672366s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:09:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:09:14.017799089 +0000 UTC m=+1667.054129936" watchObservedRunningTime="2026-10-31 00:09:14.043672366 +0000 UTC m=+1667.080003190" Oct 31 00:09:14.141532 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:14.141472 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-7c8802f8-openshift-default-74499c95cc-v5qrl"] Oct 31 00:09:14.149785 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:14.148253 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-7c8802f8-openshift-default-74499c95cc-v5qrl"] Oct 31 00:09:14.281794 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:14.279401 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-d5360e-b6fb568dc-htsps" podStartSLOduration=3.279383344 podStartE2EDuration="3.279383344s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:09:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:09:14.260901765 +0000 UTC m=+1667.297232588" watchObservedRunningTime="2026-10-31 00:09:14.279383344 +0000 UTC m=+1667.315714168" Oct 31 00:09:15.088036 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:15.088001 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="85533e83-d620-4bd3-8290-408d0c41164c" path="/var/lib/kubelet/pods/85533e83-d620-4bd3-8290-408d0c41164c/volumes" Oct 31 00:09:15.958924 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:15.958842 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:09:17.145590 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:17.145560 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-0d75be-75f9855f5-kftq4" Oct 31 00:09:17.553388 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:17.553322 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-mlkdz"] Oct 31 00:09:17.553679 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:17.553620 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-mlkdz" podUID="d9bfa829-54c3-4c65-8596-dca8560ec433" containerName="istio-proxy" containerID="cri-o://480c4ab2112a37c5eed8ef1926b62400aa5aa06dee35334733aa6ff8b8c23f55" gracePeriod=30 Oct 31 00:09:17.603064 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:17.603031 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-q7dtb"] Oct 31 00:09:17.603266 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:17.603240 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-q7dtb" podUID="1f3dbfce-11a6-4176-9a52-c9772a349dbe" containerName="istio-proxy" containerID="cri-o://1ddf6df601c51af42c951b2599fe6369fb3899aa2935bec193cde30e0f90c4e9" gracePeriod=30 Oct 31 00:09:18.543851 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:18.543823 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-0d75be-7b66b84856-j28w4" Oct 31 00:09:19.684886 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:09:19.684848 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sqv69" containerName="extractor" Oct 31 00:09:20.363808 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:20.363771 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:09:20.550919 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:20.550881 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:09:20.772799 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:20.772759 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-54ad8c-7b7f75c84c-nntg5"] Oct 31 00:09:20.773574 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:20.773555 4927 cpu_manager.go:414] "containerMap: removing container" podUID="85533e83-d620-4bd3-8290-408d0c41164c" containerName="istio-proxy" Oct 31 00:09:20.773574 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:20.773575 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85533e83-d620-4bd3-8290-408d0c41164c" containerName="istio-proxy" Oct 31 00:09:20.773696 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:20.773606 4927 cpu_manager.go:414] "containerMap: removing container" podUID="85533e83-d620-4bd3-8290-408d0c41164c" containerName="istio-proxy" Oct 31 00:09:20.773696 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:20.773612 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85533e83-d620-4bd3-8290-408d0c41164c" containerName="istio-proxy" Oct 31 00:09:20.773761 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:20.773724 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="85533e83-d620-4bd3-8290-408d0c41164c" containerName="istio-proxy" Oct 31 00:09:20.777724 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:20.777703 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-54ad8c-7b7f75c84c-nntg5" Oct 31 00:09:20.955361 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:20.950945 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w4-54ad8c\"" Oct 31 00:09:20.955361 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:20.951085 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w4-54ad8c-dockercfg-898ts\"" Oct 31 00:09:20.971553 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:20.971513 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-54ad8c-7b7f75c84c-nntg5"] Oct 31 00:09:20.971757 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:20.971607 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-54ad8c-7c679ffcbb-kmx4b"] Oct 31 00:09:20.976148 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:20.975870 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="85533e83-d620-4bd3-8290-408d0c41164c" containerName="istio-proxy" Oct 31 00:09:21.019354 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:21.005257 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-54ad8c-7c679ffcbb-kmx4b" Oct 31 00:09:21.019354 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:21.017931 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/777b44d3-70ae-4988-b2bc-63c7a7830461-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-54ad8c-7b7f75c84c-nntg5\" (UID: \"777b44d3-70ae-4988-b2bc-63c7a7830461\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-54ad8c-7b7f75c84c-nntg5" Oct 31 00:09:21.019354 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:21.018238 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-txfjg\" (UniqueName: \"kubernetes.io/projected/777b44d3-70ae-4988-b2bc-63c7a7830461-kube-api-access-txfjg\") pod \"payload-pre-processing-e2e-shared-a-w4-54ad8c-7b7f75c84c-nntg5\" (UID: \"777b44d3-70ae-4988-b2bc-63c7a7830461\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-54ad8c-7b7f75c84c-nntg5" Oct 31 00:09:21.030119 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:21.029909 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:09:21.066206 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:21.066158 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-54ad8c-7c679ffcbb-kmx4b"] Oct 31 00:09:21.145083 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:21.145042 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txfjg\" (UniqueName: \"kubernetes.io/projected/777b44d3-70ae-4988-b2bc-63c7a7830461-kube-api-access-txfjg\") pod \"payload-pre-processing-e2e-shared-a-w4-54ad8c-7b7f75c84c-nntg5\" (UID: \"777b44d3-70ae-4988-b2bc-63c7a7830461\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-54ad8c-7b7f75c84c-nntg5" Oct 31 00:09:21.145533 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:21.145490 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhxnx\" (UniqueName: \"kubernetes.io/projected/9dccc739-8fcc-4f45-b236-58338e7fcd05-kube-api-access-zhxnx\") pod \"payload-processing-e2e-shared-a-w4-54ad8c-7c679ffcbb-kmx4b\" (UID: \"9dccc739-8fcc-4f45-b236-58338e7fcd05\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-54ad8c-7c679ffcbb-kmx4b" Oct 31 00:09:21.145876 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:21.145845 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/777b44d3-70ae-4988-b2bc-63c7a7830461-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-54ad8c-7b7f75c84c-nntg5\" (UID: \"777b44d3-70ae-4988-b2bc-63c7a7830461\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-54ad8c-7b7f75c84c-nntg5" Oct 31 00:09:21.188859 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:21.188743 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9dccc739-8fcc-4f45-b236-58338e7fcd05-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-54ad8c-7c679ffcbb-kmx4b\" (UID: \"9dccc739-8fcc-4f45-b236-58338e7fcd05\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-54ad8c-7c679ffcbb-kmx4b" Oct 31 00:09:21.356035 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:21.355948 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-txfjg\" (UniqueName: \"kubernetes.io/projected/777b44d3-70ae-4988-b2bc-63c7a7830461-kube-api-access-txfjg\") pod \"payload-pre-processing-e2e-shared-a-w4-54ad8c-7b7f75c84c-nntg5\" (UID: \"777b44d3-70ae-4988-b2bc-63c7a7830461\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-54ad8c-7b7f75c84c-nntg5" Oct 31 00:09:21.376383 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:21.372556 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/777b44d3-70ae-4988-b2bc-63c7a7830461-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-54ad8c-7b7f75c84c-nntg5\" (UID: \"777b44d3-70ae-4988-b2bc-63c7a7830461\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-54ad8c-7b7f75c84c-nntg5" Oct 31 00:09:21.394586 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:21.392573 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-54ad8c-7b7f75c84c-nntg5" Oct 31 00:09:21.474388 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:21.473660 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhxnx\" (UniqueName: \"kubernetes.io/projected/9dccc739-8fcc-4f45-b236-58338e7fcd05-kube-api-access-zhxnx\") pod \"payload-processing-e2e-shared-a-w4-54ad8c-7c679ffcbb-kmx4b\" (UID: \"9dccc739-8fcc-4f45-b236-58338e7fcd05\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-54ad8c-7c679ffcbb-kmx4b" Oct 31 00:09:21.474388 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:21.473747 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9dccc739-8fcc-4f45-b236-58338e7fcd05-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-54ad8c-7c679ffcbb-kmx4b\" (UID: \"9dccc739-8fcc-4f45-b236-58338e7fcd05\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-54ad8c-7c679ffcbb-kmx4b" Oct 31 00:09:21.474595 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:21.474477 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9dccc739-8fcc-4f45-b236-58338e7fcd05-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-54ad8c-7c679ffcbb-kmx4b\" (UID: \"9dccc739-8fcc-4f45-b236-58338e7fcd05\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-54ad8c-7c679ffcbb-kmx4b" Oct 31 00:09:21.537197 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:21.537159 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhxnx\" (UniqueName: \"kubernetes.io/projected/9dccc739-8fcc-4f45-b236-58338e7fcd05-kube-api-access-zhxnx\") pod \"payload-processing-e2e-shared-a-w4-54ad8c-7c679ffcbb-kmx4b\" (UID: \"9dccc739-8fcc-4f45-b236-58338e7fcd05\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-54ad8c-7c679ffcbb-kmx4b" Oct 31 00:09:21.653352 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:21.653258 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-54ad8c-7c679ffcbb-kmx4b" Oct 31 00:09:21.778831 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:21.778798 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-54ad8c-7b7f75c84c-nntg5"] Oct 31 00:09:21.787914 ip-10-0-141-180 kubenswrapper[4927]: W1031 00:09:21.787886 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod777b44d3_70ae_4988_b2bc_63c7a7830461.slice/crio-f7ad08296104b1bc872732f88788a04b729eb4507f835fa67d561d838234ba0e WatchSource:0}: Error finding container f7ad08296104b1bc872732f88788a04b729eb4507f835fa67d561d838234ba0e: Status 404 returned error can't find the container with id f7ad08296104b1bc872732f88788a04b729eb4507f835fa67d561d838234ba0e Oct 31 00:09:22.027741 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:22.027699 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-54ad8c-7b7f75c84c-nntg5" event={"ID":"777b44d3-70ae-4988-b2bc-63c7a7830461","Type":"ContainerStarted","Data":"584615f03324988b3dc4b6b05211cca4612b3463d8c06fcd3d3318e41f6ae4c5"} Oct 31 00:09:22.027922 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:22.027767 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-54ad8c-7b7f75c84c-nntg5" event={"ID":"777b44d3-70ae-4988-b2bc-63c7a7830461","Type":"ContainerStarted","Data":"f7ad08296104b1bc872732f88788a04b729eb4507f835fa67d561d838234ba0e"} Oct 31 00:09:22.027922 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:22.027794 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-54ad8c-7b7f75c84c-nntg5" Oct 31 00:09:22.039824 ip-10-0-141-180 kubenswrapper[4927]: W1031 00:09:22.039798 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9dccc739_8fcc_4f45_b236_58338e7fcd05.slice/crio-f8f41f88675a55cec5360f521f57b72aae11f6f3969f7e63fb388b36097a9454 WatchSource:0}: Error finding container f8f41f88675a55cec5360f521f57b72aae11f6f3969f7e63fb388b36097a9454: Status 404 returned error can't find the container with id f8f41f88675a55cec5360f521f57b72aae11f6f3969f7e63fb388b36097a9454 Oct 31 00:09:22.042514 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:22.042484 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-54ad8c-7c679ffcbb-kmx4b"] Oct 31 00:09:22.150414 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:22.150365 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-54ad8c-7b7f75c84c-nntg5" podStartSLOduration=2.150350183 podStartE2EDuration="2.150350183s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:09:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:09:22.145483165 +0000 UTC m=+1675.181813987" watchObservedRunningTime="2026-10-31 00:09:22.150350183 +0000 UTC m=+1675.186681004" Oct 31 00:09:22.553654 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:22.553612 4927 patch_prober.go:29] interesting pod/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-q7dtb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 00:09:22.553810 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:22.553673 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-q7dtb" podUID="1f3dbfce-11a6-4176-9a52-c9772a349dbe" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:09:22.901593 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:22.901563 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-mlkdz" Oct 31 00:09:22.912630 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:22.912605 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-q7dtb" Oct 31 00:09:23.002398 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:23.002363 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f3dbfce-11a6-4176-9a52-c9772a349dbe-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f3dbfce-11a6-4176-9a52-c9772a349dbe-istiod-ca-cert\") pod \"1f3dbfce-11a6-4176-9a52-c9772a349dbe\" (UID: \"1f3dbfce-11a6-4176-9a52-c9772a349dbe\") " Oct 31 00:09:23.002587 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:23.002459 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d9bfa829-54c3-4c65-8596-dca8560ec433-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9bfa829-54c3-4c65-8596-dca8560ec433-istiod-ca-cert\") pod \"d9bfa829-54c3-4c65-8596-dca8560ec433\" (UID: \"d9bfa829-54c3-4c65-8596-dca8560ec433\") " Oct 31 00:09:23.002587 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:23.002516 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f3dbfce-11a6-4176-9a52-c9772a349dbe-istio-token\" (UniqueName: \"kubernetes.io/projected/1f3dbfce-11a6-4176-9a52-c9772a349dbe-istio-token\") pod \"1f3dbfce-11a6-4176-9a52-c9772a349dbe\" (UID: \"1f3dbfce-11a6-4176-9a52-c9772a349dbe\") " Oct 31 00:09:23.002587 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:23.002560 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9bfa829-54c3-4c65-8596-dca8560ec433-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9bfa829-54c3-4c65-8596-dca8560ec433-credential-socket\") pod \"d9bfa829-54c3-4c65-8596-dca8560ec433\" (UID: \"d9bfa829-54c3-4c65-8596-dca8560ec433\") " Oct 31 00:09:23.002741 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:23.002595 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9bfa829-54c3-4c65-8596-dca8560ec433-istio-token\" (UniqueName: \"kubernetes.io/projected/d9bfa829-54c3-4c65-8596-dca8560ec433-istio-token\") pod \"d9bfa829-54c3-4c65-8596-dca8560ec433\" (UID: \"d9bfa829-54c3-4c65-8596-dca8560ec433\") " Oct 31 00:09:23.002741 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:23.002638 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f3dbfce-11a6-4176-9a52-c9772a349dbe-kube-api-access-96n2w\" (UniqueName: \"kubernetes.io/projected/1f3dbfce-11a6-4176-9a52-c9772a349dbe-kube-api-access-96n2w\") pod \"1f3dbfce-11a6-4176-9a52-c9772a349dbe\" (UID: \"1f3dbfce-11a6-4176-9a52-c9772a349dbe\") " Oct 31 00:09:23.002741 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:23.002679 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9bfa829-54c3-4c65-8596-dca8560ec433-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9bfa829-54c3-4c65-8596-dca8560ec433-workload-socket\") pod \"d9bfa829-54c3-4c65-8596-dca8560ec433\" (UID: \"d9bfa829-54c3-4c65-8596-dca8560ec433\") " Oct 31 00:09:23.002954 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:23.002756 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d9bfa829-54c3-4c65-8596-dca8560ec433-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9bfa829-54c3-4c65-8596-dca8560ec433-istio-podinfo\") pod \"d9bfa829-54c3-4c65-8596-dca8560ec433\" (UID: \"d9bfa829-54c3-4c65-8596-dca8560ec433\") " Oct 31 00:09:23.002954 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:23.002787 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f3dbfce-11a6-4176-9a52-c9772a349dbe-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f3dbfce-11a6-4176-9a52-c9772a349dbe-workload-certs\") pod \"1f3dbfce-11a6-4176-9a52-c9772a349dbe\" (UID: \"1f3dbfce-11a6-4176-9a52-c9772a349dbe\") " Oct 31 00:09:23.002954 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:23.002926 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f3dbfce-11a6-4176-9a52-c9772a349dbe-istiod-ca-cert" pod "1f3dbfce-11a6-4176-9a52-c9772a349dbe" (UID: "1f3dbfce-11a6-4176-9a52-c9772a349dbe"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:09:23.003188 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:23.002977 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9bfa829-54c3-4c65-8596-dca8560ec433-kube-api-access-v9zzv\" (UniqueName: \"kubernetes.io/projected/d9bfa829-54c3-4c65-8596-dca8560ec433-kube-api-access-v9zzv\") pod \"d9bfa829-54c3-4c65-8596-dca8560ec433\" (UID: \"d9bfa829-54c3-4c65-8596-dca8560ec433\") " Oct 31 00:09:23.003188 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:23.003041 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1f3dbfce-11a6-4176-9a52-c9772a349dbe-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f3dbfce-11a6-4176-9a52-c9772a349dbe-istio-podinfo\") pod \"1f3dbfce-11a6-4176-9a52-c9772a349dbe\" (UID: \"1f3dbfce-11a6-4176-9a52-c9772a349dbe\") " Oct 31 00:09:23.003188 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:23.003091 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f3dbfce-11a6-4176-9a52-c9772a349dbe-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f3dbfce-11a6-4176-9a52-c9772a349dbe-workload-socket\") pod \"1f3dbfce-11a6-4176-9a52-c9772a349dbe\" (UID: \"1f3dbfce-11a6-4176-9a52-c9772a349dbe\") " Oct 31 00:09:23.003188 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:23.003130 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9bfa829-54c3-4c65-8596-dca8560ec433-credential-socket" pod "d9bfa829-54c3-4c65-8596-dca8560ec433" (UID: "d9bfa829-54c3-4c65-8596-dca8560ec433"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:09:23.003188 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:23.003142 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9bfa829-54c3-4c65-8596-dca8560ec433-workload-socket" pod "d9bfa829-54c3-4c65-8596-dca8560ec433" (UID: "d9bfa829-54c3-4c65-8596-dca8560ec433"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:09:23.003188 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:23.003147 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f3dbfce-11a6-4176-9a52-c9772a349dbe-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f3dbfce-11a6-4176-9a52-c9772a349dbe-credential-socket\") pod \"1f3dbfce-11a6-4176-9a52-c9772a349dbe\" (UID: \"1f3dbfce-11a6-4176-9a52-c9772a349dbe\") " Oct 31 00:09:23.003188 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:23.003178 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9bfa829-54c3-4c65-8596-dca8560ec433-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9bfa829-54c3-4c65-8596-dca8560ec433-istio-data\") pod \"d9bfa829-54c3-4c65-8596-dca8560ec433\" (UID: \"d9bfa829-54c3-4c65-8596-dca8560ec433\") " Oct 31 00:09:23.004080 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:23.003216 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f3dbfce-11a6-4176-9a52-c9772a349dbe-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f3dbfce-11a6-4176-9a52-c9772a349dbe-istio-envoy\") pod \"1f3dbfce-11a6-4176-9a52-c9772a349dbe\" (UID: \"1f3dbfce-11a6-4176-9a52-c9772a349dbe\") " Oct 31 00:09:23.004080 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:23.003279 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9bfa829-54c3-4c65-8596-dca8560ec433-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9bfa829-54c3-4c65-8596-dca8560ec433-istio-envoy\") pod \"d9bfa829-54c3-4c65-8596-dca8560ec433\" (UID: \"d9bfa829-54c3-4c65-8596-dca8560ec433\") " Oct 31 00:09:23.004080 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:23.003313 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f3dbfce-11a6-4176-9a52-c9772a349dbe-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f3dbfce-11a6-4176-9a52-c9772a349dbe-istio-data\") pod \"1f3dbfce-11a6-4176-9a52-c9772a349dbe\" (UID: \"1f3dbfce-11a6-4176-9a52-c9772a349dbe\") " Oct 31 00:09:23.004080 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:23.003321 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f3dbfce-11a6-4176-9a52-c9772a349dbe-workload-certs" pod "1f3dbfce-11a6-4176-9a52-c9772a349dbe" (UID: "1f3dbfce-11a6-4176-9a52-c9772a349dbe"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:09:23.004080 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:23.003376 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f3dbfce-11a6-4176-9a52-c9772a349dbe-workload-socket" pod "1f3dbfce-11a6-4176-9a52-c9772a349dbe" (UID: "1f3dbfce-11a6-4176-9a52-c9772a349dbe"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:09:23.004080 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:23.003385 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9bfa829-54c3-4c65-8596-dca8560ec433-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9bfa829-54c3-4c65-8596-dca8560ec433-workload-certs\") pod \"d9bfa829-54c3-4c65-8596-dca8560ec433\" (UID: \"d9bfa829-54c3-4c65-8596-dca8560ec433\") " Oct 31 00:09:23.004080 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:23.003532 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9bfa829-54c3-4c65-8596-dca8560ec433-istio-data" pod "d9bfa829-54c3-4c65-8596-dca8560ec433" (UID: "d9bfa829-54c3-4c65-8596-dca8560ec433"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:09:23.004080 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:23.003618 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9bfa829-54c3-4c65-8596-dca8560ec433-workload-certs" pod "d9bfa829-54c3-4c65-8596-dca8560ec433" (UID: "d9bfa829-54c3-4c65-8596-dca8560ec433"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:09:23.004080 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:23.003737 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f3dbfce-11a6-4176-9a52-c9772a349dbe-credential-socket" pod "1f3dbfce-11a6-4176-9a52-c9772a349dbe" (UID: "1f3dbfce-11a6-4176-9a52-c9772a349dbe"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:09:23.004080 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:23.003943 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f3dbfce-11a6-4176-9a52-c9772a349dbe-istio-data" pod "1f3dbfce-11a6-4176-9a52-c9772a349dbe" (UID: "1f3dbfce-11a6-4176-9a52-c9772a349dbe"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:09:23.004774 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:23.004131 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9bfa829-54c3-4c65-8596-dca8560ec433-istiod-ca-cert" pod "d9bfa829-54c3-4c65-8596-dca8560ec433" (UID: "d9bfa829-54c3-4c65-8596-dca8560ec433"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:09:23.004774 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:23.004216 4927 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f3dbfce-11a6-4176-9a52-c9772a349dbe-workload-socket\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:09:23.004774 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:23.004240 4927 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f3dbfce-11a6-4176-9a52-c9772a349dbe-credential-socket\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:09:23.004774 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:23.004256 4927 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9bfa829-54c3-4c65-8596-dca8560ec433-istio-data\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:09:23.004774 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:23.004269 4927 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f3dbfce-11a6-4176-9a52-c9772a349dbe-istio-data\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:09:23.004774 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:23.004282 4927 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9bfa829-54c3-4c65-8596-dca8560ec433-workload-certs\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:09:23.004774 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:23.004295 4927 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f3dbfce-11a6-4176-9a52-c9772a349dbe-istiod-ca-cert\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:09:23.004774 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:23.004308 4927 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9bfa829-54c3-4c65-8596-dca8560ec433-credential-socket\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:09:23.004774 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:23.004323 4927 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9bfa829-54c3-4c65-8596-dca8560ec433-workload-socket\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:09:23.004774 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:23.004373 4927 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f3dbfce-11a6-4176-9a52-c9772a349dbe-workload-certs\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:09:23.005847 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:23.005820 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f3dbfce-11a6-4176-9a52-c9772a349dbe-istio-envoy" pod "1f3dbfce-11a6-4176-9a52-c9772a349dbe" (UID: "1f3dbfce-11a6-4176-9a52-c9772a349dbe"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:09:23.006239 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:23.006212 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1f3dbfce-11a6-4176-9a52-c9772a349dbe-istio-podinfo" pod "1f3dbfce-11a6-4176-9a52-c9772a349dbe" (UID: "1f3dbfce-11a6-4176-9a52-c9772a349dbe"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:09:23.006391 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:23.006237 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f3dbfce-11a6-4176-9a52-c9772a349dbe-kube-api-access-96n2w" pod "1f3dbfce-11a6-4176-9a52-c9772a349dbe" (UID: "1f3dbfce-11a6-4176-9a52-c9772a349dbe"). InnerVolumeSpecName "kube-api-access-96n2w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:09:23.006391 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:23.006247 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d9bfa829-54c3-4c65-8596-dca8560ec433-istio-podinfo" pod "d9bfa829-54c3-4c65-8596-dca8560ec433" (UID: "d9bfa829-54c3-4c65-8596-dca8560ec433"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:09:23.006391 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:23.006287 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f3dbfce-11a6-4176-9a52-c9772a349dbe-istio-token" pod "1f3dbfce-11a6-4176-9a52-c9772a349dbe" (UID: "1f3dbfce-11a6-4176-9a52-c9772a349dbe"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:09:23.006391 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:23.006318 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9bfa829-54c3-4c65-8596-dca8560ec433-kube-api-access-v9zzv" pod "d9bfa829-54c3-4c65-8596-dca8560ec433" (UID: "d9bfa829-54c3-4c65-8596-dca8560ec433"). InnerVolumeSpecName "kube-api-access-v9zzv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:09:23.007000 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:23.006979 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9bfa829-54c3-4c65-8596-dca8560ec433-istio-token" pod "d9bfa829-54c3-4c65-8596-dca8560ec433" (UID: "d9bfa829-54c3-4c65-8596-dca8560ec433"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:09:23.007186 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:23.007167 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9bfa829-54c3-4c65-8596-dca8560ec433-istio-envoy" pod "d9bfa829-54c3-4c65-8596-dca8560ec433" (UID: "d9bfa829-54c3-4c65-8596-dca8560ec433"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:09:23.024245 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:23.024222 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-a251fdfb/e2e-lifecycle-model-a251fdfb-kserve-5966998657-7wkbl" Oct 31 00:09:23.042229 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:23.042195 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-a251fdfb/e2e-lifecycle-model-a251fdfb-kserve-5966998657-7wkbl" Oct 31 00:09:23.053229 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:23.053189 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-54ad8c-7c679ffcbb-kmx4b" event={"ID":"9dccc739-8fcc-4f45-b236-58338e7fcd05","Type":"ContainerStarted","Data":"fb433bc4bcfdb57d32597b60f9be58977ea893e817dae7964a189cf70f59d542"} Oct 31 00:09:23.053409 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:23.053253 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-54ad8c-7c679ffcbb-kmx4b" event={"ID":"9dccc739-8fcc-4f45-b236-58338e7fcd05","Type":"ContainerStarted","Data":"f8f41f88675a55cec5360f521f57b72aae11f6f3969f7e63fb388b36097a9454"} Oct 31 00:09:23.053409 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:23.053379 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-54ad8c-7c679ffcbb-kmx4b" Oct 31 00:09:23.054982 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:23.054958 4927 generic.go:356] "Generic (PLEG): container finished" podID="d9bfa829-54c3-4c65-8596-dca8560ec433" containerID="480c4ab2112a37c5eed8ef1926b62400aa5aa06dee35334733aa6ff8b8c23f55" exitCode=0 Oct 31 00:09:23.055096 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:23.055079 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-mlkdz" event={"ID":"d9bfa829-54c3-4c65-8596-dca8560ec433","Type":"ContainerDied","Data":"480c4ab2112a37c5eed8ef1926b62400aa5aa06dee35334733aa6ff8b8c23f55"} Oct 31 00:09:23.055142 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:23.055115 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-mlkdz" event={"ID":"d9bfa829-54c3-4c65-8596-dca8560ec433","Type":"ContainerDied","Data":"571f2163fa283dea21fb7f9d98fa137e1b13780cfcc4bc85e14867a100e047c4"} Oct 31 00:09:23.055142 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:23.055135 4927 scope.go:122] "RemoveContainer" containerID="480c4ab2112a37c5eed8ef1926b62400aa5aa06dee35334733aa6ff8b8c23f55" Oct 31 00:09:23.055280 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:23.055266 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-mlkdz" Oct 31 00:09:23.062757 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:23.062716 4927 generic.go:356] "Generic (PLEG): container finished" podID="1f3dbfce-11a6-4176-9a52-c9772a349dbe" containerID="1ddf6df601c51af42c951b2599fe6369fb3899aa2935bec193cde30e0f90c4e9" exitCode=0 Oct 31 00:09:23.063146 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:23.063114 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-q7dtb" Oct 31 00:09:23.063860 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:23.063661 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-q7dtb" event={"ID":"1f3dbfce-11a6-4176-9a52-c9772a349dbe","Type":"ContainerDied","Data":"1ddf6df601c51af42c951b2599fe6369fb3899aa2935bec193cde30e0f90c4e9"} Oct 31 00:09:23.063860 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:23.063734 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-q7dtb" event={"ID":"1f3dbfce-11a6-4176-9a52-c9772a349dbe","Type":"ContainerDied","Data":"8de2217435c525660b5dd89c6a5de7c2d313b103b06c062b85c53b66cb69035e"} Oct 31 00:09:23.094705 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:23.094602 4927 scope.go:122] "RemoveContainer" containerID="480c4ab2112a37c5eed8ef1926b62400aa5aa06dee35334733aa6ff8b8c23f55" Oct 31 00:09:23.095942 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:09:23.095914 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"480c4ab2112a37c5eed8ef1926b62400aa5aa06dee35334733aa6ff8b8c23f55\": container with ID starting with 480c4ab2112a37c5eed8ef1926b62400aa5aa06dee35334733aa6ff8b8c23f55 not found: ID does not exist" containerID="480c4ab2112a37c5eed8ef1926b62400aa5aa06dee35334733aa6ff8b8c23f55" Oct 31 00:09:23.096062 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:23.095947 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"480c4ab2112a37c5eed8ef1926b62400aa5aa06dee35334733aa6ff8b8c23f55"} err="failed to get container status \"480c4ab2112a37c5eed8ef1926b62400aa5aa06dee35334733aa6ff8b8c23f55\": rpc error: code = NotFound desc = could not find container \"480c4ab2112a37c5eed8ef1926b62400aa5aa06dee35334733aa6ff8b8c23f55\": container with ID starting with 480c4ab2112a37c5eed8ef1926b62400aa5aa06dee35334733aa6ff8b8c23f55 not found: ID does not exist" Oct 31 00:09:23.096062 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:23.095964 4927 scope.go:122] "RemoveContainer" containerID="1ddf6df601c51af42c951b2599fe6369fb3899aa2935bec193cde30e0f90c4e9" Oct 31 00:09:23.102027 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:23.102008 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-a251fdfb/e2e-lifecycle-model-a251fdfb-kserve-5966998657-7wkbl" Oct 31 00:09:23.106480 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:23.106450 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v9zzv\" (UniqueName: \"kubernetes.io/projected/d9bfa829-54c3-4c65-8596-dca8560ec433-kube-api-access-v9zzv\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:09:23.106604 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:23.106485 4927 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f3dbfce-11a6-4176-9a52-c9772a349dbe-istio-podinfo\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:09:23.106604 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:23.106504 4927 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f3dbfce-11a6-4176-9a52-c9772a349dbe-istio-envoy\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:09:23.106604 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:23.106522 4927 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9bfa829-54c3-4c65-8596-dca8560ec433-istio-envoy\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:09:23.106604 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:23.106539 4927 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9bfa829-54c3-4c65-8596-dca8560ec433-istiod-ca-cert\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:09:23.106604 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:23.106554 4927 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f3dbfce-11a6-4176-9a52-c9772a349dbe-istio-token\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:09:23.106604 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:23.106567 4927 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9bfa829-54c3-4c65-8596-dca8560ec433-istio-token\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:09:23.106604 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:23.106581 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-96n2w\" (UniqueName: \"kubernetes.io/projected/1f3dbfce-11a6-4176-9a52-c9772a349dbe-kube-api-access-96n2w\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:09:23.106604 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:23.106596 4927 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9bfa829-54c3-4c65-8596-dca8560ec433-istio-podinfo\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:09:23.114165 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:23.114143 4927 scope.go:122] "RemoveContainer" containerID="1ddf6df601c51af42c951b2599fe6369fb3899aa2935bec193cde30e0f90c4e9" Oct 31 00:09:23.114532 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:09:23.114510 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ddf6df601c51af42c951b2599fe6369fb3899aa2935bec193cde30e0f90c4e9\": container with ID starting with 1ddf6df601c51af42c951b2599fe6369fb3899aa2935bec193cde30e0f90c4e9 not found: ID does not exist" containerID="1ddf6df601c51af42c951b2599fe6369fb3899aa2935bec193cde30e0f90c4e9" Oct 31 00:09:23.114589 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:23.114542 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ddf6df601c51af42c951b2599fe6369fb3899aa2935bec193cde30e0f90c4e9"} err="failed to get container status \"1ddf6df601c51af42c951b2599fe6369fb3899aa2935bec193cde30e0f90c4e9\": rpc error: code = NotFound desc = could not find container \"1ddf6df601c51af42c951b2599fe6369fb3899aa2935bec193cde30e0f90c4e9\": container with ID starting with 1ddf6df601c51af42c951b2599fe6369fb3899aa2935bec193cde30e0f90c4e9 not found: ID does not exist" Oct 31 00:09:23.299309 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:23.299270 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-mlkdz"] Oct 31 00:09:23.303839 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:23.303807 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-mlkdz"] Oct 31 00:09:23.413361 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:23.413283 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-54ad8c-7c679ffcbb-kmx4b" podStartSLOduration=3.413263604 podStartE2EDuration="3.413263604s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:09:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:09:23.409232075 +0000 UTC m=+1676.445562898" watchObservedRunningTime="2026-10-31 00:09:23.413263604 +0000 UTC m=+1676.449594430" Oct 31 00:09:23.564428 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:23.564392 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-q7dtb"] Oct 31 00:09:23.593093 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:23.592988 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-q7dtb"] Oct 31 00:09:24.788712 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:24.788677 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-d5360e-854fc9bc5d-mc76v" Oct 31 00:09:24.789180 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:24.789118 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-d5360e-b6fb568dc-htsps" Oct 31 00:09:25.090284 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:25.090208 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f3dbfce-11a6-4176-9a52-c9772a349dbe" path="/var/lib/kubelet/pods/1f3dbfce-11a6-4176-9a52-c9772a349dbe/volumes" Oct 31 00:09:25.090841 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:25.090817 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d9bfa829-54c3-4c65-8596-dca8560ec433" path="/var/lib/kubelet/pods/d9bfa829-54c3-4c65-8596-dca8560ec433/volumes" Oct 31 00:09:25.899010 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:25.898973 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:09:27.231017 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:27.230968 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:09:27.425173 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:27.425124 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:09:29.980810 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:29.980777 4927 scope.go:122] "RemoveContainer" containerID="b64527904348f4d5e2d655dfac4d8fda191042979ea4c2768e86372c9963b6a9" Oct 31 00:09:29.992196 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:29.992166 4927 scope.go:122] "RemoveContainer" containerID="d3ffee3c871aaf87297ff91c164cbe99ed4c4463c0dc38429cc3d8b494b64d7e" Oct 31 00:09:30.002306 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:30.002287 4927 scope.go:122] "RemoveContainer" containerID="9c5d4fd5b410d5b7dc9814577810a2efcb388653d3b474f54e2819f9ed340aae" Oct 31 00:09:30.013566 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:30.013547 4927 scope.go:122] "RemoveContainer" containerID="c73aec392b2d8c6904a0b5f15b7b111f6ddbe9408e1930737fe4e1791d47d3f0" Oct 31 00:09:30.026210 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:30.026188 4927 scope.go:122] "RemoveContainer" containerID="afed5fcb9c4eb942eb2f49c6fe69ed7635be16c3dfc799753d6351802d44ee13" Oct 31 00:09:30.038518 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:30.038496 4927 scope.go:122] "RemoveContainer" containerID="bb9ada750c53abad00d3f18683cbd0419ee40dd9fb79709bd43a8742cb49854b" Oct 31 00:09:30.049522 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:30.049487 4927 scope.go:122] "RemoveContainer" containerID="78a5aeab69e4ddad53aaece7dc68148ed1bf2f4b6f7658b4723f8b9d7f0a1959" Oct 31 00:09:30.060987 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:30.060966 4927 scope.go:122] "RemoveContainer" containerID="80f0db48ea0e62be8d260e2ccc7b732ea90b42c34fc00bdba30e4f6bfa502288" Oct 31 00:09:30.075763 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:30.075736 4927 scope.go:122] "RemoveContainer" containerID="c42659543526c5fb508a31f7433fdb5de3fa9b2afdee03a97d42a92b63faceac" Oct 31 00:09:30.088862 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:30.088838 4927 scope.go:122] "RemoveContainer" containerID="62b7d68856a267415e32f509eabc516f5b0aef9c2c6522dc60570b7509d9d223" Oct 31 00:09:30.100098 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:30.100073 4927 scope.go:122] "RemoveContainer" containerID="72521bbea75240eedc7a0d0a5fd707ca08fb713c932ac30a5a205af782f1902d" Oct 31 00:09:30.110046 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:30.110017 4927 scope.go:122] "RemoveContainer" containerID="86ba12939bb302695c3e154c84a46972603c8532c1c3ee31f5693c9eb7cbdebb" Oct 31 00:09:30.121209 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:30.121185 4927 scope.go:122] "RemoveContainer" containerID="5c6bdaede9cc3fb802e3151b43e4208578d9ed6ff9dd8acc7918de4c5c3c6afe" Oct 31 00:09:30.148674 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:30.148648 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-7w9dq"] Oct 31 00:09:30.148898 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:30.148873 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-7w9dq" podUID="d93f8ab4-0aff-4eba-843d-2c07ab53d3bc" containerName="istio-proxy" containerID="cri-o://7107ae31cd4664933000482cf304270bdb91abc3d8eba2b198d7ac7eb4150851" gracePeriod=30 Oct 31 00:09:30.201938 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:30.201911 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-6gwbm"] Oct 31 00:09:30.202189 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:30.202164 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-6gwbm" podUID="c4095540-8f80-4704-9146-06c5d2cab638" containerName="istio-proxy" containerID="cri-o://b0ffaaa12da8f91486d229c1c0bb92ccbb26d83af61ba90d4486fa24a17e787a" gracePeriod=30 Oct 31 00:09:30.207944 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:30.207922 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-pc4b9"] Oct 31 00:09:30.208125 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:30.208106 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-pc4b9" podUID="83ebf014-df38-445b-b3df-d2b9e8af0595" containerName="istio-proxy" containerID="cri-o://e97887a9d8849a5e30b43a09ac63fe852803aa921fd555e35946136986381b7f" gracePeriod=30 Oct 31 00:09:30.257690 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:30.256204 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-hs4pf"] Oct 31 00:09:30.257690 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:30.256459 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-hs4pf" podUID="9625c102-c118-48b2-9c66-4373c5d839eb" containerName="istio-proxy" containerID="cri-o://7ab257803381068ac6284e73ea5650b7abea666ef936dfb3dbff7ece5af6a0c8" gracePeriod=30 Oct 31 00:09:30.536447 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:30.534417 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:09:33.068880 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:33.068849 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-54ad8c-7b7f75c84c-nntg5" Oct 31 00:09:34.079884 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:34.079856 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-54ad8c-7c679ffcbb-kmx4b" Oct 31 00:09:34.844450 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:34.844410 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-71b76a-7c9bf5b98c-sr6mg"] Oct 31 00:09:34.845555 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:34.845531 4927 cpu_manager.go:414] "containerMap: removing container" podUID="d9bfa829-54c3-4c65-8596-dca8560ec433" containerName="istio-proxy" Oct 31 00:09:34.845608 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:34.845559 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9bfa829-54c3-4c65-8596-dca8560ec433" containerName="istio-proxy" Oct 31 00:09:34.845608 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:34.845579 4927 cpu_manager.go:414] "containerMap: removing container" podUID="1f3dbfce-11a6-4176-9a52-c9772a349dbe" containerName="istio-proxy" Oct 31 00:09:34.845608 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:34.845588 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f3dbfce-11a6-4176-9a52-c9772a349dbe" containerName="istio-proxy" Oct 31 00:09:34.845878 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:34.845859 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9bfa829-54c3-4c65-8596-dca8560ec433" containerName="istio-proxy" Oct 31 00:09:34.845915 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:34.845876 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f3dbfce-11a6-4176-9a52-c9772a349dbe" containerName="istio-proxy" Oct 31 00:09:34.854759 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:34.854731 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-71b76a-7c9bf5b98c-sr6mg" Oct 31 00:09:34.905617 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:34.905583 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-71b76a-7c9bf5b98c-sr6mg"] Oct 31 00:09:34.951182 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:34.951151 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-71b76a-serving-cert\"" Oct 31 00:09:34.951465 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:34.951239 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-2nk58\"" Oct 31 00:09:34.951465 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:34.951387 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 31 00:09:34.991419 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:34.991386 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b306ffa9-b393-4c4f-a00f-9f9722b5a97a-maas-api-tls\") pod \"maas-api-e2e-api-a-71b76a-7c9bf5b98c-sr6mg\" (UID: \"b306ffa9-b393-4c4f-a00f-9f9722b5a97a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-71b76a-7c9bf5b98c-sr6mg" Oct 31 00:09:34.991615 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:34.991566 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b306ffa9-b393-4c4f-a00f-9f9722b5a97a-openshift-service-ca\") pod \"maas-api-e2e-api-a-71b76a-7c9bf5b98c-sr6mg\" (UID: \"b306ffa9-b393-4c4f-a00f-9f9722b5a97a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-71b76a-7c9bf5b98c-sr6mg" Oct 31 00:09:34.991683 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:34.991663 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b306ffa9-b393-4c4f-a00f-9f9722b5a97a-metrics-certs\") pod \"maas-api-e2e-api-a-71b76a-7c9bf5b98c-sr6mg\" (UID: \"b306ffa9-b393-4c4f-a00f-9f9722b5a97a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-71b76a-7c9bf5b98c-sr6mg" Oct 31 00:09:34.991738 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:34.991688 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pdslh\" (UniqueName: \"kubernetes.io/projected/b306ffa9-b393-4c4f-a00f-9f9722b5a97a-kube-api-access-pdslh\") pod \"maas-api-e2e-api-a-71b76a-7c9bf5b98c-sr6mg\" (UID: \"b306ffa9-b393-4c4f-a00f-9f9722b5a97a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-71b76a-7c9bf5b98c-sr6mg" Oct 31 00:09:35.025913 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:35.025873 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:09:35.116416 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:35.114838 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b306ffa9-b393-4c4f-a00f-9f9722b5a97a-metrics-certs\") pod \"maas-api-e2e-api-a-71b76a-7c9bf5b98c-sr6mg\" (UID: \"b306ffa9-b393-4c4f-a00f-9f9722b5a97a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-71b76a-7c9bf5b98c-sr6mg" Oct 31 00:09:35.116416 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:35.114893 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdslh\" (UniqueName: \"kubernetes.io/projected/b306ffa9-b393-4c4f-a00f-9f9722b5a97a-kube-api-access-pdslh\") pod \"maas-api-e2e-api-a-71b76a-7c9bf5b98c-sr6mg\" (UID: \"b306ffa9-b393-4c4f-a00f-9f9722b5a97a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-71b76a-7c9bf5b98c-sr6mg" Oct 31 00:09:35.116416 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:35.115108 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b306ffa9-b393-4c4f-a00f-9f9722b5a97a-maas-api-tls\") pod \"maas-api-e2e-api-a-71b76a-7c9bf5b98c-sr6mg\" (UID: \"b306ffa9-b393-4c4f-a00f-9f9722b5a97a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-71b76a-7c9bf5b98c-sr6mg" Oct 31 00:09:35.116416 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:35.115438 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b306ffa9-b393-4c4f-a00f-9f9722b5a97a-openshift-service-ca\") pod \"maas-api-e2e-api-a-71b76a-7c9bf5b98c-sr6mg\" (UID: \"b306ffa9-b393-4c4f-a00f-9f9722b5a97a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-71b76a-7c9bf5b98c-sr6mg" Oct 31 00:09:35.117180 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:35.116448 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b306ffa9-b393-4c4f-a00f-9f9722b5a97a-openshift-service-ca\") pod \"maas-api-e2e-api-a-71b76a-7c9bf5b98c-sr6mg\" (UID: \"b306ffa9-b393-4c4f-a00f-9f9722b5a97a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-71b76a-7c9bf5b98c-sr6mg" Oct 31 00:09:35.143387 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:35.136148 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b306ffa9-b393-4c4f-a00f-9f9722b5a97a-metrics-certs\") pod \"maas-api-e2e-api-a-71b76a-7c9bf5b98c-sr6mg\" (UID: \"b306ffa9-b393-4c4f-a00f-9f9722b5a97a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-71b76a-7c9bf5b98c-sr6mg" Oct 31 00:09:35.165762 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:35.165567 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b306ffa9-b393-4c4f-a00f-9f9722b5a97a-maas-api-tls\") pod \"maas-api-e2e-api-a-71b76a-7c9bf5b98c-sr6mg\" (UID: \"b306ffa9-b393-4c4f-a00f-9f9722b5a97a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-71b76a-7c9bf5b98c-sr6mg" Oct 31 00:09:35.187254 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:35.186944 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:09:35.197656 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:35.197571 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdslh\" (UniqueName: \"kubernetes.io/projected/b306ffa9-b393-4c4f-a00f-9f9722b5a97a-kube-api-access-pdslh\") pod \"maas-api-e2e-api-a-71b76a-7c9bf5b98c-sr6mg\" (UID: \"b306ffa9-b393-4c4f-a00f-9f9722b5a97a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-71b76a-7c9bf5b98c-sr6mg" Oct 31 00:09:35.370264 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:35.370176 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:09:35.470684 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:35.469484 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-71b76a-7c9bf5b98c-sr6mg" Oct 31 00:09:35.618482 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:35.618454 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-7w9dq" Oct 31 00:09:35.699686 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:35.699657 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-6gwbm" Oct 31 00:09:35.744038 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:35.744001 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d93f8ab4-0aff-4eba-843d-2c07ab53d3bc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d93f8ab4-0aff-4eba-843d-2c07ab53d3bc-istiod-ca-cert\") pod \"d93f8ab4-0aff-4eba-843d-2c07ab53d3bc\" (UID: \"d93f8ab4-0aff-4eba-843d-2c07ab53d3bc\") " Oct 31 00:09:35.744541 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:35.744507 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d93f8ab4-0aff-4eba-843d-2c07ab53d3bc-istiod-ca-cert" pod "d93f8ab4-0aff-4eba-843d-2c07ab53d3bc" (UID: "d93f8ab4-0aff-4eba-843d-2c07ab53d3bc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:09:35.746147 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:35.746122 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d93f8ab4-0aff-4eba-843d-2c07ab53d3bc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d93f8ab4-0aff-4eba-843d-2c07ab53d3bc-workload-socket\") pod \"d93f8ab4-0aff-4eba-843d-2c07ab53d3bc\" (UID: \"d93f8ab4-0aff-4eba-843d-2c07ab53d3bc\") " Oct 31 00:09:35.746258 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:35.746208 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d93f8ab4-0aff-4eba-843d-2c07ab53d3bc-istio-token\" (UniqueName: \"kubernetes.io/projected/d93f8ab4-0aff-4eba-843d-2c07ab53d3bc-istio-token\") pod \"d93f8ab4-0aff-4eba-843d-2c07ab53d3bc\" (UID: \"d93f8ab4-0aff-4eba-843d-2c07ab53d3bc\") " Oct 31 00:09:35.746258 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:35.746242 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d93f8ab4-0aff-4eba-843d-2c07ab53d3bc-kube-api-access-tpmjq\" (UniqueName: \"kubernetes.io/projected/d93f8ab4-0aff-4eba-843d-2c07ab53d3bc-kube-api-access-tpmjq\") pod \"d93f8ab4-0aff-4eba-843d-2c07ab53d3bc\" (UID: \"d93f8ab4-0aff-4eba-843d-2c07ab53d3bc\") " Oct 31 00:09:35.746447 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:35.746306 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d93f8ab4-0aff-4eba-843d-2c07ab53d3bc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d93f8ab4-0aff-4eba-843d-2c07ab53d3bc-istio-podinfo\") pod \"d93f8ab4-0aff-4eba-843d-2c07ab53d3bc\" (UID: \"d93f8ab4-0aff-4eba-843d-2c07ab53d3bc\") " Oct 31 00:09:35.746744 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:35.746720 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d93f8ab4-0aff-4eba-843d-2c07ab53d3bc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d93f8ab4-0aff-4eba-843d-2c07ab53d3bc-credential-socket\") pod \"d93f8ab4-0aff-4eba-843d-2c07ab53d3bc\" (UID: \"d93f8ab4-0aff-4eba-843d-2c07ab53d3bc\") " Oct 31 00:09:35.747691 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:35.746823 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d93f8ab4-0aff-4eba-843d-2c07ab53d3bc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d93f8ab4-0aff-4eba-843d-2c07ab53d3bc-istio-envoy\") pod \"d93f8ab4-0aff-4eba-843d-2c07ab53d3bc\" (UID: \"d93f8ab4-0aff-4eba-843d-2c07ab53d3bc\") " Oct 31 00:09:35.747691 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:35.746885 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d93f8ab4-0aff-4eba-843d-2c07ab53d3bc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d93f8ab4-0aff-4eba-843d-2c07ab53d3bc-istio-data\") pod \"d93f8ab4-0aff-4eba-843d-2c07ab53d3bc\" (UID: \"d93f8ab4-0aff-4eba-843d-2c07ab53d3bc\") " Oct 31 00:09:35.747691 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:35.746960 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d93f8ab4-0aff-4eba-843d-2c07ab53d3bc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d93f8ab4-0aff-4eba-843d-2c07ab53d3bc-workload-certs\") pod \"d93f8ab4-0aff-4eba-843d-2c07ab53d3bc\" (UID: \"d93f8ab4-0aff-4eba-843d-2c07ab53d3bc\") " Oct 31 00:09:35.747691 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:35.747375 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d93f8ab4-0aff-4eba-843d-2c07ab53d3bc-credential-socket" pod "d93f8ab4-0aff-4eba-843d-2c07ab53d3bc" (UID: "d93f8ab4-0aff-4eba-843d-2c07ab53d3bc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:09:35.747691 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:35.747568 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d93f8ab4-0aff-4eba-843d-2c07ab53d3bc-istio-data" pod "d93f8ab4-0aff-4eba-843d-2c07ab53d3bc" (UID: "d93f8ab4-0aff-4eba-843d-2c07ab53d3bc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:09:35.748010 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:35.747739 4927 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d93f8ab4-0aff-4eba-843d-2c07ab53d3bc-credential-socket\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:09:35.748010 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:35.747760 4927 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d93f8ab4-0aff-4eba-843d-2c07ab53d3bc-istio-data\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:09:35.748010 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:35.747775 4927 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d93f8ab4-0aff-4eba-843d-2c07ab53d3bc-istiod-ca-cert\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:09:35.748010 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:35.747829 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d93f8ab4-0aff-4eba-843d-2c07ab53d3bc-workload-socket" pod "d93f8ab4-0aff-4eba-843d-2c07ab53d3bc" (UID: "d93f8ab4-0aff-4eba-843d-2c07ab53d3bc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:09:35.748221 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:35.748004 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d93f8ab4-0aff-4eba-843d-2c07ab53d3bc-workload-certs" pod "d93f8ab4-0aff-4eba-843d-2c07ab53d3bc" (UID: "d93f8ab4-0aff-4eba-843d-2c07ab53d3bc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:09:35.751947 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:35.751910 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d93f8ab4-0aff-4eba-843d-2c07ab53d3bc-istio-envoy" pod "d93f8ab4-0aff-4eba-843d-2c07ab53d3bc" (UID: "d93f8ab4-0aff-4eba-843d-2c07ab53d3bc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:09:35.756368 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:35.752753 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d93f8ab4-0aff-4eba-843d-2c07ab53d3bc-istio-token" pod "d93f8ab4-0aff-4eba-843d-2c07ab53d3bc" (UID: "d93f8ab4-0aff-4eba-843d-2c07ab53d3bc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:09:35.756368 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:35.752950 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d93f8ab4-0aff-4eba-843d-2c07ab53d3bc-istio-podinfo" pod "d93f8ab4-0aff-4eba-843d-2c07ab53d3bc" (UID: "d93f8ab4-0aff-4eba-843d-2c07ab53d3bc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:09:35.757765 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:35.757742 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d93f8ab4-0aff-4eba-843d-2c07ab53d3bc-kube-api-access-tpmjq" pod "d93f8ab4-0aff-4eba-843d-2c07ab53d3bc" (UID: "d93f8ab4-0aff-4eba-843d-2c07ab53d3bc"). InnerVolumeSpecName "kube-api-access-tpmjq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:09:35.759620 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:35.759598 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-hs4pf" Oct 31 00:09:35.849240 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:35.849204 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4095540-8f80-4704-9146-06c5d2cab638-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4095540-8f80-4704-9146-06c5d2cab638-workload-certs\") pod \"c4095540-8f80-4704-9146-06c5d2cab638\" (UID: \"c4095540-8f80-4704-9146-06c5d2cab638\") " Oct 31 00:09:35.849435 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:35.849280 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c4095540-8f80-4704-9146-06c5d2cab638-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4095540-8f80-4704-9146-06c5d2cab638-istio-podinfo\") pod \"c4095540-8f80-4704-9146-06c5d2cab638\" (UID: \"c4095540-8f80-4704-9146-06c5d2cab638\") " Oct 31 00:09:35.849435 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:35.849321 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4095540-8f80-4704-9146-06c5d2cab638-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4095540-8f80-4704-9146-06c5d2cab638-credential-socket\") pod \"c4095540-8f80-4704-9146-06c5d2cab638\" (UID: \"c4095540-8f80-4704-9146-06c5d2cab638\") " Oct 31 00:09:35.849435 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:35.849397 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4095540-8f80-4704-9146-06c5d2cab638-kube-api-access-sqf8z\" (UniqueName: \"kubernetes.io/projected/c4095540-8f80-4704-9146-06c5d2cab638-kube-api-access-sqf8z\") pod \"c4095540-8f80-4704-9146-06c5d2cab638\" (UID: \"c4095540-8f80-4704-9146-06c5d2cab638\") " Oct 31 00:09:35.849435 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:35.849425 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4095540-8f80-4704-9146-06c5d2cab638-istio-token\" (UniqueName: \"kubernetes.io/projected/c4095540-8f80-4704-9146-06c5d2cab638-istio-token\") pod \"c4095540-8f80-4704-9146-06c5d2cab638\" (UID: \"c4095540-8f80-4704-9146-06c5d2cab638\") " Oct 31 00:09:35.849675 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:35.849476 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9625c102-c118-48b2-9c66-4373c5d839eb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9625c102-c118-48b2-9c66-4373c5d839eb-istiod-ca-cert\") pod \"9625c102-c118-48b2-9c66-4373c5d839eb\" (UID: \"9625c102-c118-48b2-9c66-4373c5d839eb\") " Oct 31 00:09:35.849675 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:35.849555 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4095540-8f80-4704-9146-06c5d2cab638-workload-certs" pod "c4095540-8f80-4704-9146-06c5d2cab638" (UID: "c4095540-8f80-4704-9146-06c5d2cab638"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:09:35.849675 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:35.849578 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9625c102-c118-48b2-9c66-4373c5d839eb-istio-token\" (UniqueName: \"kubernetes.io/projected/9625c102-c118-48b2-9c66-4373c5d839eb-istio-token\") pod \"9625c102-c118-48b2-9c66-4373c5d839eb\" (UID: \"9625c102-c118-48b2-9c66-4373c5d839eb\") " Oct 31 00:09:35.849675 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:35.849609 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9625c102-c118-48b2-9c66-4373c5d839eb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9625c102-c118-48b2-9c66-4373c5d839eb-istio-data\") pod \"9625c102-c118-48b2-9c66-4373c5d839eb\" (UID: \"9625c102-c118-48b2-9c66-4373c5d839eb\") " Oct 31 00:09:35.849675 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:35.849639 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4095540-8f80-4704-9146-06c5d2cab638-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4095540-8f80-4704-9146-06c5d2cab638-istio-envoy\") pod \"c4095540-8f80-4704-9146-06c5d2cab638\" (UID: \"c4095540-8f80-4704-9146-06c5d2cab638\") " Oct 31 00:09:35.849940 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:35.849694 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9625c102-c118-48b2-9c66-4373c5d839eb-kube-api-access-jxbr7\" (UniqueName: \"kubernetes.io/projected/9625c102-c118-48b2-9c66-4373c5d839eb-kube-api-access-jxbr7\") pod \"9625c102-c118-48b2-9c66-4373c5d839eb\" (UID: \"9625c102-c118-48b2-9c66-4373c5d839eb\") " Oct 31 00:09:35.849940 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:35.849789 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9625c102-c118-48b2-9c66-4373c5d839eb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9625c102-c118-48b2-9c66-4373c5d839eb-istio-envoy\") pod \"9625c102-c118-48b2-9c66-4373c5d839eb\" (UID: \"9625c102-c118-48b2-9c66-4373c5d839eb\") " Oct 31 00:09:35.849940 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:35.849822 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4095540-8f80-4704-9146-06c5d2cab638-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4095540-8f80-4704-9146-06c5d2cab638-istio-data\") pod \"c4095540-8f80-4704-9146-06c5d2cab638\" (UID: \"c4095540-8f80-4704-9146-06c5d2cab638\") " Oct 31 00:09:35.849940 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:35.849857 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c4095540-8f80-4704-9146-06c5d2cab638-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4095540-8f80-4704-9146-06c5d2cab638-istiod-ca-cert\") pod \"c4095540-8f80-4704-9146-06c5d2cab638\" (UID: \"c4095540-8f80-4704-9146-06c5d2cab638\") " Oct 31 00:09:35.849940 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:35.849925 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9625c102-c118-48b2-9c66-4373c5d839eb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9625c102-c118-48b2-9c66-4373c5d839eb-workload-certs\") pod \"9625c102-c118-48b2-9c66-4373c5d839eb\" (UID: \"9625c102-c118-48b2-9c66-4373c5d839eb\") " Oct 31 00:09:35.850201 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:35.849960 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4095540-8f80-4704-9146-06c5d2cab638-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4095540-8f80-4704-9146-06c5d2cab638-workload-socket\") pod \"c4095540-8f80-4704-9146-06c5d2cab638\" (UID: \"c4095540-8f80-4704-9146-06c5d2cab638\") " Oct 31 00:09:35.850201 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:35.849992 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9625c102-c118-48b2-9c66-4373c5d839eb-istio-data" pod "9625c102-c118-48b2-9c66-4373c5d839eb" (UID: "9625c102-c118-48b2-9c66-4373c5d839eb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:09:35.850201 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:35.850114 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9625c102-c118-48b2-9c66-4373c5d839eb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9625c102-c118-48b2-9c66-4373c5d839eb-workload-socket\") pod \"9625c102-c118-48b2-9c66-4373c5d839eb\" (UID: \"9625c102-c118-48b2-9c66-4373c5d839eb\") " Oct 31 00:09:35.850201 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:35.850151 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9625c102-c118-48b2-9c66-4373c5d839eb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9625c102-c118-48b2-9c66-4373c5d839eb-credential-socket\") pod \"9625c102-c118-48b2-9c66-4373c5d839eb\" (UID: \"9625c102-c118-48b2-9c66-4373c5d839eb\") " Oct 31 00:09:35.850201 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:35.850195 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9625c102-c118-48b2-9c66-4373c5d839eb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9625c102-c118-48b2-9c66-4373c5d839eb-istio-podinfo\") pod \"9625c102-c118-48b2-9c66-4373c5d839eb\" (UID: \"9625c102-c118-48b2-9c66-4373c5d839eb\") " Oct 31 00:09:35.850499 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:35.850199 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4095540-8f80-4704-9146-06c5d2cab638-credential-socket" pod "c4095540-8f80-4704-9146-06c5d2cab638" (UID: "c4095540-8f80-4704-9146-06c5d2cab638"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:09:35.850499 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:35.850302 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9625c102-c118-48b2-9c66-4373c5d839eb-istiod-ca-cert" pod "9625c102-c118-48b2-9c66-4373c5d839eb" (UID: "9625c102-c118-48b2-9c66-4373c5d839eb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:09:35.850715 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:35.850681 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9625c102-c118-48b2-9c66-4373c5d839eb-workload-certs" pod "9625c102-c118-48b2-9c66-4373c5d839eb" (UID: "9625c102-c118-48b2-9c66-4373c5d839eb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:09:35.850937 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:35.850915 4927 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9625c102-c118-48b2-9c66-4373c5d839eb-workload-certs\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:09:35.851033 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:35.850943 4927 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d93f8ab4-0aff-4eba-843d-2c07ab53d3bc-istio-token\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:09:35.851033 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:35.850960 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tpmjq\" (UniqueName: \"kubernetes.io/projected/d93f8ab4-0aff-4eba-843d-2c07ab53d3bc-kube-api-access-tpmjq\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:09:35.851033 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:35.850975 4927 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d93f8ab4-0aff-4eba-843d-2c07ab53d3bc-istio-podinfo\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:09:35.851033 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:35.850990 4927 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4095540-8f80-4704-9146-06c5d2cab638-workload-certs\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:09:35.851033 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:35.851005 4927 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d93f8ab4-0aff-4eba-843d-2c07ab53d3bc-istio-envoy\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:09:35.851033 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:35.851018 4927 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4095540-8f80-4704-9146-06c5d2cab638-credential-socket\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:09:35.851033 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:35.851032 4927 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9625c102-c118-48b2-9c66-4373c5d839eb-istiod-ca-cert\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:09:35.851470 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:35.851047 4927 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d93f8ab4-0aff-4eba-843d-2c07ab53d3bc-workload-certs\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:09:35.851470 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:35.851061 4927 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9625c102-c118-48b2-9c66-4373c5d839eb-istio-data\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:09:35.851470 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:35.851075 4927 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d93f8ab4-0aff-4eba-843d-2c07ab53d3bc-workload-socket\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:09:35.856387 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:35.856262 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9625c102-c118-48b2-9c66-4373c5d839eb-istio-podinfo" pod "9625c102-c118-48b2-9c66-4373c5d839eb" (UID: "9625c102-c118-48b2-9c66-4373c5d839eb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:09:35.861773 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:35.861401 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4095540-8f80-4704-9146-06c5d2cab638-workload-socket" pod "c4095540-8f80-4704-9146-06c5d2cab638" (UID: "c4095540-8f80-4704-9146-06c5d2cab638"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:09:35.862317 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:35.861992 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9625c102-c118-48b2-9c66-4373c5d839eb-istio-envoy" pod "9625c102-c118-48b2-9c66-4373c5d839eb" (UID: "9625c102-c118-48b2-9c66-4373c5d839eb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:09:35.862317 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:35.862040 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9625c102-c118-48b2-9c66-4373c5d839eb-workload-socket" pod "9625c102-c118-48b2-9c66-4373c5d839eb" (UID: "9625c102-c118-48b2-9c66-4373c5d839eb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:09:35.862551 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:35.862376 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4095540-8f80-4704-9146-06c5d2cab638-istio-data" pod "c4095540-8f80-4704-9146-06c5d2cab638" (UID: "c4095540-8f80-4704-9146-06c5d2cab638"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:09:35.862719 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:35.862663 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9625c102-c118-48b2-9c66-4373c5d839eb-credential-socket" pod "9625c102-c118-48b2-9c66-4373c5d839eb" (UID: "9625c102-c118-48b2-9c66-4373c5d839eb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:09:35.864791 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:35.864766 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9625c102-c118-48b2-9c66-4373c5d839eb-kube-api-access-jxbr7" pod "9625c102-c118-48b2-9c66-4373c5d839eb" (UID: "9625c102-c118-48b2-9c66-4373c5d839eb"). InnerVolumeSpecName "kube-api-access-jxbr7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:09:35.868296 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:35.865823 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4095540-8f80-4704-9146-06c5d2cab638-istiod-ca-cert" pod "c4095540-8f80-4704-9146-06c5d2cab638" (UID: "c4095540-8f80-4704-9146-06c5d2cab638"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:09:35.868296 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:35.866674 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4095540-8f80-4704-9146-06c5d2cab638-istio-token" pod "c4095540-8f80-4704-9146-06c5d2cab638" (UID: "c4095540-8f80-4704-9146-06c5d2cab638"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:09:35.869543 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:35.868557 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4095540-8f80-4704-9146-06c5d2cab638-istio-envoy" pod "c4095540-8f80-4704-9146-06c5d2cab638" (UID: "c4095540-8f80-4704-9146-06c5d2cab638"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:09:35.869543 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:35.868649 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9625c102-c118-48b2-9c66-4373c5d839eb-istio-token" pod "9625c102-c118-48b2-9c66-4373c5d839eb" (UID: "9625c102-c118-48b2-9c66-4373c5d839eb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:09:35.869959 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:35.869834 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c4095540-8f80-4704-9146-06c5d2cab638-istio-podinfo" pod "c4095540-8f80-4704-9146-06c5d2cab638" (UID: "c4095540-8f80-4704-9146-06c5d2cab638"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:09:35.870191 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:35.870143 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4095540-8f80-4704-9146-06c5d2cab638-kube-api-access-sqf8z" pod "c4095540-8f80-4704-9146-06c5d2cab638" (UID: "c4095540-8f80-4704-9146-06c5d2cab638"). InnerVolumeSpecName "kube-api-access-sqf8z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:09:35.921540 ip-10-0-141-180 kubenswrapper[4927]: W1031 00:09:35.921499 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb306ffa9_b393_4c4f_a00f_9f9722b5a97a.slice/crio-0bdde3030fbbf95e5ef25b1216e5bf49fe011d24c784da4342282494e7cc29a1 WatchSource:0}: Error finding container 0bdde3030fbbf95e5ef25b1216e5bf49fe011d24c784da4342282494e7cc29a1: Status 404 returned error can't find the container with id 0bdde3030fbbf95e5ef25b1216e5bf49fe011d24c784da4342282494e7cc29a1 Oct 31 00:09:35.926154 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:35.926131 4927 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 00:09:35.929554 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:35.929527 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-71b76a-7c9bf5b98c-sr6mg"] Oct 31 00:09:35.952783 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:35.952563 4927 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4095540-8f80-4704-9146-06c5d2cab638-workload-socket\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:09:35.952783 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:35.952596 4927 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9625c102-c118-48b2-9c66-4373c5d839eb-workload-socket\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:09:35.952783 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:35.952612 4927 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9625c102-c118-48b2-9c66-4373c5d839eb-credential-socket\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:09:35.952783 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:35.952626 4927 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9625c102-c118-48b2-9c66-4373c5d839eb-istio-podinfo\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:09:35.952783 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:35.952640 4927 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4095540-8f80-4704-9146-06c5d2cab638-istio-podinfo\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:09:35.952783 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:35.952657 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sqf8z\" (UniqueName: \"kubernetes.io/projected/c4095540-8f80-4704-9146-06c5d2cab638-kube-api-access-sqf8z\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:09:35.952783 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:35.952673 4927 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4095540-8f80-4704-9146-06c5d2cab638-istio-token\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:09:35.952783 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:35.952686 4927 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9625c102-c118-48b2-9c66-4373c5d839eb-istio-token\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:09:35.952783 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:35.952700 4927 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4095540-8f80-4704-9146-06c5d2cab638-istio-envoy\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:09:35.952783 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:35.952716 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jxbr7\" (UniqueName: \"kubernetes.io/projected/9625c102-c118-48b2-9c66-4373c5d839eb-kube-api-access-jxbr7\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:09:35.952783 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:35.952732 4927 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9625c102-c118-48b2-9c66-4373c5d839eb-istio-envoy\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:09:35.952783 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:35.952746 4927 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4095540-8f80-4704-9146-06c5d2cab638-istio-data\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:09:35.952783 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:35.952760 4927 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4095540-8f80-4704-9146-06c5d2cab638-istiod-ca-cert\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:09:35.991633 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:35.991605 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-pc4b9" Oct 31 00:09:36.155958 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:36.155920 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83ebf014-df38-445b-b3df-d2b9e8af0595-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83ebf014-df38-445b-b3df-d2b9e8af0595-istio-data\") pod \"83ebf014-df38-445b-b3df-d2b9e8af0595\" (UID: \"83ebf014-df38-445b-b3df-d2b9e8af0595\") " Oct 31 00:09:36.156407 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:36.156068 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83ebf014-df38-445b-b3df-d2b9e8af0595-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83ebf014-df38-445b-b3df-d2b9e8af0595-workload-socket\") pod \"83ebf014-df38-445b-b3df-d2b9e8af0595\" (UID: \"83ebf014-df38-445b-b3df-d2b9e8af0595\") " Oct 31 00:09:36.156407 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:36.156116 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83ebf014-df38-445b-b3df-d2b9e8af0595-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83ebf014-df38-445b-b3df-d2b9e8af0595-istio-envoy\") pod \"83ebf014-df38-445b-b3df-d2b9e8af0595\" (UID: \"83ebf014-df38-445b-b3df-d2b9e8af0595\") " Oct 31 00:09:36.156407 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:36.156151 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83ebf014-df38-445b-b3df-d2b9e8af0595-kube-api-access-m7hmq\" (UniqueName: \"kubernetes.io/projected/83ebf014-df38-445b-b3df-d2b9e8af0595-kube-api-access-m7hmq\") pod \"83ebf014-df38-445b-b3df-d2b9e8af0595\" (UID: \"83ebf014-df38-445b-b3df-d2b9e8af0595\") " Oct 31 00:09:36.156407 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:36.156219 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83ebf014-df38-445b-b3df-d2b9e8af0595-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83ebf014-df38-445b-b3df-d2b9e8af0595-credential-socket\") pod \"83ebf014-df38-445b-b3df-d2b9e8af0595\" (UID: \"83ebf014-df38-445b-b3df-d2b9e8af0595\") " Oct 31 00:09:36.156407 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:36.156317 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83ebf014-df38-445b-b3df-d2b9e8af0595-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83ebf014-df38-445b-b3df-d2b9e8af0595-workload-certs\") pod \"83ebf014-df38-445b-b3df-d2b9e8af0595\" (UID: \"83ebf014-df38-445b-b3df-d2b9e8af0595\") " Oct 31 00:09:36.156689 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:36.156556 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/83ebf014-df38-445b-b3df-d2b9e8af0595-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83ebf014-df38-445b-b3df-d2b9e8af0595-istio-podinfo\") pod \"83ebf014-df38-445b-b3df-d2b9e8af0595\" (UID: \"83ebf014-df38-445b-b3df-d2b9e8af0595\") " Oct 31 00:09:36.156747 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:36.156684 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/83ebf014-df38-445b-b3df-d2b9e8af0595-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83ebf014-df38-445b-b3df-d2b9e8af0595-istiod-ca-cert\") pod \"83ebf014-df38-445b-b3df-d2b9e8af0595\" (UID: \"83ebf014-df38-445b-b3df-d2b9e8af0595\") " Oct 31 00:09:36.156809 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:36.156756 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83ebf014-df38-445b-b3df-d2b9e8af0595-istio-token\" (UniqueName: \"kubernetes.io/projected/83ebf014-df38-445b-b3df-d2b9e8af0595-istio-token\") pod \"83ebf014-df38-445b-b3df-d2b9e8af0595\" (UID: \"83ebf014-df38-445b-b3df-d2b9e8af0595\") " Oct 31 00:09:36.158531 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:36.158499 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83ebf014-df38-445b-b3df-d2b9e8af0595-workload-socket" pod "83ebf014-df38-445b-b3df-d2b9e8af0595" (UID: "83ebf014-df38-445b-b3df-d2b9e8af0595"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:09:36.159494 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:36.159425 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83ebf014-df38-445b-b3df-d2b9e8af0595-credential-socket" pod "83ebf014-df38-445b-b3df-d2b9e8af0595" (UID: "83ebf014-df38-445b-b3df-d2b9e8af0595"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:09:36.159494 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:36.159475 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83ebf014-df38-445b-b3df-d2b9e8af0595-istio-data" pod "83ebf014-df38-445b-b3df-d2b9e8af0595" (UID: "83ebf014-df38-445b-b3df-d2b9e8af0595"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:09:36.161021 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:36.160994 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83ebf014-df38-445b-b3df-d2b9e8af0595-workload-certs" pod "83ebf014-df38-445b-b3df-d2b9e8af0595" (UID: "83ebf014-df38-445b-b3df-d2b9e8af0595"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:09:36.165495 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:36.165462 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83ebf014-df38-445b-b3df-d2b9e8af0595-istiod-ca-cert" pod "83ebf014-df38-445b-b3df-d2b9e8af0595" (UID: "83ebf014-df38-445b-b3df-d2b9e8af0595"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:09:36.168673 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:36.168647 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83ebf014-df38-445b-b3df-d2b9e8af0595-istio-envoy" pod "83ebf014-df38-445b-b3df-d2b9e8af0595" (UID: "83ebf014-df38-445b-b3df-d2b9e8af0595"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:09:36.168833 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:36.168814 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83ebf014-df38-445b-b3df-d2b9e8af0595-kube-api-access-m7hmq" pod "83ebf014-df38-445b-b3df-d2b9e8af0595" (UID: "83ebf014-df38-445b-b3df-d2b9e8af0595"). InnerVolumeSpecName "kube-api-access-m7hmq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:09:36.168966 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:36.168935 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83ebf014-df38-445b-b3df-d2b9e8af0595-istio-token" pod "83ebf014-df38-445b-b3df-d2b9e8af0595" (UID: "83ebf014-df38-445b-b3df-d2b9e8af0595"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:09:36.170933 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:36.170907 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/83ebf014-df38-445b-b3df-d2b9e8af0595-istio-podinfo" pod "83ebf014-df38-445b-b3df-d2b9e8af0595" (UID: "83ebf014-df38-445b-b3df-d2b9e8af0595"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:09:36.265909 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:36.264341 4927 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83ebf014-df38-445b-b3df-d2b9e8af0595-istio-token\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:09:36.265909 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:36.264397 4927 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83ebf014-df38-445b-b3df-d2b9e8af0595-istio-data\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:09:36.265909 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:36.264414 4927 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83ebf014-df38-445b-b3df-d2b9e8af0595-workload-socket\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:09:36.265909 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:36.264429 4927 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83ebf014-df38-445b-b3df-d2b9e8af0595-istio-envoy\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:09:36.265909 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:36.264443 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m7hmq\" (UniqueName: \"kubernetes.io/projected/83ebf014-df38-445b-b3df-d2b9e8af0595-kube-api-access-m7hmq\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:09:36.265909 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:36.264458 4927 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83ebf014-df38-445b-b3df-d2b9e8af0595-credential-socket\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:09:36.265909 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:36.264472 4927 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83ebf014-df38-445b-b3df-d2b9e8af0595-workload-certs\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:09:36.265909 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:36.264485 4927 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83ebf014-df38-445b-b3df-d2b9e8af0595-istio-podinfo\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:09:36.265909 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:36.264499 4927 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83ebf014-df38-445b-b3df-d2b9e8af0595-istiod-ca-cert\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:09:36.275669 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:36.275634 4927 generic.go:356] "Generic (PLEG): container finished" podID="83ebf014-df38-445b-b3df-d2b9e8af0595" containerID="e97887a9d8849a5e30b43a09ac63fe852803aa921fd555e35946136986381b7f" exitCode=0 Oct 31 00:09:36.275825 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:36.275675 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-pc4b9" Oct 31 00:09:36.275825 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:36.275709 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-pc4b9" event={"ID":"83ebf014-df38-445b-b3df-d2b9e8af0595","Type":"ContainerDied","Data":"e97887a9d8849a5e30b43a09ac63fe852803aa921fd555e35946136986381b7f"} Oct 31 00:09:36.275825 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:36.275769 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-pc4b9" event={"ID":"83ebf014-df38-445b-b3df-d2b9e8af0595","Type":"ContainerDied","Data":"ce9bb47db9f6682eac44426d6723b40aa4cd90737cc34a6d890129dd822d312f"} Oct 31 00:09:36.275825 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:36.275797 4927 scope.go:122] "RemoveContainer" containerID="e97887a9d8849a5e30b43a09ac63fe852803aa921fd555e35946136986381b7f" Oct 31 00:09:36.277891 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:36.277745 4927 generic.go:356] "Generic (PLEG): container finished" podID="c4095540-8f80-4704-9146-06c5d2cab638" containerID="b0ffaaa12da8f91486d229c1c0bb92ccbb26d83af61ba90d4486fa24a17e787a" exitCode=0 Oct 31 00:09:36.277891 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:36.277827 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-6gwbm" Oct 31 00:09:36.278935 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:36.277837 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-6gwbm" event={"ID":"c4095540-8f80-4704-9146-06c5d2cab638","Type":"ContainerDied","Data":"b0ffaaa12da8f91486d229c1c0bb92ccbb26d83af61ba90d4486fa24a17e787a"} Oct 31 00:09:36.278935 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:36.278773 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-6gwbm" event={"ID":"c4095540-8f80-4704-9146-06c5d2cab638","Type":"ContainerDied","Data":"05a237873bcab7c8bc18b0da2ede80f9c02df0608b78caf27483364c939e15e7"} Oct 31 00:09:36.281348 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:36.279965 4927 generic.go:356] "Generic (PLEG): container finished" podID="9625c102-c118-48b2-9c66-4373c5d839eb" containerID="7ab257803381068ac6284e73ea5650b7abea666ef936dfb3dbff7ece5af6a0c8" exitCode=0 Oct 31 00:09:36.281348 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:36.280081 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-hs4pf" event={"ID":"9625c102-c118-48b2-9c66-4373c5d839eb","Type":"ContainerDied","Data":"7ab257803381068ac6284e73ea5650b7abea666ef936dfb3dbff7ece5af6a0c8"} Oct 31 00:09:36.281348 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:36.280115 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-hs4pf" event={"ID":"9625c102-c118-48b2-9c66-4373c5d839eb","Type":"ContainerDied","Data":"9bcfde161ed4022e22e41159824e223c682c2b5568d72534f4598390e7e4e3d5"} Oct 31 00:09:36.281348 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:36.280206 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-hs4pf" Oct 31 00:09:36.284670 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:36.284632 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-71b76a-7c9bf5b98c-sr6mg" event={"ID":"b306ffa9-b393-4c4f-a00f-9f9722b5a97a","Type":"ContainerStarted","Data":"0bdde3030fbbf95e5ef25b1216e5bf49fe011d24c784da4342282494e7cc29a1"} Oct 31 00:09:36.287661 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:36.287625 4927 generic.go:356] "Generic (PLEG): container finished" podID="d93f8ab4-0aff-4eba-843d-2c07ab53d3bc" containerID="7107ae31cd4664933000482cf304270bdb91abc3d8eba2b198d7ac7eb4150851" exitCode=0 Oct 31 00:09:36.287768 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:36.287734 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-7w9dq" event={"ID":"d93f8ab4-0aff-4eba-843d-2c07ab53d3bc","Type":"ContainerDied","Data":"7107ae31cd4664933000482cf304270bdb91abc3d8eba2b198d7ac7eb4150851"} Oct 31 00:09:36.287845 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:36.287771 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-7w9dq" event={"ID":"d93f8ab4-0aff-4eba-843d-2c07ab53d3bc","Type":"ContainerDied","Data":"64d025af31dd233083670a9318d3fd5f0712b40f8d2a86e8b03de7690a4ab324"} Oct 31 00:09:36.287901 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:36.287855 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-7w9dq" Oct 31 00:09:36.297213 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:36.297183 4927 scope.go:122] "RemoveContainer" containerID="e97887a9d8849a5e30b43a09ac63fe852803aa921fd555e35946136986381b7f" Oct 31 00:09:36.297649 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:09:36.297628 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e97887a9d8849a5e30b43a09ac63fe852803aa921fd555e35946136986381b7f\": container with ID starting with e97887a9d8849a5e30b43a09ac63fe852803aa921fd555e35946136986381b7f not found: ID does not exist" containerID="e97887a9d8849a5e30b43a09ac63fe852803aa921fd555e35946136986381b7f" Oct 31 00:09:36.297752 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:36.297660 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e97887a9d8849a5e30b43a09ac63fe852803aa921fd555e35946136986381b7f"} err="failed to get container status \"e97887a9d8849a5e30b43a09ac63fe852803aa921fd555e35946136986381b7f\": rpc error: code = NotFound desc = could not find container \"e97887a9d8849a5e30b43a09ac63fe852803aa921fd555e35946136986381b7f\": container with ID starting with e97887a9d8849a5e30b43a09ac63fe852803aa921fd555e35946136986381b7f not found: ID does not exist" Oct 31 00:09:36.297752 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:36.297682 4927 scope.go:122] "RemoveContainer" containerID="b0ffaaa12da8f91486d229c1c0bb92ccbb26d83af61ba90d4486fa24a17e787a" Oct 31 00:09:36.309552 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:36.309529 4927 scope.go:122] "RemoveContainer" containerID="b0ffaaa12da8f91486d229c1c0bb92ccbb26d83af61ba90d4486fa24a17e787a" Oct 31 00:09:36.309906 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:09:36.309885 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b0ffaaa12da8f91486d229c1c0bb92ccbb26d83af61ba90d4486fa24a17e787a\": container with ID starting with b0ffaaa12da8f91486d229c1c0bb92ccbb26d83af61ba90d4486fa24a17e787a not found: ID does not exist" containerID="b0ffaaa12da8f91486d229c1c0bb92ccbb26d83af61ba90d4486fa24a17e787a" Oct 31 00:09:36.309975 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:36.309913 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0ffaaa12da8f91486d229c1c0bb92ccbb26d83af61ba90d4486fa24a17e787a"} err="failed to get container status \"b0ffaaa12da8f91486d229c1c0bb92ccbb26d83af61ba90d4486fa24a17e787a\": rpc error: code = NotFound desc = could not find container \"b0ffaaa12da8f91486d229c1c0bb92ccbb26d83af61ba90d4486fa24a17e787a\": container with ID starting with b0ffaaa12da8f91486d229c1c0bb92ccbb26d83af61ba90d4486fa24a17e787a not found: ID does not exist" Oct 31 00:09:36.309975 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:36.309929 4927 scope.go:122] "RemoveContainer" containerID="7ab257803381068ac6284e73ea5650b7abea666ef936dfb3dbff7ece5af6a0c8" Oct 31 00:09:36.319959 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:36.319937 4927 scope.go:122] "RemoveContainer" containerID="7ab257803381068ac6284e73ea5650b7abea666ef936dfb3dbff7ece5af6a0c8" Oct 31 00:09:36.320557 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:09:36.320531 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ab257803381068ac6284e73ea5650b7abea666ef936dfb3dbff7ece5af6a0c8\": container with ID starting with 7ab257803381068ac6284e73ea5650b7abea666ef936dfb3dbff7ece5af6a0c8 not found: ID does not exist" containerID="7ab257803381068ac6284e73ea5650b7abea666ef936dfb3dbff7ece5af6a0c8" Oct 31 00:09:36.320658 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:36.320560 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ab257803381068ac6284e73ea5650b7abea666ef936dfb3dbff7ece5af6a0c8"} err="failed to get container status \"7ab257803381068ac6284e73ea5650b7abea666ef936dfb3dbff7ece5af6a0c8\": rpc error: code = NotFound desc = could not find container \"7ab257803381068ac6284e73ea5650b7abea666ef936dfb3dbff7ece5af6a0c8\": container with ID starting with 7ab257803381068ac6284e73ea5650b7abea666ef936dfb3dbff7ece5af6a0c8 not found: ID does not exist" Oct 31 00:09:36.320658 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:36.320575 4927 scope.go:122] "RemoveContainer" containerID="7107ae31cd4664933000482cf304270bdb91abc3d8eba2b198d7ac7eb4150851" Oct 31 00:09:36.331007 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:36.330986 4927 scope.go:122] "RemoveContainer" containerID="7107ae31cd4664933000482cf304270bdb91abc3d8eba2b198d7ac7eb4150851" Oct 31 00:09:36.331305 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:09:36.331281 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7107ae31cd4664933000482cf304270bdb91abc3d8eba2b198d7ac7eb4150851\": container with ID starting with 7107ae31cd4664933000482cf304270bdb91abc3d8eba2b198d7ac7eb4150851 not found: ID does not exist" containerID="7107ae31cd4664933000482cf304270bdb91abc3d8eba2b198d7ac7eb4150851" Oct 31 00:09:36.331410 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:36.331310 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7107ae31cd4664933000482cf304270bdb91abc3d8eba2b198d7ac7eb4150851"} err="failed to get container status \"7107ae31cd4664933000482cf304270bdb91abc3d8eba2b198d7ac7eb4150851\": rpc error: code = NotFound desc = could not find container \"7107ae31cd4664933000482cf304270bdb91abc3d8eba2b198d7ac7eb4150851\": container with ID starting with 7107ae31cd4664933000482cf304270bdb91abc3d8eba2b198d7ac7eb4150851 not found: ID does not exist" Oct 31 00:09:36.455545 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:36.455448 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-hs4pf"] Oct 31 00:09:36.458302 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:36.458275 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-hs4pf"] Oct 31 00:09:36.568668 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:36.568625 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-pc4b9"] Oct 31 00:09:36.590819 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:36.590781 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-pc4b9"] Oct 31 00:09:36.662574 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:36.662538 4927 patch_prober.go:29] interesting pod/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-6gwbm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:09:36.662772 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:36.662602 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-6gwbm" podUID="c4095540-8f80-4704-9146-06c5d2cab638" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:09:36.707046 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:36.706962 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-6gwbm"] Oct 31 00:09:36.716926 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:36.716889 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-6gwbm"] Oct 31 00:09:36.826564 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:36.826532 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-7w9dq"] Oct 31 00:09:36.832715 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:36.832687 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-7w9dq"] Oct 31 00:09:37.093662 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:37.093572 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="83ebf014-df38-445b-b3df-d2b9e8af0595" path="/var/lib/kubelet/pods/83ebf014-df38-445b-b3df-d2b9e8af0595/volumes" Oct 31 00:09:37.094132 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:37.094114 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9625c102-c118-48b2-9c66-4373c5d839eb" path="/var/lib/kubelet/pods/9625c102-c118-48b2-9c66-4373c5d839eb/volumes" Oct 31 00:09:37.094629 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:37.094611 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c4095540-8f80-4704-9146-06c5d2cab638" path="/var/lib/kubelet/pods/c4095540-8f80-4704-9146-06c5d2cab638/volumes" Oct 31 00:09:37.095402 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:37.095382 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d93f8ab4-0aff-4eba-843d-2c07ab53d3bc" path="/var/lib/kubelet/pods/d93f8ab4-0aff-4eba-843d-2c07ab53d3bc/volumes" Oct 31 00:09:39.821315 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:39.821275 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:09:40.359288 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:40.359254 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-71b76a-7c9bf5b98c-sr6mg" event={"ID":"b306ffa9-b393-4c4f-a00f-9f9722b5a97a","Type":"ContainerStarted","Data":"b4e32f3cba7d477bb4dac18bfa8606f1ca290d495a9612a7461f0f75837097d2"} Oct 31 00:09:40.359528 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:40.359506 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-71b76a-7c9bf5b98c-sr6mg" Oct 31 00:09:40.483223 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:40.483172 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-71b76a-7c9bf5b98c-sr6mg" podStartSLOduration=2.285073015 podStartE2EDuration="6.483157678s" totalImagesPullingTime="4.198084663s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:09:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:09:40.478871808 +0000 UTC m=+1693.515202630" watchObservedRunningTime="2026-10-31 00:09:40.483157678 +0000 UTC m=+1693.519488500" Oct 31 00:09:44.413063 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:44.413031 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:09:46.374459 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:46.374424 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-71b76a-7c9bf5b98c-sr6mg" Oct 31 00:09:48.998934 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:48.998892 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:09:49.680954 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:09:49.680917 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sqv69" containerName="extractor" Oct 31 00:09:53.253346 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:53.253294 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-0d75be-5c9666c858-4qdvv"] Oct 31 00:09:53.254381 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:53.254357 4927 cpu_manager.go:414] "containerMap: removing container" podUID="9625c102-c118-48b2-9c66-4373c5d839eb" containerName="istio-proxy" Oct 31 00:09:53.254381 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:53.254381 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9625c102-c118-48b2-9c66-4373c5d839eb" containerName="istio-proxy" Oct 31 00:09:53.254560 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:53.254426 4927 cpu_manager.go:414] "containerMap: removing container" podUID="83ebf014-df38-445b-b3df-d2b9e8af0595" containerName="istio-proxy" Oct 31 00:09:53.254560 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:53.254434 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83ebf014-df38-445b-b3df-d2b9e8af0595" containerName="istio-proxy" Oct 31 00:09:53.254560 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:53.254479 4927 cpu_manager.go:414] "containerMap: removing container" podUID="c4095540-8f80-4704-9146-06c5d2cab638" containerName="istio-proxy" Oct 31 00:09:53.254560 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:53.254488 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4095540-8f80-4704-9146-06c5d2cab638" containerName="istio-proxy" Oct 31 00:09:53.254560 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:53.254507 4927 cpu_manager.go:414] "containerMap: removing container" podUID="d93f8ab4-0aff-4eba-843d-2c07ab53d3bc" containerName="istio-proxy" Oct 31 00:09:53.254560 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:53.254514 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d93f8ab4-0aff-4eba-843d-2c07ab53d3bc" containerName="istio-proxy" Oct 31 00:09:53.254757 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:53.254683 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="83ebf014-df38-445b-b3df-d2b9e8af0595" containerName="istio-proxy" Oct 31 00:09:53.254757 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:53.254703 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4095540-8f80-4704-9146-06c5d2cab638" containerName="istio-proxy" Oct 31 00:09:53.254757 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:53.254714 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="d93f8ab4-0aff-4eba-843d-2c07ab53d3bc" containerName="istio-proxy" Oct 31 00:09:53.254757 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:53.254728 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="9625c102-c118-48b2-9c66-4373c5d839eb" containerName="istio-proxy" Oct 31 00:09:53.259024 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:53.259004 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-0d75be-5c9666c858-4qdvv" Oct 31 00:09:53.315514 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:53.315481 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-0d75be-5c9666c858-4qdvv"] Oct 31 00:09:53.352472 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:53.352433 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/08022abe-d8cb-4d22-b531-d3f823e938ff-metrics-certs\") pod \"maas-api-e2e-shared-b-w6-0d75be-5c9666c858-4qdvv\" (UID: \"08022abe-d8cb-4d22-b531-d3f823e938ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-0d75be-5c9666c858-4qdvv" Oct 31 00:09:53.352666 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:53.352505 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54n9c\" (UniqueName: \"kubernetes.io/projected/08022abe-d8cb-4d22-b531-d3f823e938ff-kube-api-access-54n9c\") pod \"maas-api-e2e-shared-b-w6-0d75be-5c9666c858-4qdvv\" (UID: \"08022abe-d8cb-4d22-b531-d3f823e938ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-0d75be-5c9666c858-4qdvv" Oct 31 00:09:53.352666 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:53.352645 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/08022abe-d8cb-4d22-b531-d3f823e938ff-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w6-0d75be-5c9666c858-4qdvv\" (UID: \"08022abe-d8cb-4d22-b531-d3f823e938ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-0d75be-5c9666c858-4qdvv" Oct 31 00:09:53.352868 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:53.352672 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/08022abe-d8cb-4d22-b531-d3f823e938ff-maas-api-tls\") pod \"maas-api-e2e-shared-b-w6-0d75be-5c9666c858-4qdvv\" (UID: \"08022abe-d8cb-4d22-b531-d3f823e938ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-0d75be-5c9666c858-4qdvv" Oct 31 00:09:53.360596 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:53.360568 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w6-0d75be-serving-cert\"" Oct 31 00:09:53.454869 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:53.454726 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/08022abe-d8cb-4d22-b531-d3f823e938ff-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w6-0d75be-5c9666c858-4qdvv\" (UID: \"08022abe-d8cb-4d22-b531-d3f823e938ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-0d75be-5c9666c858-4qdvv" Oct 31 00:09:53.454869 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:53.454776 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/08022abe-d8cb-4d22-b531-d3f823e938ff-maas-api-tls\") pod \"maas-api-e2e-shared-b-w6-0d75be-5c9666c858-4qdvv\" (UID: \"08022abe-d8cb-4d22-b531-d3f823e938ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-0d75be-5c9666c858-4qdvv" Oct 31 00:09:53.455193 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:53.454942 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/08022abe-d8cb-4d22-b531-d3f823e938ff-metrics-certs\") pod \"maas-api-e2e-shared-b-w6-0d75be-5c9666c858-4qdvv\" (UID: \"08022abe-d8cb-4d22-b531-d3f823e938ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-0d75be-5c9666c858-4qdvv" Oct 31 00:09:53.455193 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:53.454980 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54n9c\" (UniqueName: \"kubernetes.io/projected/08022abe-d8cb-4d22-b531-d3f823e938ff-kube-api-access-54n9c\") pod \"maas-api-e2e-shared-b-w6-0d75be-5c9666c858-4qdvv\" (UID: \"08022abe-d8cb-4d22-b531-d3f823e938ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-0d75be-5c9666c858-4qdvv" Oct 31 00:09:53.455193 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:09:53.455149 4927 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-0d75be-serving-cert: secret "maas-api-e2e-shared-b-w6-0d75be-serving-cert" not found Oct 31 00:09:53.455469 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:09:53.455230 4927 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/08022abe-d8cb-4d22-b531-d3f823e938ff-maas-api-tls podName:08022abe-d8cb-4d22-b531-d3f823e938ff nodeName:}" failed. No retries permitted until 2026-10-31 00:09:53.95520832 +0000 UTC m=+1706.991539134 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/08022abe-d8cb-4d22-b531-d3f823e938ff-maas-api-tls") pod "maas-api-e2e-shared-b-w6-0d75be-5c9666c858-4qdvv" (UID: "08022abe-d8cb-4d22-b531-d3f823e938ff") : secret "maas-api-e2e-shared-b-w6-0d75be-serving-cert" not found Oct 31 00:09:53.455593 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:53.455563 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/08022abe-d8cb-4d22-b531-d3f823e938ff-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w6-0d75be-5c9666c858-4qdvv\" (UID: \"08022abe-d8cb-4d22-b531-d3f823e938ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-0d75be-5c9666c858-4qdvv" Oct 31 00:09:53.458152 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:53.458125 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/08022abe-d8cb-4d22-b531-d3f823e938ff-metrics-certs\") pod \"maas-api-e2e-shared-b-w6-0d75be-5c9666c858-4qdvv\" (UID: \"08022abe-d8cb-4d22-b531-d3f823e938ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-0d75be-5c9666c858-4qdvv" Oct 31 00:09:53.535813 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:53.527889 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-54n9c\" (UniqueName: \"kubernetes.io/projected/08022abe-d8cb-4d22-b531-d3f823e938ff-kube-api-access-54n9c\") pod \"maas-api-e2e-shared-b-w6-0d75be-5c9666c858-4qdvv\" (UID: \"08022abe-d8cb-4d22-b531-d3f823e938ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-0d75be-5c9666c858-4qdvv" Oct 31 00:09:53.535813 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:53.529207 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:09:53.698357 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:53.697427 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:09:53.846177 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:53.846086 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:09:53.996601 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:53.996571 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/08022abe-d8cb-4d22-b531-d3f823e938ff-maas-api-tls\") pod \"maas-api-e2e-shared-b-w6-0d75be-5c9666c858-4qdvv\" (UID: \"08022abe-d8cb-4d22-b531-d3f823e938ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-0d75be-5c9666c858-4qdvv" Oct 31 00:09:53.998966 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:53.998939 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/08022abe-d8cb-4d22-b531-d3f823e938ff-maas-api-tls\") pod \"maas-api-e2e-shared-b-w6-0d75be-5c9666c858-4qdvv\" (UID: \"08022abe-d8cb-4d22-b531-d3f823e938ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-0d75be-5c9666c858-4qdvv" Oct 31 00:09:54.173887 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:54.173853 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-0d75be-5c9666c858-4qdvv" Oct 31 00:09:54.560449 ip-10-0-141-180 kubenswrapper[4927]: W1031 00:09:54.560415 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08022abe_d8cb_4d22_b531_d3f823e938ff.slice/crio-019e3d977bf4317bb20ae98acac8cdb2a449b7f3c99f47d252b391ad013652b5 WatchSource:0}: Error finding container 019e3d977bf4317bb20ae98acac8cdb2a449b7f3c99f47d252b391ad013652b5: Status 404 returned error can't find the container with id 019e3d977bf4317bb20ae98acac8cdb2a449b7f3c99f47d252b391ad013652b5 Oct 31 00:09:54.560874 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:54.560462 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-0d75be-5c9666c858-4qdvv"] Oct 31 00:09:54.596148 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:54.596114 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-0d75be-5c9666c858-4qdvv" event={"ID":"08022abe-d8cb-4d22-b531-d3f823e938ff","Type":"ContainerStarted","Data":"019e3d977bf4317bb20ae98acac8cdb2a449b7f3c99f47d252b391ad013652b5"} Oct 31 00:09:55.606549 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:55.606505 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-0d75be-5c9666c858-4qdvv" event={"ID":"08022abe-d8cb-4d22-b531-d3f823e938ff","Type":"ContainerStarted","Data":"e5aa7e51db05d9d85bb1de7256afff7f3121a7c7d8b5c405e4df1c6a850a6655"} Oct 31 00:09:55.606921 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:55.606608 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-0d75be-5c9666c858-4qdvv" Oct 31 00:09:55.745719 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:55.745665 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-0d75be-5c9666c858-4qdvv" podStartSLOduration=1.8070445720000001 podStartE2EDuration="2.745649924s" totalImagesPullingTime="938.605352ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:09:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:09:55.744266796 +0000 UTC m=+1708.780597660" watchObservedRunningTime="2026-10-31 00:09:55.745649924 +0000 UTC m=+1708.781980747" Oct 31 00:09:58.193043 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:09:58.193006 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:10:00.572964 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:00.572921 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:10:00.970467 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:00.970422 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4e23415d-mtlsl"] Oct 31 00:10:00.995356 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:00.992907 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e23415d-mtlsl" Oct 31 00:10:01.052056 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:01.052020 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4e23415d-mtlsl"] Oct 31 00:10:01.061802 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:01.061053 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5prt8\" (UniqueName: \"kubernetes.io/projected/358f8214-1852-4a02-880e-3659b48bb92a-kube-api-access-5prt8\") pod \"maas-api-revoke-sub-4e23415d-mtlsl\" (UID: \"358f8214-1852-4a02-880e-3659b48bb92a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e23415d-mtlsl" Oct 31 00:10:01.061802 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:01.061105 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/358f8214-1852-4a02-880e-3659b48bb92a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4e23415d-mtlsl\" (UID: \"358f8214-1852-4a02-880e-3659b48bb92a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e23415d-mtlsl" Oct 31 00:10:01.061802 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:01.061238 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/358f8214-1852-4a02-880e-3659b48bb92a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4e23415d-mtlsl\" (UID: \"358f8214-1852-4a02-880e-3659b48bb92a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e23415d-mtlsl" Oct 31 00:10:01.090399 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:01.090370 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:10:01.090577 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:01.090525 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4fdzs\"" Oct 31 00:10:01.166406 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:01.163689 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5prt8\" (UniqueName: \"kubernetes.io/projected/358f8214-1852-4a02-880e-3659b48bb92a-kube-api-access-5prt8\") pod \"maas-api-revoke-sub-4e23415d-mtlsl\" (UID: \"358f8214-1852-4a02-880e-3659b48bb92a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e23415d-mtlsl" Oct 31 00:10:01.166406 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:01.163747 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/358f8214-1852-4a02-880e-3659b48bb92a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4e23415d-mtlsl\" (UID: \"358f8214-1852-4a02-880e-3659b48bb92a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e23415d-mtlsl" Oct 31 00:10:01.166406 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:01.163838 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/358f8214-1852-4a02-880e-3659b48bb92a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4e23415d-mtlsl\" (UID: \"358f8214-1852-4a02-880e-3659b48bb92a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e23415d-mtlsl" Oct 31 00:10:01.166730 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:01.166604 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/358f8214-1852-4a02-880e-3659b48bb92a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4e23415d-mtlsl\" (UID: \"358f8214-1852-4a02-880e-3659b48bb92a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e23415d-mtlsl" Oct 31 00:10:01.217717 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:01.217682 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/358f8214-1852-4a02-880e-3659b48bb92a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4e23415d-mtlsl\" (UID: \"358f8214-1852-4a02-880e-3659b48bb92a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e23415d-mtlsl" Oct 31 00:10:01.308617 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:01.308528 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-ab0ac9-openshift-default-54c6546bdf-lxwwq"] Oct 31 00:10:01.317354 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:01.314581 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5prt8\" (UniqueName: \"kubernetes.io/projected/358f8214-1852-4a02-880e-3659b48bb92a-kube-api-access-5prt8\") pod \"maas-api-revoke-sub-4e23415d-mtlsl\" (UID: \"358f8214-1852-4a02-880e-3659b48bb92a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e23415d-mtlsl" Oct 31 00:10:01.422179 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:01.417774 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:10:01.463358 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:01.434802 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-ab0ac9-openshift-default-54c6546bdf-lxwwq"] Oct 31 00:10:01.463665 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:01.455508 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-ab0ac9-openshift-default-54c6546bdf-lxwwq" Oct 31 00:10:01.466006 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:01.465717 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e23415d-mtlsl" Oct 31 00:10:01.588355 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:01.585396 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b669d7f-c767-4c4e-a524-6e774fc69034-credential-socket\") pod \"e2e-shared-a-w3-ab0ac9-openshift-default-54c6546bdf-lxwwq\" (UID: \"3b669d7f-c767-4c4e-a524-6e774fc69034\") " pod="openshift-ingress/e2e-shared-a-w3-ab0ac9-openshift-default-54c6546bdf-lxwwq" Oct 31 00:10:01.588355 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:01.585469 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smzd7\" (UniqueName: \"kubernetes.io/projected/3b669d7f-c767-4c4e-a524-6e774fc69034-kube-api-access-smzd7\") pod \"e2e-shared-a-w3-ab0ac9-openshift-default-54c6546bdf-lxwwq\" (UID: \"3b669d7f-c767-4c4e-a524-6e774fc69034\") " pod="openshift-ingress/e2e-shared-a-w3-ab0ac9-openshift-default-54c6546bdf-lxwwq" Oct 31 00:10:01.588355 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:01.585558 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b669d7f-c767-4c4e-a524-6e774fc69034-workload-certs\") pod \"e2e-shared-a-w3-ab0ac9-openshift-default-54c6546bdf-lxwwq\" (UID: \"3b669d7f-c767-4c4e-a524-6e774fc69034\") " pod="openshift-ingress/e2e-shared-a-w3-ab0ac9-openshift-default-54c6546bdf-lxwwq" Oct 31 00:10:01.588355 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:01.585630 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b669d7f-c767-4c4e-a524-6e774fc69034-workload-socket\") pod \"e2e-shared-a-w3-ab0ac9-openshift-default-54c6546bdf-lxwwq\" (UID: \"3b669d7f-c767-4c4e-a524-6e774fc69034\") " pod="openshift-ingress/e2e-shared-a-w3-ab0ac9-openshift-default-54c6546bdf-lxwwq" Oct 31 00:10:01.588355 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:01.585679 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b669d7f-c767-4c4e-a524-6e774fc69034-istio-token\") pod \"e2e-shared-a-w3-ab0ac9-openshift-default-54c6546bdf-lxwwq\" (UID: \"3b669d7f-c767-4c4e-a524-6e774fc69034\") " pod="openshift-ingress/e2e-shared-a-w3-ab0ac9-openshift-default-54c6546bdf-lxwwq" Oct 31 00:10:01.588355 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:01.585744 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b669d7f-c767-4c4e-a524-6e774fc69034-istio-envoy\") pod \"e2e-shared-a-w3-ab0ac9-openshift-default-54c6546bdf-lxwwq\" (UID: \"3b669d7f-c767-4c4e-a524-6e774fc69034\") " pod="openshift-ingress/e2e-shared-a-w3-ab0ac9-openshift-default-54c6546bdf-lxwwq" Oct 31 00:10:01.588355 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:01.585774 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b669d7f-c767-4c4e-a524-6e774fc69034-istio-podinfo\") pod \"e2e-shared-a-w3-ab0ac9-openshift-default-54c6546bdf-lxwwq\" (UID: \"3b669d7f-c767-4c4e-a524-6e774fc69034\") " pod="openshift-ingress/e2e-shared-a-w3-ab0ac9-openshift-default-54c6546bdf-lxwwq" Oct 31 00:10:01.588355 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:01.585802 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b669d7f-c767-4c4e-a524-6e774fc69034-istiod-ca-cert\") pod \"e2e-shared-a-w3-ab0ac9-openshift-default-54c6546bdf-lxwwq\" (UID: \"3b669d7f-c767-4c4e-a524-6e774fc69034\") " pod="openshift-ingress/e2e-shared-a-w3-ab0ac9-openshift-default-54c6546bdf-lxwwq" Oct 31 00:10:01.588355 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:01.585898 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b669d7f-c767-4c4e-a524-6e774fc69034-istio-data\") pod \"e2e-shared-a-w3-ab0ac9-openshift-default-54c6546bdf-lxwwq\" (UID: \"3b669d7f-c767-4c4e-a524-6e774fc69034\") " pod="openshift-ingress/e2e-shared-a-w3-ab0ac9-openshift-default-54c6546bdf-lxwwq" Oct 31 00:10:01.670977 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:01.615298 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-ab0ac9-openshift-default-dockercfg-kgmvq\"" Oct 31 00:10:01.670977 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:01.670878 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-0d75be-5c9666c858-4qdvv" Oct 31 00:10:01.673257 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:01.673224 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:10:01.726557 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:01.698312 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b669d7f-c767-4c4e-a524-6e774fc69034-credential-socket\") pod \"e2e-shared-a-w3-ab0ac9-openshift-default-54c6546bdf-lxwwq\" (UID: \"3b669d7f-c767-4c4e-a524-6e774fc69034\") " pod="openshift-ingress/e2e-shared-a-w3-ab0ac9-openshift-default-54c6546bdf-lxwwq" Oct 31 00:10:01.726557 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:01.698399 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smzd7\" (UniqueName: \"kubernetes.io/projected/3b669d7f-c767-4c4e-a524-6e774fc69034-kube-api-access-smzd7\") pod \"e2e-shared-a-w3-ab0ac9-openshift-default-54c6546bdf-lxwwq\" (UID: \"3b669d7f-c767-4c4e-a524-6e774fc69034\") " pod="openshift-ingress/e2e-shared-a-w3-ab0ac9-openshift-default-54c6546bdf-lxwwq" Oct 31 00:10:01.726557 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:01.698515 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b669d7f-c767-4c4e-a524-6e774fc69034-workload-certs\") pod \"e2e-shared-a-w3-ab0ac9-openshift-default-54c6546bdf-lxwwq\" (UID: \"3b669d7f-c767-4c4e-a524-6e774fc69034\") " pod="openshift-ingress/e2e-shared-a-w3-ab0ac9-openshift-default-54c6546bdf-lxwwq" Oct 31 00:10:01.726557 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:01.698587 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b669d7f-c767-4c4e-a524-6e774fc69034-workload-socket\") pod \"e2e-shared-a-w3-ab0ac9-openshift-default-54c6546bdf-lxwwq\" (UID: \"3b669d7f-c767-4c4e-a524-6e774fc69034\") " pod="openshift-ingress/e2e-shared-a-w3-ab0ac9-openshift-default-54c6546bdf-lxwwq" Oct 31 00:10:01.726557 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:01.698630 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b669d7f-c767-4c4e-a524-6e774fc69034-istio-token\") pod \"e2e-shared-a-w3-ab0ac9-openshift-default-54c6546bdf-lxwwq\" (UID: \"3b669d7f-c767-4c4e-a524-6e774fc69034\") " pod="openshift-ingress/e2e-shared-a-w3-ab0ac9-openshift-default-54c6546bdf-lxwwq" Oct 31 00:10:01.726557 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:01.698695 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b669d7f-c767-4c4e-a524-6e774fc69034-istio-envoy\") pod \"e2e-shared-a-w3-ab0ac9-openshift-default-54c6546bdf-lxwwq\" (UID: \"3b669d7f-c767-4c4e-a524-6e774fc69034\") " pod="openshift-ingress/e2e-shared-a-w3-ab0ac9-openshift-default-54c6546bdf-lxwwq" Oct 31 00:10:01.726557 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:01.698726 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b669d7f-c767-4c4e-a524-6e774fc69034-istio-podinfo\") pod \"e2e-shared-a-w3-ab0ac9-openshift-default-54c6546bdf-lxwwq\" (UID: \"3b669d7f-c767-4c4e-a524-6e774fc69034\") " pod="openshift-ingress/e2e-shared-a-w3-ab0ac9-openshift-default-54c6546bdf-lxwwq" Oct 31 00:10:01.726557 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:01.698770 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b669d7f-c767-4c4e-a524-6e774fc69034-istiod-ca-cert\") pod \"e2e-shared-a-w3-ab0ac9-openshift-default-54c6546bdf-lxwwq\" (UID: \"3b669d7f-c767-4c4e-a524-6e774fc69034\") " pod="openshift-ingress/e2e-shared-a-w3-ab0ac9-openshift-default-54c6546bdf-lxwwq" Oct 31 00:10:01.726557 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:01.698882 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b669d7f-c767-4c4e-a524-6e774fc69034-istio-data\") pod \"e2e-shared-a-w3-ab0ac9-openshift-default-54c6546bdf-lxwwq\" (UID: \"3b669d7f-c767-4c4e-a524-6e774fc69034\") " pod="openshift-ingress/e2e-shared-a-w3-ab0ac9-openshift-default-54c6546bdf-lxwwq" Oct 31 00:10:01.736771 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:01.727538 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b669d7f-c767-4c4e-a524-6e774fc69034-istio-data\") pod \"e2e-shared-a-w3-ab0ac9-openshift-default-54c6546bdf-lxwwq\" (UID: \"3b669d7f-c767-4c4e-a524-6e774fc69034\") " pod="openshift-ingress/e2e-shared-a-w3-ab0ac9-openshift-default-54c6546bdf-lxwwq" Oct 31 00:10:01.736771 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:01.727731 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b669d7f-c767-4c4e-a524-6e774fc69034-credential-socket\") pod \"e2e-shared-a-w3-ab0ac9-openshift-default-54c6546bdf-lxwwq\" (UID: \"3b669d7f-c767-4c4e-a524-6e774fc69034\") " pod="openshift-ingress/e2e-shared-a-w3-ab0ac9-openshift-default-54c6546bdf-lxwwq" Oct 31 00:10:01.736771 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:01.728118 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b669d7f-c767-4c4e-a524-6e774fc69034-workload-certs\") pod \"e2e-shared-a-w3-ab0ac9-openshift-default-54c6546bdf-lxwwq\" (UID: \"3b669d7f-c767-4c4e-a524-6e774fc69034\") " pod="openshift-ingress/e2e-shared-a-w3-ab0ac9-openshift-default-54c6546bdf-lxwwq" Oct 31 00:10:01.736771 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:01.728311 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b669d7f-c767-4c4e-a524-6e774fc69034-workload-socket\") pod \"e2e-shared-a-w3-ab0ac9-openshift-default-54c6546bdf-lxwwq\" (UID: \"3b669d7f-c767-4c4e-a524-6e774fc69034\") " pod="openshift-ingress/e2e-shared-a-w3-ab0ac9-openshift-default-54c6546bdf-lxwwq" Oct 31 00:10:01.736771 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:01.729019 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b669d7f-c767-4c4e-a524-6e774fc69034-istiod-ca-cert\") pod \"e2e-shared-a-w3-ab0ac9-openshift-default-54c6546bdf-lxwwq\" (UID: \"3b669d7f-c767-4c4e-a524-6e774fc69034\") " pod="openshift-ingress/e2e-shared-a-w3-ab0ac9-openshift-default-54c6546bdf-lxwwq" Oct 31 00:10:01.736771 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:01.729495 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b669d7f-c767-4c4e-a524-6e774fc69034-istio-podinfo\") pod \"e2e-shared-a-w3-ab0ac9-openshift-default-54c6546bdf-lxwwq\" (UID: \"3b669d7f-c767-4c4e-a524-6e774fc69034\") " pod="openshift-ingress/e2e-shared-a-w3-ab0ac9-openshift-default-54c6546bdf-lxwwq" Oct 31 00:10:01.761425 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:01.761324 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b669d7f-c767-4c4e-a524-6e774fc69034-istio-envoy\") pod \"e2e-shared-a-w3-ab0ac9-openshift-default-54c6546bdf-lxwwq\" (UID: \"3b669d7f-c767-4c4e-a524-6e774fc69034\") " pod="openshift-ingress/e2e-shared-a-w3-ab0ac9-openshift-default-54c6546bdf-lxwwq" Oct 31 00:10:01.786017 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:01.785956 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-smzd7\" (UniqueName: \"kubernetes.io/projected/3b669d7f-c767-4c4e-a524-6e774fc69034-kube-api-access-smzd7\") pod \"e2e-shared-a-w3-ab0ac9-openshift-default-54c6546bdf-lxwwq\" (UID: \"3b669d7f-c767-4c4e-a524-6e774fc69034\") " pod="openshift-ingress/e2e-shared-a-w3-ab0ac9-openshift-default-54c6546bdf-lxwwq" Oct 31 00:10:01.786604 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:01.786576 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b669d7f-c767-4c4e-a524-6e774fc69034-istio-token\") pod \"e2e-shared-a-w3-ab0ac9-openshift-default-54c6546bdf-lxwwq\" (UID: \"3b669d7f-c767-4c4e-a524-6e774fc69034\") " pod="openshift-ingress/e2e-shared-a-w3-ab0ac9-openshift-default-54c6546bdf-lxwwq" Oct 31 00:10:02.057170 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:02.057116 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-ab0ac9-openshift-default-54c6546bdf-lxwwq" Oct 31 00:10:02.188351 ip-10-0-141-180 kubenswrapper[4927]: W1031 00:10:02.188303 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod358f8214_1852_4a02_880e_3659b48bb92a.slice/crio-2835e7b695fab414e13ba336556b8fe089b6e0fd5e8c39ffbf0236fc5284bfb1 WatchSource:0}: Error finding container 2835e7b695fab414e13ba336556b8fe089b6e0fd5e8c39ffbf0236fc5284bfb1: Status 404 returned error can't find the container with id 2835e7b695fab414e13ba336556b8fe089b6e0fd5e8c39ffbf0236fc5284bfb1 Oct 31 00:10:02.188762 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:02.188737 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4e23415d-mtlsl"] Oct 31 00:10:02.484394 ip-10-0-141-180 kubenswrapper[4927]: W1031 00:10:02.484322 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b669d7f_c767_4c4e_a524_6e774fc69034.slice/crio-2b98a1f2be9d1217e59058d87a39b60061352fdea6585f2400892a4fedcf1da5 WatchSource:0}: Error finding container 2b98a1f2be9d1217e59058d87a39b60061352fdea6585f2400892a4fedcf1da5: Status 404 returned error can't find the container with id 2b98a1f2be9d1217e59058d87a39b60061352fdea6585f2400892a4fedcf1da5 Oct 31 00:10:02.487007 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:02.486983 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-ab0ac9-openshift-default-54c6546bdf-lxwwq"] Oct 31 00:10:02.487663 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:02.487624 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:10:02.487730 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:02.487713 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:10:02.487796 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:02.487780 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:10:02.550368 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:02.550312 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-d5360e-788c77fc85-gkswq"] Oct 31 00:10:02.557860 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:02.557821 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-d5360e-788c77fc85-gkswq" Oct 31 00:10:02.605544 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:02.605396 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-d5360e-788c77fc85-gkswq"] Oct 31 00:10:02.634444 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:02.618265 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w0-d5360e-serving-cert\"" Oct 31 00:10:02.648939 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:02.648315 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e3fe218a-9831-44c3-a5eb-09e14994459f-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-d5360e-788c77fc85-gkswq\" (UID: \"e3fe218a-9831-44c3-a5eb-09e14994459f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-d5360e-788c77fc85-gkswq" Oct 31 00:10:02.648939 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:02.648472 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e3fe218a-9831-44c3-a5eb-09e14994459f-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-d5360e-788c77fc85-gkswq\" (UID: \"e3fe218a-9831-44c3-a5eb-09e14994459f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-d5360e-788c77fc85-gkswq" Oct 31 00:10:02.648939 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:02.648581 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e3fe218a-9831-44c3-a5eb-09e14994459f-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-d5360e-788c77fc85-gkswq\" (UID: \"e3fe218a-9831-44c3-a5eb-09e14994459f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-d5360e-788c77fc85-gkswq" Oct 31 00:10:02.648939 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:02.648682 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5cps\" (UniqueName: \"kubernetes.io/projected/e3fe218a-9831-44c3-a5eb-09e14994459f-kube-api-access-p5cps\") pod \"maas-api-e2e-shared-a-w0-d5360e-788c77fc85-gkswq\" (UID: \"e3fe218a-9831-44c3-a5eb-09e14994459f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-d5360e-788c77fc85-gkswq" Oct 31 00:10:02.750136 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:02.750041 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e3fe218a-9831-44c3-a5eb-09e14994459f-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-d5360e-788c77fc85-gkswq\" (UID: \"e3fe218a-9831-44c3-a5eb-09e14994459f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-d5360e-788c77fc85-gkswq" Oct 31 00:10:02.750766 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:02.750740 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e3fe218a-9831-44c3-a5eb-09e14994459f-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-d5360e-788c77fc85-gkswq\" (UID: \"e3fe218a-9831-44c3-a5eb-09e14994459f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-d5360e-788c77fc85-gkswq" Oct 31 00:10:02.751228 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:10:02.750835 4927 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-d5360e-serving-cert: secret "maas-api-e2e-shared-a-w0-d5360e-serving-cert" not found Oct 31 00:10:02.751228 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:02.750990 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e3fe218a-9831-44c3-a5eb-09e14994459f-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-d5360e-788c77fc85-gkswq\" (UID: \"e3fe218a-9831-44c3-a5eb-09e14994459f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-d5360e-788c77fc85-gkswq" Oct 31 00:10:02.751228 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:10:02.751006 4927 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3fe218a-9831-44c3-a5eb-09e14994459f-maas-api-tls podName:e3fe218a-9831-44c3-a5eb-09e14994459f nodeName:}" failed. No retries permitted until 2026-10-31 00:10:03.250990445 +0000 UTC m=+1716.287321244 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e3fe218a-9831-44c3-a5eb-09e14994459f-maas-api-tls") pod "maas-api-e2e-shared-a-w0-d5360e-788c77fc85-gkswq" (UID: "e3fe218a-9831-44c3-a5eb-09e14994459f") : secret "maas-api-e2e-shared-a-w0-d5360e-serving-cert" not found Oct 31 00:10:02.751228 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:02.751081 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5cps\" (UniqueName: \"kubernetes.io/projected/e3fe218a-9831-44c3-a5eb-09e14994459f-kube-api-access-p5cps\") pod \"maas-api-e2e-shared-a-w0-d5360e-788c77fc85-gkswq\" (UID: \"e3fe218a-9831-44c3-a5eb-09e14994459f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-d5360e-788c77fc85-gkswq" Oct 31 00:10:02.751883 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:02.751689 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e3fe218a-9831-44c3-a5eb-09e14994459f-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-d5360e-788c77fc85-gkswq\" (UID: \"e3fe218a-9831-44c3-a5eb-09e14994459f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-d5360e-788c77fc85-gkswq" Oct 31 00:10:02.753499 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:02.753467 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e3fe218a-9831-44c3-a5eb-09e14994459f-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-d5360e-788c77fc85-gkswq\" (UID: \"e3fe218a-9831-44c3-a5eb-09e14994459f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-d5360e-788c77fc85-gkswq" Oct 31 00:10:02.754817 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:02.754794 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:10:02.776421 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:02.776379 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e23415d-mtlsl" event={"ID":"358f8214-1852-4a02-880e-3659b48bb92a","Type":"ContainerStarted","Data":"d082b3e8a811385fe51b2a4d6dffdb4b0020fa0afc5caa5bf30562be21ab7651"} Oct 31 00:10:02.777023 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:02.776678 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e23415d-mtlsl" event={"ID":"358f8214-1852-4a02-880e-3659b48bb92a","Type":"ContainerStarted","Data":"2835e7b695fab414e13ba336556b8fe089b6e0fd5e8c39ffbf0236fc5284bfb1"} Oct 31 00:10:02.814964 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:02.813763 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-ab0ac9-openshift-default-54c6546bdf-lxwwq" event={"ID":"3b669d7f-c767-4c4e-a524-6e774fc69034","Type":"ContainerStarted","Data":"e2cdeea856504f4f745d34fae7ed87897794b11dec004edbaf555fde8f4bb24e"} Oct 31 00:10:02.814964 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:02.813846 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-ab0ac9-openshift-default-54c6546bdf-lxwwq" event={"ID":"3b669d7f-c767-4c4e-a524-6e774fc69034","Type":"ContainerStarted","Data":"2b98a1f2be9d1217e59058d87a39b60061352fdea6585f2400892a4fedcf1da5"} Oct 31 00:10:02.893535 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:02.893229 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5cps\" (UniqueName: \"kubernetes.io/projected/e3fe218a-9831-44c3-a5eb-09e14994459f-kube-api-access-p5cps\") pod \"maas-api-e2e-shared-a-w0-d5360e-788c77fc85-gkswq\" (UID: \"e3fe218a-9831-44c3-a5eb-09e14994459f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-d5360e-788c77fc85-gkswq" Oct 31 00:10:02.939530 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:02.939462 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e23415d-mtlsl" podStartSLOduration=2.939443354 podStartE2EDuration="2.939443354s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:10:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:10:02.934701071 +0000 UTC m=+1715.971031906" watchObservedRunningTime="2026-10-31 00:10:02.939443354 +0000 UTC m=+1715.975774175" Oct 31 00:10:02.954289 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:02.954258 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:10:03.064412 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:03.061653 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-ab0ac9-openshift-default-54c6546bdf-lxwwq" Oct 31 00:10:03.064412 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:03.061797 4927 patch_prober.go:29] interesting pod/e2e-shared-a-w3-ab0ac9-openshift-default-54c6546bdf-lxwwq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused" start-of-body= Oct 31 00:10:03.064412 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:03.061841 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-ab0ac9-openshift-default-54c6546bdf-lxwwq" podUID="3b669d7f-c767-4c4e-a524-6e774fc69034" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused" Oct 31 00:10:03.128969 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:03.128880 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:10:03.299355 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:03.292323 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e3fe218a-9831-44c3-a5eb-09e14994459f-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-d5360e-788c77fc85-gkswq\" (UID: \"e3fe218a-9831-44c3-a5eb-09e14994459f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-d5360e-788c77fc85-gkswq" Oct 31 00:10:03.299355 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:03.296407 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e3fe218a-9831-44c3-a5eb-09e14994459f-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-d5360e-788c77fc85-gkswq\" (UID: \"e3fe218a-9831-44c3-a5eb-09e14994459f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-d5360e-788c77fc85-gkswq" Oct 31 00:10:03.496666 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:03.496634 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-d5360e-788c77fc85-gkswq" Oct 31 00:10:03.842461 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:03.842365 4927 generic.go:356] "Generic (PLEG): container finished" podID="358f8214-1852-4a02-880e-3659b48bb92a" containerID="d082b3e8a811385fe51b2a4d6dffdb4b0020fa0afc5caa5bf30562be21ab7651" exitCode=0 Oct 31 00:10:03.842461 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:03.842410 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e23415d-mtlsl" event={"ID":"358f8214-1852-4a02-880e-3659b48bb92a","Type":"ContainerDied","Data":"d082b3e8a811385fe51b2a4d6dffdb4b0020fa0afc5caa5bf30562be21ab7651"} Oct 31 00:10:03.950061 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:03.949993 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-ab0ac9-openshift-default-54c6546bdf-lxwwq" podStartSLOduration=2.949973848 podStartE2EDuration="2.949973848s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:10:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:10:03.079612933 +0000 UTC m=+1716.115943756" watchObservedRunningTime="2026-10-31 00:10:03.949973848 +0000 UTC m=+1716.986304670" Oct 31 00:10:03.965084 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:03.965049 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-d5360e-788c77fc85-gkswq"] Oct 31 00:10:04.064959 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:04.064926 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-ab0ac9-openshift-default-54c6546bdf-lxwwq" Oct 31 00:10:04.852295 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:04.852253 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-d5360e-788c77fc85-gkswq" event={"ID":"e3fe218a-9831-44c3-a5eb-09e14994459f","Type":"ContainerStarted","Data":"502882133dbbfd58969cd003340eaf3262df532050fb38ef651e3629d5336a8a"} Oct 31 00:10:04.852776 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:04.852622 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-ab0ac9-openshift-default-54c6546bdf-lxwwq" Oct 31 00:10:04.854923 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:04.854898 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-ab0ac9-openshift-default-54c6546bdf-lxwwq" Oct 31 00:10:05.147364 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:05.147259 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e23415d-mtlsl" Oct 31 00:10:05.254862 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:05.254819 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/358f8214-1852-4a02-880e-3659b48bb92a-kube-api-access-5prt8\" (UniqueName: \"kubernetes.io/projected/358f8214-1852-4a02-880e-3659b48bb92a-kube-api-access-5prt8\") pod \"358f8214-1852-4a02-880e-3659b48bb92a\" (UID: \"358f8214-1852-4a02-880e-3659b48bb92a\") " Oct 31 00:10:05.255081 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:05.254934 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/358f8214-1852-4a02-880e-3659b48bb92a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/358f8214-1852-4a02-880e-3659b48bb92a-maas-api-openshift-ca\") pod \"358f8214-1852-4a02-880e-3659b48bb92a\" (UID: \"358f8214-1852-4a02-880e-3659b48bb92a\") " Oct 31 00:10:05.255081 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:05.255003 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/358f8214-1852-4a02-880e-3659b48bb92a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/358f8214-1852-4a02-880e-3659b48bb92a-maas-api-xks-ca\") pod \"358f8214-1852-4a02-880e-3659b48bb92a\" (UID: \"358f8214-1852-4a02-880e-3659b48bb92a\") " Oct 31 00:10:05.255371 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:05.255310 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/358f8214-1852-4a02-880e-3659b48bb92a-maas-api-openshift-ca" pod "358f8214-1852-4a02-880e-3659b48bb92a" (UID: "358f8214-1852-4a02-880e-3659b48bb92a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:10:05.255716 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:05.255694 4927 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/358f8214-1852-4a02-880e-3659b48bb92a-maas-api-openshift-ca\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:10:05.257321 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:05.257292 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/358f8214-1852-4a02-880e-3659b48bb92a-kube-api-access-5prt8" pod "358f8214-1852-4a02-880e-3659b48bb92a" (UID: "358f8214-1852-4a02-880e-3659b48bb92a"). InnerVolumeSpecName "kube-api-access-5prt8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:10:05.257489 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:05.257365 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/358f8214-1852-4a02-880e-3659b48bb92a-maas-api-xks-ca" pod "358f8214-1852-4a02-880e-3659b48bb92a" (UID: "358f8214-1852-4a02-880e-3659b48bb92a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:10:05.357854 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:05.357814 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5prt8\" (UniqueName: \"kubernetes.io/projected/358f8214-1852-4a02-880e-3659b48bb92a-kube-api-access-5prt8\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:10:05.357854 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:05.357855 4927 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/358f8214-1852-4a02-880e-3659b48bb92a-maas-api-xks-ca\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:10:05.863875 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:05.863835 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-d5360e-788c77fc85-gkswq" event={"ID":"e3fe218a-9831-44c3-a5eb-09e14994459f","Type":"ContainerStarted","Data":"fdc6c293ed30d34db466ce67c69be2fb33cd7c9ad334aac595af65735a2fc755"} Oct 31 00:10:05.864385 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:05.863934 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-d5360e-788c77fc85-gkswq" Oct 31 00:10:05.865515 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:05.865490 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e23415d-mtlsl" Oct 31 00:10:05.865624 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:05.865521 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e23415d-mtlsl" event={"ID":"358f8214-1852-4a02-880e-3659b48bb92a","Type":"ContainerDied","Data":"2835e7b695fab414e13ba336556b8fe089b6e0fd5e8c39ffbf0236fc5284bfb1"} Oct 31 00:10:05.865624 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:05.865559 4927 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2835e7b695fab414e13ba336556b8fe089b6e0fd5e8c39ffbf0236fc5284bfb1" Oct 31 00:10:05.872522 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:05.872497 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:10:06.022842 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:06.020880 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-d5360e-788c77fc85-gkswq" podStartSLOduration=3.067867278 podStartE2EDuration="4.020854129s" totalImagesPullingTime="952.986851ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:10:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:10:06.005563886 +0000 UTC m=+1719.041894708" watchObservedRunningTime="2026-10-31 00:10:06.020854129 +0000 UTC m=+1719.057184952" Oct 31 00:10:06.040366 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:06.038171 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:10:06.250359 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:06.249232 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:10:07.974119 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:07.971002 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:10:08.060959 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:08.060916 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f77eb00c-hprr6"] Oct 31 00:10:08.113414 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:08.113372 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f77eb00c-hprr6"] Oct 31 00:10:09.087939 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:09.087905 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="541d0cfd-a005-480d-84e7-a345701a23a1" path="/var/lib/kubelet/pods/541d0cfd-a005-480d-84e7-a345701a23a1/volumes" Oct 31 00:10:10.684669 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:10.684626 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:10:10.940444 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:10.938760 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:10:11.035352 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:11.026980 4927 generic.go:356] "Generic (PLEG): container finished" podID="c51c586a-f2d8-48f9-8813-3b6e4f360e5e" containerID="aabe1a791b1aa93cf5e47255b266a37d6cdc8ec03231adcc7bae54204476a0b3" exitCode=0 Oct 31 00:10:11.035352 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:11.027146 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-71b76a-openshift-default-77db967c97-lh4ld" event={"ID":"c51c586a-f2d8-48f9-8813-3b6e4f360e5e","Type":"ContainerDied","Data":"aabe1a791b1aa93cf5e47255b266a37d6cdc8ec03231adcc7bae54204476a0b3"} Oct 31 00:10:11.035352 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:11.027837 4927 scope.go:122] "RemoveContainer" containerID="aabe1a791b1aa93cf5e47255b266a37d6cdc8ec03231adcc7bae54204476a0b3" Oct 31 00:10:11.035352 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:11.032268 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:10:11.035352 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:11.032365 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:10:11.035352 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:11.032456 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:10:11.576545 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:11.576479 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-ab0ac9-7978cbf74c-qxdvb"] Oct 31 00:10:11.578774 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:11.578744 4927 cpu_manager.go:414] "containerMap: removing container" podUID="358f8214-1852-4a02-880e-3659b48bb92a" containerName="revoke-keys" Oct 31 00:10:11.578774 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:11.578771 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="358f8214-1852-4a02-880e-3659b48bb92a" containerName="revoke-keys" Oct 31 00:10:11.579196 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:11.579178 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="358f8214-1852-4a02-880e-3659b48bb92a" containerName="revoke-keys" Oct 31 00:10:11.588766 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:11.588724 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-ab0ac9-7978cbf74c-qxdvb"] Oct 31 00:10:11.588848 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:11.588779 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-ab0ac9-847986794b-bbf7n"] Oct 31 00:10:11.588907 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:11.588892 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-ab0ac9-7978cbf74c-qxdvb" Oct 31 00:10:11.594567 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:11.594543 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-ab0ac9-847986794b-bbf7n"] Oct 31 00:10:11.594670 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:11.594654 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-ab0ac9-847986794b-bbf7n" Oct 31 00:10:11.629597 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:11.629560 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-849gc\" (UniqueName: \"kubernetes.io/projected/5a2e97c5-6cd5-498d-b83f-13e94fc34b28-kube-api-access-849gc\") pod \"payload-processing-e2e-shared-a-w3-ab0ac9-847986794b-bbf7n\" (UID: \"5a2e97c5-6cd5-498d-b83f-13e94fc34b28\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-ab0ac9-847986794b-bbf7n" Oct 31 00:10:11.629767 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:11.629616 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6j6g\" (UniqueName: \"kubernetes.io/projected/6874430f-3348-469c-adc3-a605fa1ff16a-kube-api-access-m6j6g\") pod \"payload-pre-processing-e2e-shared-a-w3-ab0ac9-7978cbf74c-qxdvb\" (UID: \"6874430f-3348-469c-adc3-a605fa1ff16a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-ab0ac9-7978cbf74c-qxdvb" Oct 31 00:10:11.630019 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:11.629975 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a2e97c5-6cd5-498d-b83f-13e94fc34b28-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-ab0ac9-847986794b-bbf7n\" (UID: \"5a2e97c5-6cd5-498d-b83f-13e94fc34b28\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-ab0ac9-847986794b-bbf7n" Oct 31 00:10:11.630375 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:11.630351 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6874430f-3348-469c-adc3-a605fa1ff16a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-ab0ac9-7978cbf74c-qxdvb\" (UID: \"6874430f-3348-469c-adc3-a605fa1ff16a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-ab0ac9-7978cbf74c-qxdvb" Oct 31 00:10:11.642540 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:11.642513 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-ab0ac9-dockercfg-7zvzb\"" Oct 31 00:10:11.643085 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:11.643065 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-ab0ac9\"" Oct 31 00:10:11.731526 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:11.731486 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-849gc\" (UniqueName: \"kubernetes.io/projected/5a2e97c5-6cd5-498d-b83f-13e94fc34b28-kube-api-access-849gc\") pod \"payload-processing-e2e-shared-a-w3-ab0ac9-847986794b-bbf7n\" (UID: \"5a2e97c5-6cd5-498d-b83f-13e94fc34b28\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-ab0ac9-847986794b-bbf7n" Oct 31 00:10:11.732007 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:11.731535 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6j6g\" (UniqueName: \"kubernetes.io/projected/6874430f-3348-469c-adc3-a605fa1ff16a-kube-api-access-m6j6g\") pod \"payload-pre-processing-e2e-shared-a-w3-ab0ac9-7978cbf74c-qxdvb\" (UID: \"6874430f-3348-469c-adc3-a605fa1ff16a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-ab0ac9-7978cbf74c-qxdvb" Oct 31 00:10:11.732007 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:11.731636 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a2e97c5-6cd5-498d-b83f-13e94fc34b28-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-ab0ac9-847986794b-bbf7n\" (UID: \"5a2e97c5-6cd5-498d-b83f-13e94fc34b28\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-ab0ac9-847986794b-bbf7n" Oct 31 00:10:11.732007 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:11.731761 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6874430f-3348-469c-adc3-a605fa1ff16a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-ab0ac9-7978cbf74c-qxdvb\" (UID: \"6874430f-3348-469c-adc3-a605fa1ff16a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-ab0ac9-7978cbf74c-qxdvb" Oct 31 00:10:11.732632 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:11.732609 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a2e97c5-6cd5-498d-b83f-13e94fc34b28-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-ab0ac9-847986794b-bbf7n\" (UID: \"5a2e97c5-6cd5-498d-b83f-13e94fc34b28\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-ab0ac9-847986794b-bbf7n" Oct 31 00:10:11.732736 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:11.732711 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6874430f-3348-469c-adc3-a605fa1ff16a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-ab0ac9-7978cbf74c-qxdvb\" (UID: \"6874430f-3348-469c-adc3-a605fa1ff16a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-ab0ac9-7978cbf74c-qxdvb" Oct 31 00:10:11.793775 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:11.793698 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6j6g\" (UniqueName: \"kubernetes.io/projected/6874430f-3348-469c-adc3-a605fa1ff16a-kube-api-access-m6j6g\") pod \"payload-pre-processing-e2e-shared-a-w3-ab0ac9-7978cbf74c-qxdvb\" (UID: \"6874430f-3348-469c-adc3-a605fa1ff16a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-ab0ac9-7978cbf74c-qxdvb" Oct 31 00:10:11.793943 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:11.793856 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-849gc\" (UniqueName: \"kubernetes.io/projected/5a2e97c5-6cd5-498d-b83f-13e94fc34b28-kube-api-access-849gc\") pod \"payload-processing-e2e-shared-a-w3-ab0ac9-847986794b-bbf7n\" (UID: \"5a2e97c5-6cd5-498d-b83f-13e94fc34b28\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-ab0ac9-847986794b-bbf7n" Oct 31 00:10:11.831191 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:11.831143 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:10:11.879491 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:11.879458 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-d5360e-788c77fc85-gkswq" Oct 31 00:10:11.913726 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:11.913686 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-ab0ac9-7978cbf74c-qxdvb" Oct 31 00:10:11.919800 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:11.919772 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-ab0ac9-847986794b-bbf7n" Oct 31 00:10:12.047672 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:12.047490 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-71b76a-openshift-default-77db967c97-lh4ld" event={"ID":"c51c586a-f2d8-48f9-8813-3b6e4f360e5e","Type":"ContainerStarted","Data":"2a76af001dfaa02e742381689a56aec54d0870d03c8513bf137af374832cd231"} Oct 31 00:10:12.257738 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:12.257688 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-54ad8c-5c7f67b585-gn54m"] Oct 31 00:10:12.264417 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:12.264393 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-54ad8c-5c7f67b585-gn54m" Oct 31 00:10:12.326152 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:12.325605 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w4-54ad8c-serving-cert\"" Oct 31 00:10:12.352983 ip-10-0-141-180 kubenswrapper[4927]: W1031 00:10:12.352946 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6874430f_3348_469c_adc3_a605fa1ff16a.slice/crio-e820c91c27c945ee5a30e9759980e4561a049d39b78dd0f5266ab93b86e91d66 WatchSource:0}: Error finding container e820c91c27c945ee5a30e9759980e4561a049d39b78dd0f5266ab93b86e91d66: Status 404 returned error can't find the container with id e820c91c27c945ee5a30e9759980e4561a049d39b78dd0f5266ab93b86e91d66 Oct 31 00:10:12.354030 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:12.354008 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-54ad8c-5c7f67b585-gn54m"] Oct 31 00:10:12.359417 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:12.359392 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-ab0ac9-7978cbf74c-qxdvb"] Oct 31 00:10:12.366398 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:12.366281 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ecf922c-de0b-4e64-a2a2-07067e279990-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-54ad8c-5c7f67b585-gn54m\" (UID: \"6ecf922c-de0b-4e64-a2a2-07067e279990\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-54ad8c-5c7f67b585-gn54m" Oct 31 00:10:12.366398 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:12.366389 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6ecf922c-de0b-4e64-a2a2-07067e279990-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-54ad8c-5c7f67b585-gn54m\" (UID: \"6ecf922c-de0b-4e64-a2a2-07067e279990\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-54ad8c-5c7f67b585-gn54m" Oct 31 00:10:12.366655 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:12.366416 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wbjzr\" (UniqueName: \"kubernetes.io/projected/6ecf922c-de0b-4e64-a2a2-07067e279990-kube-api-access-wbjzr\") pod \"maas-api-e2e-shared-a-w4-54ad8c-5c7f67b585-gn54m\" (UID: \"6ecf922c-de0b-4e64-a2a2-07067e279990\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-54ad8c-5c7f67b585-gn54m" Oct 31 00:10:12.366655 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:12.366526 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6ecf922c-de0b-4e64-a2a2-07067e279990-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-54ad8c-5c7f67b585-gn54m\" (UID: \"6ecf922c-de0b-4e64-a2a2-07067e279990\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-54ad8c-5c7f67b585-gn54m" Oct 31 00:10:12.372775 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:12.372733 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-ab0ac9-847986794b-bbf7n"] Oct 31 00:10:12.372880 ip-10-0-141-180 kubenswrapper[4927]: W1031 00:10:12.372735 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a2e97c5_6cd5_498d_b83f_13e94fc34b28.slice/crio-c420c20352966c1d9cdb747c723c23924fd480fd79a097ab09b49fc211f5bfed WatchSource:0}: Error finding container c420c20352966c1d9cdb747c723c23924fd480fd79a097ab09b49fc211f5bfed: Status 404 returned error can't find the container with id c420c20352966c1d9cdb747c723c23924fd480fd79a097ab09b49fc211f5bfed Oct 31 00:10:12.445379 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:12.445274 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:10:12.467614 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:12.467589 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6ecf922c-de0b-4e64-a2a2-07067e279990-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-54ad8c-5c7f67b585-gn54m\" (UID: \"6ecf922c-de0b-4e64-a2a2-07067e279990\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-54ad8c-5c7f67b585-gn54m" Oct 31 00:10:12.467755 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:12.467712 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ecf922c-de0b-4e64-a2a2-07067e279990-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-54ad8c-5c7f67b585-gn54m\" (UID: \"6ecf922c-de0b-4e64-a2a2-07067e279990\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-54ad8c-5c7f67b585-gn54m" Oct 31 00:10:12.467817 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:12.467783 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6ecf922c-de0b-4e64-a2a2-07067e279990-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-54ad8c-5c7f67b585-gn54m\" (UID: \"6ecf922c-de0b-4e64-a2a2-07067e279990\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-54ad8c-5c7f67b585-gn54m" Oct 31 00:10:12.467817 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:12.467808 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbjzr\" (UniqueName: \"kubernetes.io/projected/6ecf922c-de0b-4e64-a2a2-07067e279990-kube-api-access-wbjzr\") pod \"maas-api-e2e-shared-a-w4-54ad8c-5c7f67b585-gn54m\" (UID: \"6ecf922c-de0b-4e64-a2a2-07067e279990\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-54ad8c-5c7f67b585-gn54m" Oct 31 00:10:12.468385 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:12.468363 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ecf922c-de0b-4e64-a2a2-07067e279990-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-54ad8c-5c7f67b585-gn54m\" (UID: \"6ecf922c-de0b-4e64-a2a2-07067e279990\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-54ad8c-5c7f67b585-gn54m" Oct 31 00:10:12.468485 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:10:12.468469 4927 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-54ad8c-serving-cert: secret "maas-api-e2e-shared-a-w4-54ad8c-serving-cert" not found Oct 31 00:10:12.468543 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:10:12.468526 4927 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6ecf922c-de0b-4e64-a2a2-07067e279990-maas-api-tls podName:6ecf922c-de0b-4e64-a2a2-07067e279990 nodeName:}" failed. No retries permitted until 2026-10-31 00:10:12.968507072 +0000 UTC m=+1726.004837886 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6ecf922c-de0b-4e64-a2a2-07067e279990-maas-api-tls") pod "maas-api-e2e-shared-a-w4-54ad8c-5c7f67b585-gn54m" (UID: "6ecf922c-de0b-4e64-a2a2-07067e279990") : secret "maas-api-e2e-shared-a-w4-54ad8c-serving-cert" not found Oct 31 00:10:12.470444 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:12.470404 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6ecf922c-de0b-4e64-a2a2-07067e279990-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-54ad8c-5c7f67b585-gn54m\" (UID: \"6ecf922c-de0b-4e64-a2a2-07067e279990\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-54ad8c-5c7f67b585-gn54m" Oct 31 00:10:12.548220 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:12.548150 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wbjzr\" (UniqueName: \"kubernetes.io/projected/6ecf922c-de0b-4e64-a2a2-07067e279990-kube-api-access-wbjzr\") pod \"maas-api-e2e-shared-a-w4-54ad8c-5c7f67b585-gn54m\" (UID: \"6ecf922c-de0b-4e64-a2a2-07067e279990\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-54ad8c-5c7f67b585-gn54m" Oct 31 00:10:12.602975 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:12.602033 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-71b76a-openshift-default-77db967c97-lh4ld" Oct 31 00:10:12.603382 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:12.603299 4927 patch_prober.go:29] interesting pod/e2e-api-a-71b76a-openshift-default-77db967c97-lh4ld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" start-of-body= Oct 31 00:10:12.603617 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:12.603547 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-71b76a-openshift-default-77db967c97-lh4ld" podUID="c51c586a-f2d8-48f9-8813-3b6e4f360e5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" Oct 31 00:10:12.990864 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:12.990831 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6ecf922c-de0b-4e64-a2a2-07067e279990-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-54ad8c-5c7f67b585-gn54m\" (UID: \"6ecf922c-de0b-4e64-a2a2-07067e279990\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-54ad8c-5c7f67b585-gn54m" Oct 31 00:10:12.993501 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:12.993480 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6ecf922c-de0b-4e64-a2a2-07067e279990-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-54ad8c-5c7f67b585-gn54m\" (UID: \"6ecf922c-de0b-4e64-a2a2-07067e279990\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-54ad8c-5c7f67b585-gn54m" Oct 31 00:10:13.059146 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:13.059105 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-ab0ac9-847986794b-bbf7n" event={"ID":"5a2e97c5-6cd5-498d-b83f-13e94fc34b28","Type":"ContainerStarted","Data":"c6b04a8bcb3c58da0ce7fafb8e2b818a26b6eecea7ee531ea7a28a64310ac02b"} Oct 31 00:10:13.059366 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:13.059172 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-ab0ac9-847986794b-bbf7n" event={"ID":"5a2e97c5-6cd5-498d-b83f-13e94fc34b28","Type":"ContainerStarted","Data":"c420c20352966c1d9cdb747c723c23924fd480fd79a097ab09b49fc211f5bfed"} Oct 31 00:10:13.059366 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:13.059220 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-ab0ac9-847986794b-bbf7n" Oct 31 00:10:13.061107 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:13.061079 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-ab0ac9-7978cbf74c-qxdvb" event={"ID":"6874430f-3348-469c-adc3-a605fa1ff16a","Type":"ContainerStarted","Data":"6c47ebef07bf10252586dc9a818cf86f00d375138f965e29cbf91a4a60748630"} Oct 31 00:10:13.061244 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:13.061122 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-ab0ac9-7978cbf74c-qxdvb" event={"ID":"6874430f-3348-469c-adc3-a605fa1ff16a","Type":"ContainerStarted","Data":"e820c91c27c945ee5a30e9759980e4561a049d39b78dd0f5266ab93b86e91d66"} Oct 31 00:10:13.061295 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:13.061282 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-ab0ac9-7978cbf74c-qxdvb" Oct 31 00:10:13.181099 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:13.181065 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-54ad8c-5c7f67b585-gn54m" Oct 31 00:10:13.202770 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:13.202719 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-ab0ac9-847986794b-bbf7n" podStartSLOduration=2.202705756 podStartE2EDuration="2.202705756s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:10:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:10:13.196084866 +0000 UTC m=+1726.232415689" watchObservedRunningTime="2026-10-31 00:10:13.202705756 +0000 UTC m=+1726.239036577" Oct 31 00:10:13.411561 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:13.411493 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-ab0ac9-7978cbf74c-qxdvb" podStartSLOduration=2.411473078 podStartE2EDuration="2.411473078s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:10:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:10:13.369389342 +0000 UTC m=+1726.405720199" watchObservedRunningTime="2026-10-31 00:10:13.411473078 +0000 UTC m=+1726.447803901" Oct 31 00:10:13.627678 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:13.627584 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-71b76a-openshift-default-77db967c97-lh4ld" Oct 31 00:10:13.628030 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:13.627999 4927 patch_prober.go:29] interesting pod/e2e-api-a-71b76a-openshift-default-77db967c97-lh4ld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" start-of-body= Oct 31 00:10:13.659237 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:13.659199 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-71b76a-openshift-default-77db967c97-lh4ld" podUID="c51c586a-f2d8-48f9-8813-3b6e4f360e5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" Oct 31 00:10:13.659908 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:13.658660 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:10:13.839553 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:13.831313 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-54ad8c-5c7f67b585-gn54m"] Oct 31 00:10:14.070354 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:14.070309 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-54ad8c-5c7f67b585-gn54m" event={"ID":"6ecf922c-de0b-4e64-a2a2-07067e279990","Type":"ContainerStarted","Data":"527ff3aafa8bec41550e55d55c9c1e85df9cb10d3096f337ca27f62b55202c00"} Oct 31 00:10:14.596355 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:14.596296 4927 patch_prober.go:29] interesting pod/e2e-api-a-71b76a-openshift-default-77db967c97-lh4ld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" start-of-body= Oct 31 00:10:14.596531 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:14.596380 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-71b76a-openshift-default-77db967c97-lh4ld" podUID="c51c586a-f2d8-48f9-8813-3b6e4f360e5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" Oct 31 00:10:15.088073 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:15.088043 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-54ad8c-5c7f67b585-gn54m" Oct 31 00:10:15.088489 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:15.088097 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-54ad8c-5c7f67b585-gn54m" event={"ID":"6ecf922c-de0b-4e64-a2a2-07067e279990","Type":"ContainerStarted","Data":"f95528cfc6a6f0a4ca51e4c57ab61b569a21076b6a102f43c39cd0f8de0c23ef"} Oct 31 00:10:15.596544 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:15.596508 4927 patch_prober.go:29] interesting pod/e2e-api-a-71b76a-openshift-default-77db967c97-lh4ld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" start-of-body= Oct 31 00:10:15.596721 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:15.596564 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-71b76a-openshift-default-77db967c97-lh4ld" podUID="c51c586a-f2d8-48f9-8813-3b6e4f360e5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" Oct 31 00:10:16.596316 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:16.596279 4927 patch_prober.go:29] interesting pod/e2e-api-a-71b76a-openshift-default-77db967c97-lh4ld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" start-of-body= Oct 31 00:10:16.596769 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:16.596365 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-71b76a-openshift-default-77db967c97-lh4ld" podUID="c51c586a-f2d8-48f9-8813-3b6e4f360e5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" Oct 31 00:10:17.187139 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:17.187104 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:10:17.596301 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:17.596212 4927 patch_prober.go:29] interesting pod/e2e-api-a-71b76a-openshift-default-77db967c97-lh4ld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" start-of-body= Oct 31 00:10:17.596301 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:17.596266 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-71b76a-openshift-default-77db967c97-lh4ld" podUID="c51c586a-f2d8-48f9-8813-3b6e4f360e5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" Oct 31 00:10:18.280067 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:18.280035 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:10:18.600975 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:18.600900 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-71b76a-openshift-default-77db967c97-lh4ld" Oct 31 00:10:18.601953 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:18.601934 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-71b76a-openshift-default-77db967c97-lh4ld" Oct 31 00:10:18.775154 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:18.775103 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-54ad8c-5c7f67b585-gn54m" podStartSLOduration=5.87068205 podStartE2EDuration="6.775087137s" totalImagesPullingTime="904.405087ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:10:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:10:15.241739855 +0000 UTC m=+1728.278070698" watchObservedRunningTime="2026-10-31 00:10:18.775087137 +0000 UTC m=+1731.811418055" Oct 31 00:10:19.350210 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:19.350164 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8bt76"] Oct 31 00:10:19.350456 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:19.350430 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8bt76" podUID="8789e67e-5656-4479-9b4f-fdc746fcfd15" containerName="istio-proxy" containerID="cri-o://648dcd6d004af2c4231082241023009a70d09509111860829d6171775e879774" gracePeriod=30 Oct 31 00:10:19.355142 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:19.355118 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-t762r"] Oct 31 00:10:19.355303 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:19.355283 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-t762r" podUID="b9e9d9b1-22af-48d6-b1de-b1c28107ace1" containerName="istio-proxy" containerID="cri-o://0548d73c9bd7e0ba046a631af25ddd93a5a5321d0e09857824458683c21f6a8e" gracePeriod=30 Oct 31 00:10:19.400977 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:19.400946 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-xkbnw"] Oct 31 00:10:19.401172 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:19.401148 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-xkbnw" podUID="469ac12b-4c77-403a-a190-af4037422211" containerName="istio-proxy" containerID="cri-o://4177f70e567ff945d5d30decef1ae28787d68a5859d631971494bd201d921f06" gracePeriod=30 Oct 31 00:10:19.680538 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:10:19.680500 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sqv69" containerName="extractor" Oct 31 00:10:21.093894 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:21.093868 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-54ad8c-5c7f67b585-gn54m" Oct 31 00:10:21.754368 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:21.754286 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:10:22.162999 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:22.162966 4927 patch_prober.go:29] interesting pod/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-t762r container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 00:10:22.163400 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:22.163028 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-t762r" podUID="b9e9d9b1-22af-48d6-b1de-b1c28107ace1" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:10:24.074029 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:24.074001 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-ab0ac9-7978cbf74c-qxdvb" Oct 31 00:10:24.074452 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:24.074066 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-ab0ac9-847986794b-bbf7n" Oct 31 00:10:24.719022 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:24.718986 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-t762r" Oct 31 00:10:24.765977 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:24.765953 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8bt76" Oct 31 00:10:24.769533 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:24.769513 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-xkbnw" Oct 31 00:10:24.777867 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:24.777838 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b9e9d9b1-22af-48d6-b1de-b1c28107ace1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9e9d9b1-22af-48d6-b1de-b1c28107ace1-istio-podinfo\") pod \"b9e9d9b1-22af-48d6-b1de-b1c28107ace1\" (UID: \"b9e9d9b1-22af-48d6-b1de-b1c28107ace1\") " Oct 31 00:10:24.777980 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:24.777921 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9e9d9b1-22af-48d6-b1de-b1c28107ace1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9e9d9b1-22af-48d6-b1de-b1c28107ace1-credential-socket\") pod \"b9e9d9b1-22af-48d6-b1de-b1c28107ace1\" (UID: \"b9e9d9b1-22af-48d6-b1de-b1c28107ace1\") " Oct 31 00:10:24.777980 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:24.777957 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b9e9d9b1-22af-48d6-b1de-b1c28107ace1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9e9d9b1-22af-48d6-b1de-b1c28107ace1-istiod-ca-cert\") pod \"b9e9d9b1-22af-48d6-b1de-b1c28107ace1\" (UID: \"b9e9d9b1-22af-48d6-b1de-b1c28107ace1\") " Oct 31 00:10:24.778071 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:24.778056 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9e9d9b1-22af-48d6-b1de-b1c28107ace1-kube-api-access-8r7wf\" (UniqueName: \"kubernetes.io/projected/b9e9d9b1-22af-48d6-b1de-b1c28107ace1-kube-api-access-8r7wf\") pod \"b9e9d9b1-22af-48d6-b1de-b1c28107ace1\" (UID: \"b9e9d9b1-22af-48d6-b1de-b1c28107ace1\") " Oct 31 00:10:24.778137 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:24.778122 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9e9d9b1-22af-48d6-b1de-b1c28107ace1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9e9d9b1-22af-48d6-b1de-b1c28107ace1-workload-certs\") pod \"b9e9d9b1-22af-48d6-b1de-b1c28107ace1\" (UID: \"b9e9d9b1-22af-48d6-b1de-b1c28107ace1\") " Oct 31 00:10:24.778214 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:24.778182 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9e9d9b1-22af-48d6-b1de-b1c28107ace1-credential-socket" pod "b9e9d9b1-22af-48d6-b1de-b1c28107ace1" (UID: "b9e9d9b1-22af-48d6-b1de-b1c28107ace1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:10:24.778440 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:24.778283 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9e9d9b1-22af-48d6-b1de-b1c28107ace1-istiod-ca-cert" pod "b9e9d9b1-22af-48d6-b1de-b1c28107ace1" (UID: "b9e9d9b1-22af-48d6-b1de-b1c28107ace1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:10:24.778440 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:24.778314 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9e9d9b1-22af-48d6-b1de-b1c28107ace1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9e9d9b1-22af-48d6-b1de-b1c28107ace1-workload-socket\") pod \"b9e9d9b1-22af-48d6-b1de-b1c28107ace1\" (UID: \"b9e9d9b1-22af-48d6-b1de-b1c28107ace1\") " Oct 31 00:10:24.778572 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:24.778457 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9e9d9b1-22af-48d6-b1de-b1c28107ace1-workload-certs" pod "b9e9d9b1-22af-48d6-b1de-b1c28107ace1" (UID: "b9e9d9b1-22af-48d6-b1de-b1c28107ace1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:10:24.778572 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:24.778493 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9e9d9b1-22af-48d6-b1de-b1c28107ace1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9e9d9b1-22af-48d6-b1de-b1c28107ace1-istio-data\") pod \"b9e9d9b1-22af-48d6-b1de-b1c28107ace1\" (UID: \"b9e9d9b1-22af-48d6-b1de-b1c28107ace1\") " Oct 31 00:10:24.778572 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:24.778509 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9e9d9b1-22af-48d6-b1de-b1c28107ace1-workload-socket" pod "b9e9d9b1-22af-48d6-b1de-b1c28107ace1" (UID: "b9e9d9b1-22af-48d6-b1de-b1c28107ace1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:10:24.778572 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:24.778524 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9e9d9b1-22af-48d6-b1de-b1c28107ace1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9e9d9b1-22af-48d6-b1de-b1c28107ace1-istio-envoy\") pod \"b9e9d9b1-22af-48d6-b1de-b1c28107ace1\" (UID: \"b9e9d9b1-22af-48d6-b1de-b1c28107ace1\") " Oct 31 00:10:24.778572 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:24.778570 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9e9d9b1-22af-48d6-b1de-b1c28107ace1-istio-token\" (UniqueName: \"kubernetes.io/projected/b9e9d9b1-22af-48d6-b1de-b1c28107ace1-istio-token\") pod \"b9e9d9b1-22af-48d6-b1de-b1c28107ace1\" (UID: \"b9e9d9b1-22af-48d6-b1de-b1c28107ace1\") " Oct 31 00:10:24.778886 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:24.778659 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9e9d9b1-22af-48d6-b1de-b1c28107ace1-istio-data" pod "b9e9d9b1-22af-48d6-b1de-b1c28107ace1" (UID: "b9e9d9b1-22af-48d6-b1de-b1c28107ace1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:10:24.779168 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:24.779145 4927 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9e9d9b1-22af-48d6-b1de-b1c28107ace1-workload-socket\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:10:24.779219 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:24.779176 4927 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9e9d9b1-22af-48d6-b1de-b1c28107ace1-istio-data\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:10:24.779219 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:24.779193 4927 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9e9d9b1-22af-48d6-b1de-b1c28107ace1-credential-socket\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:10:24.779219 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:24.779207 4927 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9e9d9b1-22af-48d6-b1de-b1c28107ace1-istiod-ca-cert\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:10:24.779316 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:24.779221 4927 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9e9d9b1-22af-48d6-b1de-b1c28107ace1-workload-certs\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:10:24.780542 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:24.780522 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9e9d9b1-22af-48d6-b1de-b1c28107ace1-istio-envoy" pod "b9e9d9b1-22af-48d6-b1de-b1c28107ace1" (UID: "b9e9d9b1-22af-48d6-b1de-b1c28107ace1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:10:24.780542 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:24.780530 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9e9d9b1-22af-48d6-b1de-b1c28107ace1-istio-token" pod "b9e9d9b1-22af-48d6-b1de-b1c28107ace1" (UID: "b9e9d9b1-22af-48d6-b1de-b1c28107ace1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:10:24.780678 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:24.780663 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9e9d9b1-22af-48d6-b1de-b1c28107ace1-kube-api-access-8r7wf" pod "b9e9d9b1-22af-48d6-b1de-b1c28107ace1" (UID: "b9e9d9b1-22af-48d6-b1de-b1c28107ace1"). InnerVolumeSpecName "kube-api-access-8r7wf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:10:24.780726 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:24.780674 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b9e9d9b1-22af-48d6-b1de-b1c28107ace1-istio-podinfo" pod "b9e9d9b1-22af-48d6-b1de-b1c28107ace1" (UID: "b9e9d9b1-22af-48d6-b1de-b1c28107ace1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:10:24.880001 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:24.879918 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8789e67e-5656-4479-9b4f-fdc746fcfd15-kube-api-access-92jrp\" (UniqueName: \"kubernetes.io/projected/8789e67e-5656-4479-9b4f-fdc746fcfd15-kube-api-access-92jrp\") pod \"8789e67e-5656-4479-9b4f-fdc746fcfd15\" (UID: \"8789e67e-5656-4479-9b4f-fdc746fcfd15\") " Oct 31 00:10:24.880001 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:24.879969 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/469ac12b-4c77-403a-a190-af4037422211-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/469ac12b-4c77-403a-a190-af4037422211-istio-envoy\") pod \"469ac12b-4c77-403a-a190-af4037422211\" (UID: \"469ac12b-4c77-403a-a190-af4037422211\") " Oct 31 00:10:24.880245 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:24.880005 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/469ac12b-4c77-403a-a190-af4037422211-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/469ac12b-4c77-403a-a190-af4037422211-istiod-ca-cert\") pod \"469ac12b-4c77-403a-a190-af4037422211\" (UID: \"469ac12b-4c77-403a-a190-af4037422211\") " Oct 31 00:10:24.880500 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:24.880432 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/469ac12b-4c77-403a-a190-af4037422211-istiod-ca-cert" pod "469ac12b-4c77-403a-a190-af4037422211" (UID: "469ac12b-4c77-403a-a190-af4037422211"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:10:24.880500 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:24.880486 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8789e67e-5656-4479-9b4f-fdc746fcfd15-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8789e67e-5656-4479-9b4f-fdc746fcfd15-istiod-ca-cert\") pod \"8789e67e-5656-4479-9b4f-fdc746fcfd15\" (UID: \"8789e67e-5656-4479-9b4f-fdc746fcfd15\") " Oct 31 00:10:24.880663 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:24.880571 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8789e67e-5656-4479-9b4f-fdc746fcfd15-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8789e67e-5656-4479-9b4f-fdc746fcfd15-istio-podinfo\") pod \"8789e67e-5656-4479-9b4f-fdc746fcfd15\" (UID: \"8789e67e-5656-4479-9b4f-fdc746fcfd15\") " Oct 31 00:10:24.880724 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:24.880686 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/469ac12b-4c77-403a-a190-af4037422211-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/469ac12b-4c77-403a-a190-af4037422211-workload-socket\") pod \"469ac12b-4c77-403a-a190-af4037422211\" (UID: \"469ac12b-4c77-403a-a190-af4037422211\") " Oct 31 00:10:24.880785 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:24.880756 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/469ac12b-4c77-403a-a190-af4037422211-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/469ac12b-4c77-403a-a190-af4037422211-credential-socket\") pod \"469ac12b-4c77-403a-a190-af4037422211\" (UID: \"469ac12b-4c77-403a-a190-af4037422211\") " Oct 31 00:10:24.880846 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:24.880789 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8789e67e-5656-4479-9b4f-fdc746fcfd15-istio-token\" (UniqueName: \"kubernetes.io/projected/8789e67e-5656-4479-9b4f-fdc746fcfd15-istio-token\") pod \"8789e67e-5656-4479-9b4f-fdc746fcfd15\" (UID: \"8789e67e-5656-4479-9b4f-fdc746fcfd15\") " Oct 31 00:10:24.880846 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:24.880834 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8789e67e-5656-4479-9b4f-fdc746fcfd15-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8789e67e-5656-4479-9b4f-fdc746fcfd15-workload-certs\") pod \"8789e67e-5656-4479-9b4f-fdc746fcfd15\" (UID: \"8789e67e-5656-4479-9b4f-fdc746fcfd15\") " Oct 31 00:10:24.880951 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:24.880869 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8789e67e-5656-4479-9b4f-fdc746fcfd15-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8789e67e-5656-4479-9b4f-fdc746fcfd15-istio-envoy\") pod \"8789e67e-5656-4479-9b4f-fdc746fcfd15\" (UID: \"8789e67e-5656-4479-9b4f-fdc746fcfd15\") " Oct 31 00:10:24.880951 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:24.880900 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/469ac12b-4c77-403a-a190-af4037422211-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/469ac12b-4c77-403a-a190-af4037422211-istio-data\") pod \"469ac12b-4c77-403a-a190-af4037422211\" (UID: \"469ac12b-4c77-403a-a190-af4037422211\") " Oct 31 00:10:24.880951 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:24.880946 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/469ac12b-4c77-403a-a190-af4037422211-istio-token\" (UniqueName: \"kubernetes.io/projected/469ac12b-4c77-403a-a190-af4037422211-istio-token\") pod \"469ac12b-4c77-403a-a190-af4037422211\" (UID: \"469ac12b-4c77-403a-a190-af4037422211\") " Oct 31 00:10:24.881105 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:24.880987 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/469ac12b-4c77-403a-a190-af4037422211-kube-api-access-rx6l7\" (UniqueName: \"kubernetes.io/projected/469ac12b-4c77-403a-a190-af4037422211-kube-api-access-rx6l7\") pod \"469ac12b-4c77-403a-a190-af4037422211\" (UID: \"469ac12b-4c77-403a-a190-af4037422211\") " Oct 31 00:10:24.881105 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:24.881046 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/469ac12b-4c77-403a-a190-af4037422211-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/469ac12b-4c77-403a-a190-af4037422211-workload-certs\") pod \"469ac12b-4c77-403a-a190-af4037422211\" (UID: \"469ac12b-4c77-403a-a190-af4037422211\") " Oct 31 00:10:24.881105 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:24.881093 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8789e67e-5656-4479-9b4f-fdc746fcfd15-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8789e67e-5656-4479-9b4f-fdc746fcfd15-workload-socket\") pod \"8789e67e-5656-4479-9b4f-fdc746fcfd15\" (UID: \"8789e67e-5656-4479-9b4f-fdc746fcfd15\") " Oct 31 00:10:24.881277 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:24.881127 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8789e67e-5656-4479-9b4f-fdc746fcfd15-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8789e67e-5656-4479-9b4f-fdc746fcfd15-credential-socket\") pod \"8789e67e-5656-4479-9b4f-fdc746fcfd15\" (UID: \"8789e67e-5656-4479-9b4f-fdc746fcfd15\") " Oct 31 00:10:24.881277 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:24.881207 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/469ac12b-4c77-403a-a190-af4037422211-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/469ac12b-4c77-403a-a190-af4037422211-istio-podinfo\") pod \"469ac12b-4c77-403a-a190-af4037422211\" (UID: \"469ac12b-4c77-403a-a190-af4037422211\") " Oct 31 00:10:24.881277 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:24.881244 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8789e67e-5656-4479-9b4f-fdc746fcfd15-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8789e67e-5656-4479-9b4f-fdc746fcfd15-istio-data\") pod \"8789e67e-5656-4479-9b4f-fdc746fcfd15\" (UID: \"8789e67e-5656-4479-9b4f-fdc746fcfd15\") " Oct 31 00:10:24.881838 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:24.881815 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8r7wf\" (UniqueName: \"kubernetes.io/projected/b9e9d9b1-22af-48d6-b1de-b1c28107ace1-kube-api-access-8r7wf\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:10:24.881925 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:24.881843 4927 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9e9d9b1-22af-48d6-b1de-b1c28107ace1-istio-envoy\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:10:24.881925 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:24.881858 4927 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9e9d9b1-22af-48d6-b1de-b1c28107ace1-istio-token\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:10:24.881925 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:24.881913 4927 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/469ac12b-4c77-403a-a190-af4037422211-istiod-ca-cert\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:10:24.882087 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:24.881930 4927 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9e9d9b1-22af-48d6-b1de-b1c28107ace1-istio-podinfo\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:10:24.882501 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:24.882470 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8789e67e-5656-4479-9b4f-fdc746fcfd15-istio-data" pod "8789e67e-5656-4479-9b4f-fdc746fcfd15" (UID: "8789e67e-5656-4479-9b4f-fdc746fcfd15"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:10:24.882746 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:24.882728 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8789e67e-5656-4479-9b4f-fdc746fcfd15-istiod-ca-cert" pod "8789e67e-5656-4479-9b4f-fdc746fcfd15" (UID: "8789e67e-5656-4479-9b4f-fdc746fcfd15"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:10:24.883121 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:24.883087 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/469ac12b-4c77-403a-a190-af4037422211-istio-data" pod "469ac12b-4c77-403a-a190-af4037422211" (UID: "469ac12b-4c77-403a-a190-af4037422211"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:10:24.883271 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:24.883254 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/469ac12b-4c77-403a-a190-af4037422211-workload-socket" pod "469ac12b-4c77-403a-a190-af4037422211" (UID: "469ac12b-4c77-403a-a190-af4037422211"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:10:24.883452 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:24.883434 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/469ac12b-4c77-403a-a190-af4037422211-credential-socket" pod "469ac12b-4c77-403a-a190-af4037422211" (UID: "469ac12b-4c77-403a-a190-af4037422211"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:10:24.885215 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:24.884978 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8789e67e-5656-4479-9b4f-fdc746fcfd15-istio-podinfo" pod "8789e67e-5656-4479-9b4f-fdc746fcfd15" (UID: "8789e67e-5656-4479-9b4f-fdc746fcfd15"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:10:24.885215 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:24.885118 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8789e67e-5656-4479-9b4f-fdc746fcfd15-workload-certs" pod "8789e67e-5656-4479-9b4f-fdc746fcfd15" (UID: "8789e67e-5656-4479-9b4f-fdc746fcfd15"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:10:24.885399 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:24.885250 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/469ac12b-4c77-403a-a190-af4037422211-workload-certs" pod "469ac12b-4c77-403a-a190-af4037422211" (UID: "469ac12b-4c77-403a-a190-af4037422211"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:10:24.885399 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:24.885268 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8789e67e-5656-4479-9b4f-fdc746fcfd15-workload-socket" pod "8789e67e-5656-4479-9b4f-fdc746fcfd15" (UID: "8789e67e-5656-4479-9b4f-fdc746fcfd15"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:10:24.885820 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:24.885767 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8789e67e-5656-4479-9b4f-fdc746fcfd15-credential-socket" pod "8789e67e-5656-4479-9b4f-fdc746fcfd15" (UID: "8789e67e-5656-4479-9b4f-fdc746fcfd15"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:10:24.886037 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:24.886009 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8789e67e-5656-4479-9b4f-fdc746fcfd15-istio-token" pod "8789e67e-5656-4479-9b4f-fdc746fcfd15" (UID: "8789e67e-5656-4479-9b4f-fdc746fcfd15"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:10:24.887720 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:24.887669 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/469ac12b-4c77-403a-a190-af4037422211-istio-podinfo" pod "469ac12b-4c77-403a-a190-af4037422211" (UID: "469ac12b-4c77-403a-a190-af4037422211"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:10:24.887720 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:24.887703 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8789e67e-5656-4479-9b4f-fdc746fcfd15-istio-envoy" pod "8789e67e-5656-4479-9b4f-fdc746fcfd15" (UID: "8789e67e-5656-4479-9b4f-fdc746fcfd15"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:10:24.887880 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:24.887736 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/469ac12b-4c77-403a-a190-af4037422211-istio-token" pod "469ac12b-4c77-403a-a190-af4037422211" (UID: "469ac12b-4c77-403a-a190-af4037422211"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:10:24.887880 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:24.887797 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8789e67e-5656-4479-9b4f-fdc746fcfd15-kube-api-access-92jrp" pod "8789e67e-5656-4479-9b4f-fdc746fcfd15" (UID: "8789e67e-5656-4479-9b4f-fdc746fcfd15"). InnerVolumeSpecName "kube-api-access-92jrp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:10:24.888567 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:24.888545 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/469ac12b-4c77-403a-a190-af4037422211-istio-envoy" pod "469ac12b-4c77-403a-a190-af4037422211" (UID: "469ac12b-4c77-403a-a190-af4037422211"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:10:24.889358 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:24.889311 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/469ac12b-4c77-403a-a190-af4037422211-kube-api-access-rx6l7" pod "469ac12b-4c77-403a-a190-af4037422211" (UID: "469ac12b-4c77-403a-a190-af4037422211"). InnerVolumeSpecName "kube-api-access-rx6l7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:10:24.983379 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:24.983120 4927 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/469ac12b-4c77-403a-a190-af4037422211-workload-socket\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:10:24.983379 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:24.983156 4927 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/469ac12b-4c77-403a-a190-af4037422211-credential-socket\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:10:24.983379 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:24.983179 4927 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8789e67e-5656-4479-9b4f-fdc746fcfd15-istio-token\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:10:24.983379 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:24.983192 4927 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8789e67e-5656-4479-9b4f-fdc746fcfd15-workload-certs\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:10:24.983379 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:24.983209 4927 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8789e67e-5656-4479-9b4f-fdc746fcfd15-istio-envoy\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:10:24.983379 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:24.983224 4927 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/469ac12b-4c77-403a-a190-af4037422211-istio-data\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:10:24.983379 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:24.983238 4927 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/469ac12b-4c77-403a-a190-af4037422211-istio-token\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:10:24.983379 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:24.983252 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rx6l7\" (UniqueName: \"kubernetes.io/projected/469ac12b-4c77-403a-a190-af4037422211-kube-api-access-rx6l7\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:10:24.983379 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:24.983268 4927 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/469ac12b-4c77-403a-a190-af4037422211-workload-certs\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:10:24.983379 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:24.983282 4927 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8789e67e-5656-4479-9b4f-fdc746fcfd15-workload-socket\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:10:24.983379 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:24.983294 4927 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8789e67e-5656-4479-9b4f-fdc746fcfd15-credential-socket\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:10:24.983379 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:24.983307 4927 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/469ac12b-4c77-403a-a190-af4037422211-istio-podinfo\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:10:24.983379 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:24.983319 4927 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8789e67e-5656-4479-9b4f-fdc746fcfd15-istio-data\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:10:24.983379 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:24.983349 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-92jrp\" (UniqueName: \"kubernetes.io/projected/8789e67e-5656-4479-9b4f-fdc746fcfd15-kube-api-access-92jrp\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:10:24.983379 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:24.983364 4927 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/469ac12b-4c77-403a-a190-af4037422211-istio-envoy\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:10:24.983379 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:24.983377 4927 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8789e67e-5656-4479-9b4f-fdc746fcfd15-istiod-ca-cert\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:10:24.983379 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:24.983391 4927 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8789e67e-5656-4479-9b4f-fdc746fcfd15-istio-podinfo\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:10:25.174554 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:25.174518 4927 generic.go:356] "Generic (PLEG): container finished" podID="469ac12b-4c77-403a-a190-af4037422211" containerID="4177f70e567ff945d5d30decef1ae28787d68a5859d631971494bd201d921f06" exitCode=0 Oct 31 00:10:25.175022 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:25.174573 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-xkbnw" Oct 31 00:10:25.175022 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:25.174593 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-xkbnw" event={"ID":"469ac12b-4c77-403a-a190-af4037422211","Type":"ContainerDied","Data":"4177f70e567ff945d5d30decef1ae28787d68a5859d631971494bd201d921f06"} Oct 31 00:10:25.175022 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:25.174646 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-xkbnw" event={"ID":"469ac12b-4c77-403a-a190-af4037422211","Type":"ContainerDied","Data":"77d2a04204ae0f8f966370e9bc266cbee6d66c42043bb78e13b701d4c499b7fb"} Oct 31 00:10:25.175022 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:25.174669 4927 scope.go:122] "RemoveContainer" containerID="4177f70e567ff945d5d30decef1ae28787d68a5859d631971494bd201d921f06" Oct 31 00:10:25.176445 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:25.176418 4927 generic.go:356] "Generic (PLEG): container finished" podID="8789e67e-5656-4479-9b4f-fdc746fcfd15" containerID="648dcd6d004af2c4231082241023009a70d09509111860829d6171775e879774" exitCode=0 Oct 31 00:10:25.176571 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:25.176461 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8bt76" Oct 31 00:10:25.176571 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:25.176506 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8bt76" event={"ID":"8789e67e-5656-4479-9b4f-fdc746fcfd15","Type":"ContainerDied","Data":"648dcd6d004af2c4231082241023009a70d09509111860829d6171775e879774"} Oct 31 00:10:25.176571 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:25.176559 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8bt76" event={"ID":"8789e67e-5656-4479-9b4f-fdc746fcfd15","Type":"ContainerDied","Data":"e0b61239a82c4d05a730737da3bd3c89c9f9c1d5b1f08da54ca7ff370ca87687"} Oct 31 00:10:25.178400 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:25.178376 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-t762r" event={"ID":"b9e9d9b1-22af-48d6-b1de-b1c28107ace1","Type":"ContainerDied","Data":"0548d73c9bd7e0ba046a631af25ddd93a5a5321d0e09857824458683c21f6a8e"} Oct 31 00:10:25.178400 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:25.178394 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-t762r" Oct 31 00:10:25.178574 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:25.178322 4927 generic.go:356] "Generic (PLEG): container finished" podID="b9e9d9b1-22af-48d6-b1de-b1c28107ace1" containerID="0548d73c9bd7e0ba046a631af25ddd93a5a5321d0e09857824458683c21f6a8e" exitCode=0 Oct 31 00:10:25.178574 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:25.178421 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-t762r" event={"ID":"b9e9d9b1-22af-48d6-b1de-b1c28107ace1","Type":"ContainerDied","Data":"02a281219231a3ac0f94f7781dced895007d23ab7c69f7a337780855815f5605"} Oct 31 00:10:25.188042 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:25.188023 4927 scope.go:122] "RemoveContainer" containerID="4177f70e567ff945d5d30decef1ae28787d68a5859d631971494bd201d921f06" Oct 31 00:10:25.188343 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:10:25.188311 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4177f70e567ff945d5d30decef1ae28787d68a5859d631971494bd201d921f06\": container with ID starting with 4177f70e567ff945d5d30decef1ae28787d68a5859d631971494bd201d921f06 not found: ID does not exist" containerID="4177f70e567ff945d5d30decef1ae28787d68a5859d631971494bd201d921f06" Oct 31 00:10:25.188400 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:25.188349 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4177f70e567ff945d5d30decef1ae28787d68a5859d631971494bd201d921f06"} err="failed to get container status \"4177f70e567ff945d5d30decef1ae28787d68a5859d631971494bd201d921f06\": rpc error: code = NotFound desc = could not find container \"4177f70e567ff945d5d30decef1ae28787d68a5859d631971494bd201d921f06\": container with ID starting with 4177f70e567ff945d5d30decef1ae28787d68a5859d631971494bd201d921f06 not found: ID does not exist" Oct 31 00:10:25.188400 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:25.188367 4927 scope.go:122] "RemoveContainer" containerID="648dcd6d004af2c4231082241023009a70d09509111860829d6171775e879774" Oct 31 00:10:25.204505 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:25.204443 4927 scope.go:122] "RemoveContainer" containerID="648dcd6d004af2c4231082241023009a70d09509111860829d6171775e879774" Oct 31 00:10:25.205207 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:10:25.205156 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"648dcd6d004af2c4231082241023009a70d09509111860829d6171775e879774\": container with ID starting with 648dcd6d004af2c4231082241023009a70d09509111860829d6171775e879774 not found: ID does not exist" containerID="648dcd6d004af2c4231082241023009a70d09509111860829d6171775e879774" Oct 31 00:10:25.205207 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:25.205193 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"648dcd6d004af2c4231082241023009a70d09509111860829d6171775e879774"} err="failed to get container status \"648dcd6d004af2c4231082241023009a70d09509111860829d6171775e879774\": rpc error: code = NotFound desc = could not find container \"648dcd6d004af2c4231082241023009a70d09509111860829d6171775e879774\": container with ID starting with 648dcd6d004af2c4231082241023009a70d09509111860829d6171775e879774 not found: ID does not exist" Oct 31 00:10:25.205436 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:25.205218 4927 scope.go:122] "RemoveContainer" containerID="0548d73c9bd7e0ba046a631af25ddd93a5a5321d0e09857824458683c21f6a8e" Oct 31 00:10:25.224424 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:25.224404 4927 scope.go:122] "RemoveContainer" containerID="0548d73c9bd7e0ba046a631af25ddd93a5a5321d0e09857824458683c21f6a8e" Oct 31 00:10:25.224779 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:10:25.224751 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0548d73c9bd7e0ba046a631af25ddd93a5a5321d0e09857824458683c21f6a8e\": container with ID starting with 0548d73c9bd7e0ba046a631af25ddd93a5a5321d0e09857824458683c21f6a8e not found: ID does not exist" containerID="0548d73c9bd7e0ba046a631af25ddd93a5a5321d0e09857824458683c21f6a8e" Oct 31 00:10:25.224865 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:25.224790 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0548d73c9bd7e0ba046a631af25ddd93a5a5321d0e09857824458683c21f6a8e"} err="failed to get container status \"0548d73c9bd7e0ba046a631af25ddd93a5a5321d0e09857824458683c21f6a8e\": rpc error: code = NotFound desc = could not find container \"0548d73c9bd7e0ba046a631af25ddd93a5a5321d0e09857824458683c21f6a8e\": container with ID starting with 0548d73c9bd7e0ba046a631af25ddd93a5a5321d0e09857824458683c21f6a8e not found: ID does not exist" Oct 31 00:10:25.345420 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:25.345386 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-xkbnw"] Oct 31 00:10:25.350388 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:25.350359 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-xkbnw"] Oct 31 00:10:25.483517 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:25.483429 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8bt76"] Oct 31 00:10:25.493238 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:25.493208 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8bt76"] Oct 31 00:10:25.647649 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:25.647614 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-t762r"] Oct 31 00:10:25.655676 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:25.655638 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-t762r"] Oct 31 00:10:26.376550 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:26.376514 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:10:27.091925 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:27.091893 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="469ac12b-4c77-403a-a190-af4037422211" path="/var/lib/kubelet/pods/469ac12b-4c77-403a-a190-af4037422211/volumes" Oct 31 00:10:27.092443 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:27.092428 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8789e67e-5656-4479-9b4f-fdc746fcfd15" path="/var/lib/kubelet/pods/8789e67e-5656-4479-9b4f-fdc746fcfd15/volumes" Oct 31 00:10:27.092929 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:27.092908 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b9e9d9b1-22af-48d6-b1de-b1c28107ace1" path="/var/lib/kubelet/pods/b9e9d9b1-22af-48d6-b1de-b1c28107ace1/volumes" Oct 31 00:10:27.575205 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:27.575168 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:10:29.297483 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:29.296465 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-46b586/simulator-w0-46b586-kserve-67588b8dbd-z8xmn" podUID="9a25e252-223d-41f2-abf7-632067c77e0b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:10:29.829877 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:29.829831 4927 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-5c3374/premium-w1-5c3374-kserve-6c56d4dcd-llwcl" podUID="dc813f02-e077-4251-8c1e-fea0986b0f36" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:10:30.068632 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:30.068585 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-a251fdfb/e2e-lifecycle-model-a251fdfb-kserve-5966998657-7wkbl" podUID="e9c18450-ac2a-4e6a-a712-00c2a755535d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.191:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:10:30.216474 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:30.213865 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-27e82b/distinct-w2-27e82b-kserve-75f475fcd9-mccsz" podUID="4ba8ced4-e93f-46ac-87c5-3d0b7fadc02d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:10:30.257679 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:30.257631 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5c3374/premium-w1-5c3374-kserve-6c56d4dcd-llwcl" podUID="dc813f02-e077-4251-8c1e-fea0986b0f36" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:10:30.278269 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:30.278227 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-35a187/distinct-2-w6-35a187-kserve-7ff4b6dc55-944g6" podUID="429ee092-42d8-475e-b0c6-b5a9930766fc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.113:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:10:30.320499 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:30.320454 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fe01fa/distinct-2-w5-fe01fa-kserve-79b99966d8-dkhzd" podUID="d5c1c9e8-ce47-4868-860d-b8413ff74ab5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.111:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 00:10:30.341781 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:30.341744 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-46b586/simulator-w0-46b586-kserve-67588b8dbd-z8xmn" podUID="9a25e252-223d-41f2-abf7-632067c77e0b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 00:10:30.342362 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:30.342324 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-46b586/simulator-w0-46b586-kserve-67588b8dbd-z8xmn" Oct 31 00:10:30.439621 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:30.439591 4927 scope.go:122] "RemoveContainer" containerID="114568ad1163e197c82f77d023d51d1b633d9e71899ffd33099b988dc40bc7b1" Oct 31 00:10:30.719581 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:30.717531 4927 scope.go:122] "RemoveContainer" containerID="188a1e9b3dd78fe1bb546ad232bf8e0c359561dc0b114ef937b2023182f1eba6" Oct 31 00:10:30.914371 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:30.914320 4927 scope.go:122] "RemoveContainer" containerID="929566ae98b0eb7a0ca1bb9d566a078364e0ed57cd4a769374a3e9fc79214289" Oct 31 00:10:31.125478 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:31.125374 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:10:31.229375 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:31.226139 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-27e82b/distinct-w2-27e82b-kserve-75f475fcd9-mccsz" podUID="4ba8ced4-e93f-46ac-87c5-3d0b7fadc02d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/ready\": context deadline exceeded" Oct 31 00:10:31.229375 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:31.226580 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-27e82b/distinct-w2-27e82b-kserve-75f475fcd9-mccsz" Oct 31 00:10:31.263482 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:31.263434 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5c3374/premium-w1-5c3374-kserve-6c56d4dcd-llwcl" podUID="dc813f02-e077-4251-8c1e-fea0986b0f36" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded" Oct 31 00:10:31.263918 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:31.263893 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5c3374/premium-w1-5c3374-kserve-6c56d4dcd-llwcl" Oct 31 00:10:31.303379 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:31.291401 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-35a187/distinct-2-w6-35a187-kserve-7ff4b6dc55-944g6" podUID="429ee092-42d8-475e-b0c6-b5a9930766fc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.113:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:10:31.303379 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:31.291968 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-35a187/distinct-2-w6-35a187-kserve-7ff4b6dc55-944g6" Oct 31 00:10:31.349914 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:31.349873 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-46b586/simulator-w0-46b586-kserve-67588b8dbd-z8xmn" podUID="9a25e252-223d-41f2-abf7-632067c77e0b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:10:31.350740 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:31.350710 4927 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-27e82b/distinct-w2-27e82b-kserve-75f475fcd9-mccsz" podUID="4ba8ced4-e93f-46ac-87c5-3d0b7fadc02d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/health\": context deadline exceeded" Oct 31 00:10:31.360554 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:31.359069 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fe01fa/distinct-2-w5-fe01fa-kserve-79b99966d8-dkhzd" podUID="d5c1c9e8-ce47-4868-860d-b8413ff74ab5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.111:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:10:31.360554 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:31.359526 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-fe01fa/distinct-2-w5-fe01fa-kserve-79b99966d8-dkhzd" Oct 31 00:10:31.541188 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:31.541081 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-fe01fa/distinct-2-w5-fe01fa-kserve-79b99966d8-dkhzd" Oct 31 00:10:32.102934 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:32.102851 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-35a187/distinct-2-w6-35a187-kserve-7ff4b6dc55-944g6" Oct 31 00:10:32.102934 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:32.102912 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-27e82b/distinct-w2-27e82b-kserve-75f475fcd9-mccsz" Oct 31 00:10:32.106957 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:32.106935 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5c3374/premium-w1-5c3374-kserve-6c56d4dcd-llwcl" Oct 31 00:10:32.186028 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:32.184749 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-46b586/simulator-w0-46b586-kserve-67588b8dbd-z8xmn" Oct 31 00:10:35.701310 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:35.701273 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:10:36.163497 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:36.163456 4927 patch_prober.go:29] interesting pod/e2e-shared-a-w0-d5360e-openshift-default-fbccfc549-d828w container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:10:36.163682 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:36.163518 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w0-d5360e-openshift-default-fbccfc549-d828w" podUID="37108b54-01b5-4e2c-add9-4e3aa1f3cd94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:10:40.293094 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:40.293058 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:10:41.377754 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:41.377718 4927 generic.go:356] "Generic (PLEG): container finished" podID="61bb8f6c-7dd6-4e87-ac44-5374dd11c0dc" containerID="294e83eb65d8d48cedf6847168059337f49522ad8337e6d31732cabd4f76172e" exitCode=0 Oct 31 00:10:41.378325 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:41.377791 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-0d75be-openshift-default-57f99d9cbc-smttn" event={"ID":"61bb8f6c-7dd6-4e87-ac44-5374dd11c0dc","Type":"ContainerDied","Data":"294e83eb65d8d48cedf6847168059337f49522ad8337e6d31732cabd4f76172e"} Oct 31 00:10:41.378325 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:41.378230 4927 scope.go:122] "RemoveContainer" containerID="294e83eb65d8d48cedf6847168059337f49522ad8337e6d31732cabd4f76172e" Oct 31 00:10:41.379689 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:41.379668 4927 generic.go:356] "Generic (PLEG): container finished" podID="37108b54-01b5-4e2c-add9-4e3aa1f3cd94" containerID="1afa8b2c79b4cf4621ad2544e1b2e946d465ab8c3fd31dbf19c27ef7e50497fb" exitCode=0 Oct 31 00:10:41.379802 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:41.379739 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-d5360e-openshift-default-fbccfc549-d828w" event={"ID":"37108b54-01b5-4e2c-add9-4e3aa1f3cd94","Type":"ContainerDied","Data":"1afa8b2c79b4cf4621ad2544e1b2e946d465ab8c3fd31dbf19c27ef7e50497fb"} Oct 31 00:10:41.379980 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:41.379952 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:10:41.380035 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:41.380022 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:10:41.380080 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:41.380058 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:10:41.380119 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:41.380087 4927 scope.go:122] "RemoveContainer" containerID="1afa8b2c79b4cf4621ad2544e1b2e946d465ab8c3fd31dbf19c27ef7e50497fb" Oct 31 00:10:41.381887 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:41.381858 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:10:41.381988 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:41.381934 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:10:41.382061 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:41.381982 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:10:42.390032 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:42.389992 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-0d75be-openshift-default-57f99d9cbc-smttn" event={"ID":"61bb8f6c-7dd6-4e87-ac44-5374dd11c0dc","Type":"ContainerStarted","Data":"17083e75a5e87c8c1227a281fb6b74a9245389f2e1d42a190bbcaaad71dba081"} Oct 31 00:10:42.392153 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:42.392126 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-d5360e-openshift-default-fbccfc549-d828w" event={"ID":"37108b54-01b5-4e2c-add9-4e3aa1f3cd94","Type":"ContainerStarted","Data":"8b5f60f2b79a29f7928e516a727b54d3bd66b732c36c54850aaacf43c6888d51"} Oct 31 00:10:42.687242 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:42.687211 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w6-0d75be-openshift-default-57f99d9cbc-smttn" Oct 31 00:10:42.688414 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:42.688389 4927 patch_prober.go:29] interesting pod/e2e-shared-b-w6-0d75be-openshift-default-57f99d9cbc-smttn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" start-of-body= Oct 31 00:10:42.688480 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:42.688442 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-0d75be-openshift-default-57f99d9cbc-smttn" podUID="61bb8f6c-7dd6-4e87-ac44-5374dd11c0dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" Oct 31 00:10:43.075531 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:43.074730 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-d5360e-openshift-default-fbccfc549-d828w" Oct 31 00:10:43.075531 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:43.074830 4927 patch_prober.go:29] interesting pod/e2e-shared-a-w0-d5360e-openshift-default-fbccfc549-d828w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" start-of-body= Oct 31 00:10:43.075531 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:43.074864 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-d5360e-openshift-default-fbccfc549-d828w" podUID="37108b54-01b5-4e2c-add9-4e3aa1f3cd94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" Oct 31 00:10:43.233479 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:43.233439 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w6-0d75be-openshift-default-57f99d9cbc-smttn" Oct 31 00:10:43.687139 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:43.687100 4927 patch_prober.go:29] interesting pod/e2e-shared-b-w6-0d75be-openshift-default-57f99d9cbc-smttn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" start-of-body= Oct 31 00:10:43.687552 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:43.687166 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-0d75be-openshift-default-57f99d9cbc-smttn" podUID="61bb8f6c-7dd6-4e87-ac44-5374dd11c0dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" Oct 31 00:10:44.066853 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:44.066754 4927 patch_prober.go:29] interesting pod/e2e-shared-a-w0-d5360e-openshift-default-fbccfc549-d828w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" start-of-body= Oct 31 00:10:44.067121 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:44.067088 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-d5360e-openshift-default-fbccfc549-d828w" podUID="37108b54-01b5-4e2c-add9-4e3aa1f3cd94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" Oct 31 00:10:44.703579 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:44.702667 4927 patch_prober.go:29] interesting pod/e2e-shared-b-w6-0d75be-openshift-default-57f99d9cbc-smttn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" start-of-body= Oct 31 00:10:44.703579 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:44.702741 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-0d75be-openshift-default-57f99d9cbc-smttn" podUID="61bb8f6c-7dd6-4e87-ac44-5374dd11c0dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" Oct 31 00:10:45.063265 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:45.063168 4927 patch_prober.go:29] interesting pod/e2e-shared-a-w0-d5360e-openshift-default-fbccfc549-d828w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" start-of-body= Oct 31 00:10:45.063545 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:45.063514 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-d5360e-openshift-default-fbccfc549-d828w" podUID="37108b54-01b5-4e2c-add9-4e3aa1f3cd94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" Oct 31 00:10:45.072476 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:45.072444 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:10:45.687612 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:45.687572 4927 patch_prober.go:29] interesting pod/e2e-shared-b-w6-0d75be-openshift-default-57f99d9cbc-smttn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" start-of-body= Oct 31 00:10:45.687827 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:45.687625 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-0d75be-openshift-default-57f99d9cbc-smttn" podUID="61bb8f6c-7dd6-4e87-ac44-5374dd11c0dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" Oct 31 00:10:46.066416 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:46.064370 4927 patch_prober.go:29] interesting pod/e2e-shared-a-w0-d5360e-openshift-default-fbccfc549-d828w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" start-of-body= Oct 31 00:10:46.066416 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:46.064448 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-d5360e-openshift-default-fbccfc549-d828w" podUID="37108b54-01b5-4e2c-add9-4e3aa1f3cd94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" Oct 31 00:10:46.687122 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:46.687090 4927 patch_prober.go:29] interesting pod/e2e-shared-b-w6-0d75be-openshift-default-57f99d9cbc-smttn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" start-of-body= Oct 31 00:10:46.687395 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:46.687371 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-0d75be-openshift-default-57f99d9cbc-smttn" podUID="61bb8f6c-7dd6-4e87-ac44-5374dd11c0dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" Oct 31 00:10:47.062013 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:47.061916 4927 patch_prober.go:29] interesting pod/e2e-shared-a-w0-d5360e-openshift-default-fbccfc549-d828w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" start-of-body= Oct 31 00:10:47.062287 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:47.062254 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-d5360e-openshift-default-fbccfc549-d828w" podUID="37108b54-01b5-4e2c-add9-4e3aa1f3cd94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" Oct 31 00:10:47.349410 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:47.349176 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:10:47.687390 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:47.687358 4927 patch_prober.go:29] interesting pod/e2e-shared-b-w6-0d75be-openshift-default-57f99d9cbc-smttn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" start-of-body= Oct 31 00:10:47.687624 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:47.687603 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-0d75be-openshift-default-57f99d9cbc-smttn" podUID="61bb8f6c-7dd6-4e87-ac44-5374dd11c0dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" Oct 31 00:10:48.037199 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:48.037098 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:10:48.060939 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:48.060897 4927 patch_prober.go:29] interesting pod/e2e-shared-a-w0-d5360e-openshift-default-fbccfc549-d828w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" start-of-body= Oct 31 00:10:48.061120 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:48.061004 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-d5360e-openshift-default-fbccfc549-d828w" podUID="37108b54-01b5-4e2c-add9-4e3aa1f3cd94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" Oct 31 00:10:48.704670 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:48.704638 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w6-0d75be-openshift-default-57f99d9cbc-smttn" Oct 31 00:10:48.708983 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:48.708949 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w6-0d75be-openshift-default-57f99d9cbc-smttn" Oct 31 00:10:49.065695 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:49.065619 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-d5360e-openshift-default-fbccfc549-d828w" Oct 31 00:10:49.066000 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:49.065977 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-d5360e-openshift-default-fbccfc549-d828w" Oct 31 00:10:49.066779 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:49.066760 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-d5360e-openshift-default-fbccfc549-d828w" Oct 31 00:10:49.367203 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:49.367123 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-39e3a8-openshift-default-6b6658797-ghqtx"] Oct 31 00:10:49.367887 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:49.367871 4927 cpu_manager.go:414] "containerMap: removing container" podUID="b9e9d9b1-22af-48d6-b1de-b1c28107ace1" containerName="istio-proxy" Oct 31 00:10:49.367975 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:49.367888 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9e9d9b1-22af-48d6-b1de-b1c28107ace1" containerName="istio-proxy" Oct 31 00:10:49.367975 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:49.367904 4927 cpu_manager.go:414] "containerMap: removing container" podUID="8789e67e-5656-4479-9b4f-fdc746fcfd15" containerName="istio-proxy" Oct 31 00:10:49.367975 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:49.367909 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8789e67e-5656-4479-9b4f-fdc746fcfd15" containerName="istio-proxy" Oct 31 00:10:49.367975 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:49.367926 4927 cpu_manager.go:414] "containerMap: removing container" podUID="469ac12b-4c77-403a-a190-af4037422211" containerName="istio-proxy" Oct 31 00:10:49.367975 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:49.367932 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="469ac12b-4c77-403a-a190-af4037422211" containerName="istio-proxy" Oct 31 00:10:49.368193 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:49.368066 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="469ac12b-4c77-403a-a190-af4037422211" containerName="istio-proxy" Oct 31 00:10:49.368193 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:49.368084 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9e9d9b1-22af-48d6-b1de-b1c28107ace1" containerName="istio-proxy" Oct 31 00:10:49.368193 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:49.368093 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="8789e67e-5656-4479-9b4f-fdc746fcfd15" containerName="istio-proxy" Oct 31 00:10:49.371361 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:49.371342 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-39e3a8-openshift-default-6b6658797-ghqtx" Oct 31 00:10:49.426749 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:49.426710 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-39e3a8-openshift-default-dockercfg-769cr\"" Oct 31 00:10:49.433989 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:49.433958 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-39e3a8-openshift-default-6b6658797-ghqtx"] Oct 31 00:10:49.470523 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:49.470496 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a19c11a-45ef-434e-9667-e75e17898834-workload-certs\") pod \"e2e-api-b-39e3a8-openshift-default-6b6658797-ghqtx\" (UID: \"7a19c11a-45ef-434e-9667-e75e17898834\") " pod="openshift-ingress/e2e-api-b-39e3a8-openshift-default-6b6658797-ghqtx" Oct 31 00:10:49.470707 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:49.470535 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a19c11a-45ef-434e-9667-e75e17898834-istio-podinfo\") pod \"e2e-api-b-39e3a8-openshift-default-6b6658797-ghqtx\" (UID: \"7a19c11a-45ef-434e-9667-e75e17898834\") " pod="openshift-ingress/e2e-api-b-39e3a8-openshift-default-6b6658797-ghqtx" Oct 31 00:10:49.470707 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:49.470583 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a19c11a-45ef-434e-9667-e75e17898834-istio-token\") pod \"e2e-api-b-39e3a8-openshift-default-6b6658797-ghqtx\" (UID: \"7a19c11a-45ef-434e-9667-e75e17898834\") " pod="openshift-ingress/e2e-api-b-39e3a8-openshift-default-6b6658797-ghqtx" Oct 31 00:10:49.470834 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:49.470757 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a19c11a-45ef-434e-9667-e75e17898834-istio-envoy\") pod \"e2e-api-b-39e3a8-openshift-default-6b6658797-ghqtx\" (UID: \"7a19c11a-45ef-434e-9667-e75e17898834\") " pod="openshift-ingress/e2e-api-b-39e3a8-openshift-default-6b6658797-ghqtx" Oct 31 00:10:49.470941 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:49.470916 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a19c11a-45ef-434e-9667-e75e17898834-istiod-ca-cert\") pod \"e2e-api-b-39e3a8-openshift-default-6b6658797-ghqtx\" (UID: \"7a19c11a-45ef-434e-9667-e75e17898834\") " pod="openshift-ingress/e2e-api-b-39e3a8-openshift-default-6b6658797-ghqtx" Oct 31 00:10:49.471016 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:49.470970 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a19c11a-45ef-434e-9667-e75e17898834-credential-socket\") pod \"e2e-api-b-39e3a8-openshift-default-6b6658797-ghqtx\" (UID: \"7a19c11a-45ef-434e-9667-e75e17898834\") " pod="openshift-ingress/e2e-api-b-39e3a8-openshift-default-6b6658797-ghqtx" Oct 31 00:10:49.471072 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:49.471049 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a19c11a-45ef-434e-9667-e75e17898834-workload-socket\") pod \"e2e-api-b-39e3a8-openshift-default-6b6658797-ghqtx\" (UID: \"7a19c11a-45ef-434e-9667-e75e17898834\") " pod="openshift-ingress/e2e-api-b-39e3a8-openshift-default-6b6658797-ghqtx" Oct 31 00:10:49.471132 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:49.471091 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qphrp\" (UniqueName: \"kubernetes.io/projected/7a19c11a-45ef-434e-9667-e75e17898834-kube-api-access-qphrp\") pod \"e2e-api-b-39e3a8-openshift-default-6b6658797-ghqtx\" (UID: \"7a19c11a-45ef-434e-9667-e75e17898834\") " pod="openshift-ingress/e2e-api-b-39e3a8-openshift-default-6b6658797-ghqtx" Oct 31 00:10:49.471132 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:49.471120 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a19c11a-45ef-434e-9667-e75e17898834-istio-data\") pod \"e2e-api-b-39e3a8-openshift-default-6b6658797-ghqtx\" (UID: \"7a19c11a-45ef-434e-9667-e75e17898834\") " pod="openshift-ingress/e2e-api-b-39e3a8-openshift-default-6b6658797-ghqtx" Oct 31 00:10:49.586809 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:49.573270 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a19c11a-45ef-434e-9667-e75e17898834-istio-token\") pod \"e2e-api-b-39e3a8-openshift-default-6b6658797-ghqtx\" (UID: \"7a19c11a-45ef-434e-9667-e75e17898834\") " pod="openshift-ingress/e2e-api-b-39e3a8-openshift-default-6b6658797-ghqtx" Oct 31 00:10:49.586809 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:49.573479 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a19c11a-45ef-434e-9667-e75e17898834-istio-envoy\") pod \"e2e-api-b-39e3a8-openshift-default-6b6658797-ghqtx\" (UID: \"7a19c11a-45ef-434e-9667-e75e17898834\") " pod="openshift-ingress/e2e-api-b-39e3a8-openshift-default-6b6658797-ghqtx" Oct 31 00:10:49.586809 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:49.573603 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a19c11a-45ef-434e-9667-e75e17898834-istiod-ca-cert\") pod \"e2e-api-b-39e3a8-openshift-default-6b6658797-ghqtx\" (UID: \"7a19c11a-45ef-434e-9667-e75e17898834\") " pod="openshift-ingress/e2e-api-b-39e3a8-openshift-default-6b6658797-ghqtx" Oct 31 00:10:49.586809 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:49.573647 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a19c11a-45ef-434e-9667-e75e17898834-credential-socket\") pod \"e2e-api-b-39e3a8-openshift-default-6b6658797-ghqtx\" (UID: \"7a19c11a-45ef-434e-9667-e75e17898834\") " pod="openshift-ingress/e2e-api-b-39e3a8-openshift-default-6b6658797-ghqtx" Oct 31 00:10:49.586809 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:49.573704 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a19c11a-45ef-434e-9667-e75e17898834-workload-socket\") pod \"e2e-api-b-39e3a8-openshift-default-6b6658797-ghqtx\" (UID: \"7a19c11a-45ef-434e-9667-e75e17898834\") " pod="openshift-ingress/e2e-api-b-39e3a8-openshift-default-6b6658797-ghqtx" Oct 31 00:10:49.586809 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:49.573785 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qphrp\" (UniqueName: \"kubernetes.io/projected/7a19c11a-45ef-434e-9667-e75e17898834-kube-api-access-qphrp\") pod \"e2e-api-b-39e3a8-openshift-default-6b6658797-ghqtx\" (UID: \"7a19c11a-45ef-434e-9667-e75e17898834\") " pod="openshift-ingress/e2e-api-b-39e3a8-openshift-default-6b6658797-ghqtx" Oct 31 00:10:49.586809 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:49.573814 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a19c11a-45ef-434e-9667-e75e17898834-istio-data\") pod \"e2e-api-b-39e3a8-openshift-default-6b6658797-ghqtx\" (UID: \"7a19c11a-45ef-434e-9667-e75e17898834\") " pod="openshift-ingress/e2e-api-b-39e3a8-openshift-default-6b6658797-ghqtx" Oct 31 00:10:49.586809 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:49.573943 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a19c11a-45ef-434e-9667-e75e17898834-workload-certs\") pod \"e2e-api-b-39e3a8-openshift-default-6b6658797-ghqtx\" (UID: \"7a19c11a-45ef-434e-9667-e75e17898834\") " pod="openshift-ingress/e2e-api-b-39e3a8-openshift-default-6b6658797-ghqtx" Oct 31 00:10:49.586809 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:49.574019 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a19c11a-45ef-434e-9667-e75e17898834-istio-podinfo\") pod \"e2e-api-b-39e3a8-openshift-default-6b6658797-ghqtx\" (UID: \"7a19c11a-45ef-434e-9667-e75e17898834\") " pod="openshift-ingress/e2e-api-b-39e3a8-openshift-default-6b6658797-ghqtx" Oct 31 00:10:49.586809 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:49.585100 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a19c11a-45ef-434e-9667-e75e17898834-credential-socket\") pod \"e2e-api-b-39e3a8-openshift-default-6b6658797-ghqtx\" (UID: \"7a19c11a-45ef-434e-9667-e75e17898834\") " pod="openshift-ingress/e2e-api-b-39e3a8-openshift-default-6b6658797-ghqtx" Oct 31 00:10:49.588566 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:49.588536 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a19c11a-45ef-434e-9667-e75e17898834-istiod-ca-cert\") pod \"e2e-api-b-39e3a8-openshift-default-6b6658797-ghqtx\" (UID: \"7a19c11a-45ef-434e-9667-e75e17898834\") " pod="openshift-ingress/e2e-api-b-39e3a8-openshift-default-6b6658797-ghqtx" Oct 31 00:10:49.589947 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:49.588971 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a19c11a-45ef-434e-9667-e75e17898834-workload-socket\") pod \"e2e-api-b-39e3a8-openshift-default-6b6658797-ghqtx\" (UID: \"7a19c11a-45ef-434e-9667-e75e17898834\") " pod="openshift-ingress/e2e-api-b-39e3a8-openshift-default-6b6658797-ghqtx" Oct 31 00:10:49.590112 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:49.589596 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a19c11a-45ef-434e-9667-e75e17898834-istio-data\") pod \"e2e-api-b-39e3a8-openshift-default-6b6658797-ghqtx\" (UID: \"7a19c11a-45ef-434e-9667-e75e17898834\") " pod="openshift-ingress/e2e-api-b-39e3a8-openshift-default-6b6658797-ghqtx" Oct 31 00:10:49.590241 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:49.589850 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a19c11a-45ef-434e-9667-e75e17898834-workload-certs\") pod \"e2e-api-b-39e3a8-openshift-default-6b6658797-ghqtx\" (UID: \"7a19c11a-45ef-434e-9667-e75e17898834\") " pod="openshift-ingress/e2e-api-b-39e3a8-openshift-default-6b6658797-ghqtx" Oct 31 00:10:49.622399 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:49.605736 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a19c11a-45ef-434e-9667-e75e17898834-istio-envoy\") pod \"e2e-api-b-39e3a8-openshift-default-6b6658797-ghqtx\" (UID: \"7a19c11a-45ef-434e-9667-e75e17898834\") " pod="openshift-ingress/e2e-api-b-39e3a8-openshift-default-6b6658797-ghqtx" Oct 31 00:10:49.622399 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:49.622131 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a19c11a-45ef-434e-9667-e75e17898834-istio-podinfo\") pod \"e2e-api-b-39e3a8-openshift-default-6b6658797-ghqtx\" (UID: \"7a19c11a-45ef-434e-9667-e75e17898834\") " pod="openshift-ingress/e2e-api-b-39e3a8-openshift-default-6b6658797-ghqtx" Oct 31 00:10:49.623231 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:49.623198 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:10:49.663222 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:49.663182 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qphrp\" (UniqueName: \"kubernetes.io/projected/7a19c11a-45ef-434e-9667-e75e17898834-kube-api-access-qphrp\") pod \"e2e-api-b-39e3a8-openshift-default-6b6658797-ghqtx\" (UID: \"7a19c11a-45ef-434e-9667-e75e17898834\") " pod="openshift-ingress/e2e-api-b-39e3a8-openshift-default-6b6658797-ghqtx" Oct 31 00:10:49.679352 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:49.673706 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a19c11a-45ef-434e-9667-e75e17898834-istio-token\") pod \"e2e-api-b-39e3a8-openshift-default-6b6658797-ghqtx\" (UID: \"7a19c11a-45ef-434e-9667-e75e17898834\") " pod="openshift-ingress/e2e-api-b-39e3a8-openshift-default-6b6658797-ghqtx" Oct 31 00:10:49.694557 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:49.688630 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-39e3a8-openshift-default-6b6658797-ghqtx" Oct 31 00:10:49.783347 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:10:49.783289 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sqv69" containerName="extractor" Oct 31 00:10:49.796376 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:49.796320 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:10:50.013297 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:50.013259 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:10:50.233772 ip-10-0-141-180 kubenswrapper[4927]: W1031 00:10:50.233740 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a19c11a_45ef_434e_9667_e75e17898834.slice/crio-079aa604033378b84f3b0900109d737a9c0665e94c9b4800f81f4b464e670138 WatchSource:0}: Error finding container 079aa604033378b84f3b0900109d737a9c0665e94c9b4800f81f4b464e670138: Status 404 returned error can't find the container with id 079aa604033378b84f3b0900109d737a9c0665e94c9b4800f81f4b464e670138 Oct 31 00:10:50.234244 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:50.234216 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-39e3a8-openshift-default-6b6658797-ghqtx"] Oct 31 00:10:50.236560 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:50.236526 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:10:50.236639 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:50.236595 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:10:50.237170 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:50.237155 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:10:50.660940 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:50.660903 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-39e3a8-openshift-default-6b6658797-ghqtx" event={"ID":"7a19c11a-45ef-434e-9667-e75e17898834","Type":"ContainerStarted","Data":"77edaee9d31170d4678826952d723309b37e54f7f38e8893aa8fc44583812e4e"} Oct 31 00:10:50.661150 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:50.660960 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-39e3a8-openshift-default-6b6658797-ghqtx" event={"ID":"7a19c11a-45ef-434e-9667-e75e17898834","Type":"ContainerStarted","Data":"079aa604033378b84f3b0900109d737a9c0665e94c9b4800f81f4b464e670138"} Oct 31 00:10:50.689798 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:50.689771 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-39e3a8-openshift-default-6b6658797-ghqtx" Oct 31 00:10:50.786035 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:50.785981 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-39e3a8-openshift-default-6b6658797-ghqtx" podStartSLOduration=1.7859658569999999 podStartE2EDuration="1.785965857s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:10:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:10:50.783954528 +0000 UTC m=+1763.820285362" watchObservedRunningTime="2026-10-31 00:10:50.785965857 +0000 UTC m=+1763.822296678" Oct 31 00:10:51.696327 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:51.696291 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-39e3a8-openshift-default-6b6658797-ghqtx" Oct 31 00:10:52.678922 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:52.678884 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-39e3a8-openshift-default-6b6658797-ghqtx" Oct 31 00:10:52.679822 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:52.679800 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-39e3a8-openshift-default-6b6658797-ghqtx" Oct 31 00:10:53.587551 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:53.587469 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:10:53.836997 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:53.836960 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:10:54.703052 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:54.703019 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:10:56.719524 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:56.719487 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:10:56.974404 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:56.972966 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:10:57.130237 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:57.130198 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-39e3a8-545496b975-zvh7k"] Oct 31 00:10:57.151573 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:57.151526 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-39e3a8-545496b975-zvh7k" Oct 31 00:10:57.224287 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:57.222277 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ml6x\" (UniqueName: \"kubernetes.io/projected/bb4e1d22-509e-4c3f-b877-8094320fc9ed-kube-api-access-8ml6x\") pod \"payload-pre-processing-e2e-api-b-39e3a8-545496b975-zvh7k\" (UID: \"bb4e1d22-509e-4c3f-b877-8094320fc9ed\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-39e3a8-545496b975-zvh7k" Oct 31 00:10:57.224287 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:57.222498 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb4e1d22-509e-4c3f-b877-8094320fc9ed-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-39e3a8-545496b975-zvh7k\" (UID: \"bb4e1d22-509e-4c3f-b877-8094320fc9ed\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-39e3a8-545496b975-zvh7k" Oct 31 00:10:57.238285 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:57.236610 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-39e3a8\"" Oct 31 00:10:57.238285 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:57.236750 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-39e3a8-dockercfg-6d6d7\"" Oct 31 00:10:57.249419 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:57.248748 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-39e3a8-545496b975-zvh7k"] Oct 31 00:10:57.323758 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:57.323724 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-39e3a8-77d67d47f8-msm6p"] Oct 31 00:10:57.329724 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:57.324958 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb4e1d22-509e-4c3f-b877-8094320fc9ed-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-39e3a8-545496b975-zvh7k\" (UID: \"bb4e1d22-509e-4c3f-b877-8094320fc9ed\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-39e3a8-545496b975-zvh7k" Oct 31 00:10:57.330034 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:57.325912 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb4e1d22-509e-4c3f-b877-8094320fc9ed-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-39e3a8-545496b975-zvh7k\" (UID: \"bb4e1d22-509e-4c3f-b877-8094320fc9ed\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-39e3a8-545496b975-zvh7k" Oct 31 00:10:57.330494 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:57.330470 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ml6x\" (UniqueName: \"kubernetes.io/projected/bb4e1d22-509e-4c3f-b877-8094320fc9ed-kube-api-access-8ml6x\") pod \"payload-pre-processing-e2e-api-b-39e3a8-545496b975-zvh7k\" (UID: \"bb4e1d22-509e-4c3f-b877-8094320fc9ed\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-39e3a8-545496b975-zvh7k" Oct 31 00:10:57.338788 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:57.338765 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-39e3a8-77d67d47f8-msm6p" Oct 31 00:10:57.445356 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:57.441583 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d639aac3-8752-495f-8c71-fdc050a03016-plugins-config-volume\") pod \"payload-processing-e2e-api-b-39e3a8-77d67d47f8-msm6p\" (UID: \"d639aac3-8752-495f-8c71-fdc050a03016\") " pod="openshift-ingress/payload-processing-e2e-api-b-39e3a8-77d67d47f8-msm6p" Oct 31 00:10:57.445356 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:57.441659 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhcnk\" (UniqueName: \"kubernetes.io/projected/d639aac3-8752-495f-8c71-fdc050a03016-kube-api-access-qhcnk\") pod \"payload-processing-e2e-api-b-39e3a8-77d67d47f8-msm6p\" (UID: \"d639aac3-8752-495f-8c71-fdc050a03016\") " pod="openshift-ingress/payload-processing-e2e-api-b-39e3a8-77d67d47f8-msm6p" Oct 31 00:10:57.486361 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:57.448446 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ml6x\" (UniqueName: \"kubernetes.io/projected/bb4e1d22-509e-4c3f-b877-8094320fc9ed-kube-api-access-8ml6x\") pod \"payload-pre-processing-e2e-api-b-39e3a8-545496b975-zvh7k\" (UID: \"bb4e1d22-509e-4c3f-b877-8094320fc9ed\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-39e3a8-545496b975-zvh7k" Oct 31 00:10:57.486361 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:57.474552 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-39e3a8-77d67d47f8-msm6p"] Oct 31 00:10:57.542541 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:57.542402 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-39e3a8-545496b975-zvh7k" Oct 31 00:10:57.554366 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:57.543773 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d639aac3-8752-495f-8c71-fdc050a03016-plugins-config-volume\") pod \"payload-processing-e2e-api-b-39e3a8-77d67d47f8-msm6p\" (UID: \"d639aac3-8752-495f-8c71-fdc050a03016\") " pod="openshift-ingress/payload-processing-e2e-api-b-39e3a8-77d67d47f8-msm6p" Oct 31 00:10:57.554366 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:57.543844 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhcnk\" (UniqueName: \"kubernetes.io/projected/d639aac3-8752-495f-8c71-fdc050a03016-kube-api-access-qhcnk\") pod \"payload-processing-e2e-api-b-39e3a8-77d67d47f8-msm6p\" (UID: \"d639aac3-8752-495f-8c71-fdc050a03016\") " pod="openshift-ingress/payload-processing-e2e-api-b-39e3a8-77d67d47f8-msm6p" Oct 31 00:10:57.554366 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:57.550945 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d639aac3-8752-495f-8c71-fdc050a03016-plugins-config-volume\") pod \"payload-processing-e2e-api-b-39e3a8-77d67d47f8-msm6p\" (UID: \"d639aac3-8752-495f-8c71-fdc050a03016\") " pod="openshift-ingress/payload-processing-e2e-api-b-39e3a8-77d67d47f8-msm6p" Oct 31 00:10:57.606497 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:57.606468 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhcnk\" (UniqueName: \"kubernetes.io/projected/d639aac3-8752-495f-8c71-fdc050a03016-kube-api-access-qhcnk\") pod \"payload-processing-e2e-api-b-39e3a8-77d67d47f8-msm6p\" (UID: \"d639aac3-8752-495f-8c71-fdc050a03016\") " pod="openshift-ingress/payload-processing-e2e-api-b-39e3a8-77d67d47f8-msm6p" Oct 31 00:10:57.673413 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:57.673382 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-39e3a8-77d67d47f8-msm6p" Oct 31 00:10:57.955308 ip-10-0-141-180 kubenswrapper[4927]: W1031 00:10:57.955281 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb4e1d22_509e_4c3f_b877_8094320fc9ed.slice/crio-df2cc1246d1ecfd32bacbe85effb277e14f1ffea2f8920cdd98be7d613c00bc3 WatchSource:0}: Error finding container df2cc1246d1ecfd32bacbe85effb277e14f1ffea2f8920cdd98be7d613c00bc3: Status 404 returned error can't find the container with id df2cc1246d1ecfd32bacbe85effb277e14f1ffea2f8920cdd98be7d613c00bc3 Oct 31 00:10:57.955676 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:57.955525 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-39e3a8-545496b975-zvh7k"] Oct 31 00:10:58.085326 ip-10-0-141-180 kubenswrapper[4927]: W1031 00:10:58.085299 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd639aac3_8752_495f_8c71_fdc050a03016.slice/crio-a0cf8d283bcd81efd941211c6f197286fae82c1a733d877604b2eea34894b6fe WatchSource:0}: Error finding container a0cf8d283bcd81efd941211c6f197286fae82c1a733d877604b2eea34894b6fe: Status 404 returned error can't find the container with id a0cf8d283bcd81efd941211c6f197286fae82c1a733d877604b2eea34894b6fe Oct 31 00:10:58.085551 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:58.085530 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-39e3a8-77d67d47f8-msm6p"] Oct 31 00:10:58.737378 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:58.737325 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-39e3a8-77d67d47f8-msm6p" event={"ID":"d639aac3-8752-495f-8c71-fdc050a03016","Type":"ContainerStarted","Data":"f0cf05276ff8f0af48fa34cdc41c1996d82b68c2f0f9dd0ef839a5dde96cfeae"} Oct 31 00:10:58.737694 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:58.737406 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-39e3a8-77d67d47f8-msm6p" event={"ID":"d639aac3-8752-495f-8c71-fdc050a03016","Type":"ContainerStarted","Data":"a0cf8d283bcd81efd941211c6f197286fae82c1a733d877604b2eea34894b6fe"} Oct 31 00:10:58.737694 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:58.737462 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-39e3a8-77d67d47f8-msm6p" Oct 31 00:10:58.739232 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:58.739204 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-39e3a8-545496b975-zvh7k" event={"ID":"bb4e1d22-509e-4c3f-b877-8094320fc9ed","Type":"ContainerStarted","Data":"3b73a43624872dcf2600dbd50de0b019e2514dbf3c1fafb527f23ec7f5ce5fd0"} Oct 31 00:10:58.739400 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:58.739247 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-39e3a8-545496b975-zvh7k" event={"ID":"bb4e1d22-509e-4c3f-b877-8094320fc9ed","Type":"ContainerStarted","Data":"df2cc1246d1ecfd32bacbe85effb277e14f1ffea2f8920cdd98be7d613c00bc3"} Oct 31 00:10:58.739400 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:58.739297 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-39e3a8-545496b975-zvh7k" Oct 31 00:10:58.860428 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:58.860376 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-39e3a8-77d67d47f8-msm6p" podStartSLOduration=1.860362178 podStartE2EDuration="1.860362178s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:10:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:10:58.855730453 +0000 UTC m=+1771.892061277" watchObservedRunningTime="2026-10-31 00:10:58.860362178 +0000 UTC m=+1771.896693000" Oct 31 00:10:59.832747 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:10:59.832707 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:11:04.590028 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:04.589981 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:11:08.203861 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:08.203799 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:11:09.368504 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:09.368466 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:11:09.751386 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:09.751353 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-39e3a8-545496b975-zvh7k" Oct 31 00:11:09.751580 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:09.751515 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-39e3a8-77d67d47f8-msm6p" Oct 31 00:11:09.872625 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:09.872572 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-39e3a8-545496b975-zvh7k" podStartSLOduration=12.872557722 podStartE2EDuration="12.872557722s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:10:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:10:59.004045397 +0000 UTC m=+1772.040376255" watchObservedRunningTime="2026-10-31 00:11:09.872557722 +0000 UTC m=+1782.908888543" Oct 31 00:11:09.976035 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:09.975996 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:11:10.574628 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:10.574582 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-03c703-openshift-default-78bd587bff-5m2n8"] Oct 31 00:11:10.579871 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:10.579846 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-03c703-openshift-default-78bd587bff-5m2n8" Oct 31 00:11:10.634290 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:10.634256 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-03c703-openshift-default-dockercfg-zxhtq\"" Oct 31 00:11:10.658781 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:10.658746 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ee78c4f-f1a0-4cc0-8441-106def585fff-workload-certs\") pod \"e2e-inf-a-03c703-openshift-default-78bd587bff-5m2n8\" (UID: \"9ee78c4f-f1a0-4cc0-8441-106def585fff\") " pod="openshift-ingress/e2e-inf-a-03c703-openshift-default-78bd587bff-5m2n8" Oct 31 00:11:10.659872 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:10.659839 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ee78c4f-f1a0-4cc0-8441-106def585fff-istio-envoy\") pod \"e2e-inf-a-03c703-openshift-default-78bd587bff-5m2n8\" (UID: \"9ee78c4f-f1a0-4cc0-8441-106def585fff\") " pod="openshift-ingress/e2e-inf-a-03c703-openshift-default-78bd587bff-5m2n8" Oct 31 00:11:10.660788 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:10.660762 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ee78c4f-f1a0-4cc0-8441-106def585fff-istiod-ca-cert\") pod \"e2e-inf-a-03c703-openshift-default-78bd587bff-5m2n8\" (UID: \"9ee78c4f-f1a0-4cc0-8441-106def585fff\") " pod="openshift-ingress/e2e-inf-a-03c703-openshift-default-78bd587bff-5m2n8" Oct 31 00:11:10.661165 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:10.661140 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ee78c4f-f1a0-4cc0-8441-106def585fff-credential-socket\") pod \"e2e-inf-a-03c703-openshift-default-78bd587bff-5m2n8\" (UID: \"9ee78c4f-f1a0-4cc0-8441-106def585fff\") " pod="openshift-ingress/e2e-inf-a-03c703-openshift-default-78bd587bff-5m2n8" Oct 31 00:11:10.661657 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:10.661634 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ee78c4f-f1a0-4cc0-8441-106def585fff-istio-podinfo\") pod \"e2e-inf-a-03c703-openshift-default-78bd587bff-5m2n8\" (UID: \"9ee78c4f-f1a0-4cc0-8441-106def585fff\") " pod="openshift-ingress/e2e-inf-a-03c703-openshift-default-78bd587bff-5m2n8" Oct 31 00:11:10.661980 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:10.661958 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ee78c4f-f1a0-4cc0-8441-106def585fff-istio-data\") pod \"e2e-inf-a-03c703-openshift-default-78bd587bff-5m2n8\" (UID: \"9ee78c4f-f1a0-4cc0-8441-106def585fff\") " pod="openshift-ingress/e2e-inf-a-03c703-openshift-default-78bd587bff-5m2n8" Oct 31 00:11:10.662151 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:10.662130 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fktm9\" (UniqueName: \"kubernetes.io/projected/9ee78c4f-f1a0-4cc0-8441-106def585fff-kube-api-access-fktm9\") pod \"e2e-inf-a-03c703-openshift-default-78bd587bff-5m2n8\" (UID: \"9ee78c4f-f1a0-4cc0-8441-106def585fff\") " pod="openshift-ingress/e2e-inf-a-03c703-openshift-default-78bd587bff-5m2n8" Oct 31 00:11:10.662251 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:10.662206 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ee78c4f-f1a0-4cc0-8441-106def585fff-istio-token\") pod \"e2e-inf-a-03c703-openshift-default-78bd587bff-5m2n8\" (UID: \"9ee78c4f-f1a0-4cc0-8441-106def585fff\") " pod="openshift-ingress/e2e-inf-a-03c703-openshift-default-78bd587bff-5m2n8" Oct 31 00:11:10.662251 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:10.662234 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ee78c4f-f1a0-4cc0-8441-106def585fff-workload-socket\") pod \"e2e-inf-a-03c703-openshift-default-78bd587bff-5m2n8\" (UID: \"9ee78c4f-f1a0-4cc0-8441-106def585fff\") " pod="openshift-ingress/e2e-inf-a-03c703-openshift-default-78bd587bff-5m2n8" Oct 31 00:11:10.677897 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:10.677871 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:11:10.695096 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:10.695062 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-03c703-openshift-default-78bd587bff-5m2n8"] Oct 31 00:11:10.767354 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:10.766457 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ee78c4f-f1a0-4cc0-8441-106def585fff-workload-certs\") pod \"e2e-inf-a-03c703-openshift-default-78bd587bff-5m2n8\" (UID: \"9ee78c4f-f1a0-4cc0-8441-106def585fff\") " pod="openshift-ingress/e2e-inf-a-03c703-openshift-default-78bd587bff-5m2n8" Oct 31 00:11:10.767354 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:10.766606 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ee78c4f-f1a0-4cc0-8441-106def585fff-istio-envoy\") pod \"e2e-inf-a-03c703-openshift-default-78bd587bff-5m2n8\" (UID: \"9ee78c4f-f1a0-4cc0-8441-106def585fff\") " pod="openshift-ingress/e2e-inf-a-03c703-openshift-default-78bd587bff-5m2n8" Oct 31 00:11:10.767354 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:10.766644 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ee78c4f-f1a0-4cc0-8441-106def585fff-istiod-ca-cert\") pod \"e2e-inf-a-03c703-openshift-default-78bd587bff-5m2n8\" (UID: \"9ee78c4f-f1a0-4cc0-8441-106def585fff\") " pod="openshift-ingress/e2e-inf-a-03c703-openshift-default-78bd587bff-5m2n8" Oct 31 00:11:10.767354 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:10.766691 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ee78c4f-f1a0-4cc0-8441-106def585fff-credential-socket\") pod \"e2e-inf-a-03c703-openshift-default-78bd587bff-5m2n8\" (UID: \"9ee78c4f-f1a0-4cc0-8441-106def585fff\") " pod="openshift-ingress/e2e-inf-a-03c703-openshift-default-78bd587bff-5m2n8" Oct 31 00:11:10.767354 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:10.766732 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ee78c4f-f1a0-4cc0-8441-106def585fff-istio-podinfo\") pod \"e2e-inf-a-03c703-openshift-default-78bd587bff-5m2n8\" (UID: \"9ee78c4f-f1a0-4cc0-8441-106def585fff\") " pod="openshift-ingress/e2e-inf-a-03c703-openshift-default-78bd587bff-5m2n8" Oct 31 00:11:10.767354 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:10.766974 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ee78c4f-f1a0-4cc0-8441-106def585fff-istio-data\") pod \"e2e-inf-a-03c703-openshift-default-78bd587bff-5m2n8\" (UID: \"9ee78c4f-f1a0-4cc0-8441-106def585fff\") " pod="openshift-ingress/e2e-inf-a-03c703-openshift-default-78bd587bff-5m2n8" Oct 31 00:11:10.767354 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:10.767036 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fktm9\" (UniqueName: \"kubernetes.io/projected/9ee78c4f-f1a0-4cc0-8441-106def585fff-kube-api-access-fktm9\") pod \"e2e-inf-a-03c703-openshift-default-78bd587bff-5m2n8\" (UID: \"9ee78c4f-f1a0-4cc0-8441-106def585fff\") " pod="openshift-ingress/e2e-inf-a-03c703-openshift-default-78bd587bff-5m2n8" Oct 31 00:11:10.767354 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:10.767150 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ee78c4f-f1a0-4cc0-8441-106def585fff-istio-token\") pod \"e2e-inf-a-03c703-openshift-default-78bd587bff-5m2n8\" (UID: \"9ee78c4f-f1a0-4cc0-8441-106def585fff\") " pod="openshift-ingress/e2e-inf-a-03c703-openshift-default-78bd587bff-5m2n8" Oct 31 00:11:10.767354 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:10.767175 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ee78c4f-f1a0-4cc0-8441-106def585fff-workload-socket\") pod \"e2e-inf-a-03c703-openshift-default-78bd587bff-5m2n8\" (UID: \"9ee78c4f-f1a0-4cc0-8441-106def585fff\") " pod="openshift-ingress/e2e-inf-a-03c703-openshift-default-78bd587bff-5m2n8" Oct 31 00:11:10.768045 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:10.767889 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ee78c4f-f1a0-4cc0-8441-106def585fff-workload-socket\") pod \"e2e-inf-a-03c703-openshift-default-78bd587bff-5m2n8\" (UID: \"9ee78c4f-f1a0-4cc0-8441-106def585fff\") " pod="openshift-ingress/e2e-inf-a-03c703-openshift-default-78bd587bff-5m2n8" Oct 31 00:11:10.783701 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:10.768228 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ee78c4f-f1a0-4cc0-8441-106def585fff-workload-certs\") pod \"e2e-inf-a-03c703-openshift-default-78bd587bff-5m2n8\" (UID: \"9ee78c4f-f1a0-4cc0-8441-106def585fff\") " pod="openshift-ingress/e2e-inf-a-03c703-openshift-default-78bd587bff-5m2n8" Oct 31 00:11:10.824356 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:10.818744 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ee78c4f-f1a0-4cc0-8441-106def585fff-istio-data\") pod \"e2e-inf-a-03c703-openshift-default-78bd587bff-5m2n8\" (UID: \"9ee78c4f-f1a0-4cc0-8441-106def585fff\") " pod="openshift-ingress/e2e-inf-a-03c703-openshift-default-78bd587bff-5m2n8" Oct 31 00:11:10.824356 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:10.819231 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ee78c4f-f1a0-4cc0-8441-106def585fff-credential-socket\") pod \"e2e-inf-a-03c703-openshift-default-78bd587bff-5m2n8\" (UID: \"9ee78c4f-f1a0-4cc0-8441-106def585fff\") " pod="openshift-ingress/e2e-inf-a-03c703-openshift-default-78bd587bff-5m2n8" Oct 31 00:11:10.824356 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:10.819432 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ee78c4f-f1a0-4cc0-8441-106def585fff-istiod-ca-cert\") pod \"e2e-inf-a-03c703-openshift-default-78bd587bff-5m2n8\" (UID: \"9ee78c4f-f1a0-4cc0-8441-106def585fff\") " pod="openshift-ingress/e2e-inf-a-03c703-openshift-default-78bd587bff-5m2n8" Oct 31 00:11:10.856409 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:10.851237 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ee78c4f-f1a0-4cc0-8441-106def585fff-istio-podinfo\") pod \"e2e-inf-a-03c703-openshift-default-78bd587bff-5m2n8\" (UID: \"9ee78c4f-f1a0-4cc0-8441-106def585fff\") " pod="openshift-ingress/e2e-inf-a-03c703-openshift-default-78bd587bff-5m2n8" Oct 31 00:11:10.882412 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:10.876821 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ee78c4f-f1a0-4cc0-8441-106def585fff-istio-envoy\") pod \"e2e-inf-a-03c703-openshift-default-78bd587bff-5m2n8\" (UID: \"9ee78c4f-f1a0-4cc0-8441-106def585fff\") " pod="openshift-ingress/e2e-inf-a-03c703-openshift-default-78bd587bff-5m2n8" Oct 31 00:11:10.924155 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:10.924123 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fktm9\" (UniqueName: \"kubernetes.io/projected/9ee78c4f-f1a0-4cc0-8441-106def585fff-kube-api-access-fktm9\") pod \"e2e-inf-a-03c703-openshift-default-78bd587bff-5m2n8\" (UID: \"9ee78c4f-f1a0-4cc0-8441-106def585fff\") " pod="openshift-ingress/e2e-inf-a-03c703-openshift-default-78bd587bff-5m2n8" Oct 31 00:11:10.946355 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:10.939696 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:11:10.996357 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:10.976489 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ee78c4f-f1a0-4cc0-8441-106def585fff-istio-token\") pod \"e2e-inf-a-03c703-openshift-default-78bd587bff-5m2n8\" (UID: \"9ee78c4f-f1a0-4cc0-8441-106def585fff\") " pod="openshift-ingress/e2e-inf-a-03c703-openshift-default-78bd587bff-5m2n8" Oct 31 00:11:11.220517 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:11.216604 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-03c703-openshift-default-78bd587bff-5m2n8" Oct 31 00:11:11.844437 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:11.844392 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-03c703-openshift-default-78bd587bff-5m2n8"] Oct 31 00:11:11.846472 ip-10-0-141-180 kubenswrapper[4927]: W1031 00:11:11.846437 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ee78c4f_f1a0_4cc0_8441_106def585fff.slice/crio-7843f7ce7891c3a4b27fd0179a97fb0d6fbe9ea064c0c1009a2a2b29e9875b42 WatchSource:0}: Error finding container 7843f7ce7891c3a4b27fd0179a97fb0d6fbe9ea064c0c1009a2a2b29e9875b42: Status 404 returned error can't find the container with id 7843f7ce7891c3a4b27fd0179a97fb0d6fbe9ea064c0c1009a2a2b29e9875b42 Oct 31 00:11:11.850110 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:11.850068 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:11:11.850214 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:11.850162 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:11:11.850214 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:11.850204 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:11:12.039354 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:12.037969 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-03c703-openshift-default-78bd587bff-5m2n8" event={"ID":"9ee78c4f-f1a0-4cc0-8441-106def585fff","Type":"ContainerStarted","Data":"7843f7ce7891c3a4b27fd0179a97fb0d6fbe9ea064c0c1009a2a2b29e9875b42"} Oct 31 00:11:13.055487 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:13.055451 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-03c703-openshift-default-78bd587bff-5m2n8" event={"ID":"9ee78c4f-f1a0-4cc0-8441-106def585fff","Type":"ContainerStarted","Data":"b203e94aec128fce45da60bb93ab9e2259f6808bbc342b332cb76a4071ddd011"} Oct 31 00:11:13.228466 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:13.228430 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-03c703-openshift-default-78bd587bff-5m2n8" Oct 31 00:11:13.241700 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:13.241644 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-03c703-openshift-default-78bd587bff-5m2n8" podStartSLOduration=3.241625496 podStartE2EDuration="3.241625496s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:11:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:11:13.188624726 +0000 UTC m=+1786.224955542" watchObservedRunningTime="2026-10-31 00:11:13.241625496 +0000 UTC m=+1786.277956320" Oct 31 00:11:13.243120 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:13.243092 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-03c703-openshift-default-78bd587bff-5m2n8" Oct 31 00:11:13.949384 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:13.949342 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-ab0ac9-5b7888b747-pxq8l"] Oct 31 00:11:13.954565 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:13.954539 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-ab0ac9-5b7888b747-pxq8l" Oct 31 00:11:14.040357 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:14.024883 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-ab0ac9-5b7888b747-pxq8l"] Oct 31 00:11:14.067844 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:14.067808 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w3-ab0ac9-serving-cert\"" Oct 31 00:11:14.070122 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:14.070100 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-03c703-openshift-default-78bd587bff-5m2n8" Oct 31 00:11:14.071903 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:14.071880 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-03c703-openshift-default-78bd587bff-5m2n8" Oct 31 00:11:14.074854 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:14.074831 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e8587ee7-1b21-409a-8c2d-ca9d7b69be4a-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-ab0ac9-5b7888b747-pxq8l\" (UID: \"e8587ee7-1b21-409a-8c2d-ca9d7b69be4a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-ab0ac9-5b7888b747-pxq8l" Oct 31 00:11:14.076213 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:14.075512 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lz649\" (UniqueName: \"kubernetes.io/projected/e8587ee7-1b21-409a-8c2d-ca9d7b69be4a-kube-api-access-lz649\") pod \"maas-api-e2e-shared-a-w3-ab0ac9-5b7888b747-pxq8l\" (UID: \"e8587ee7-1b21-409a-8c2d-ca9d7b69be4a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-ab0ac9-5b7888b747-pxq8l" Oct 31 00:11:14.076213 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:14.075587 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8587ee7-1b21-409a-8c2d-ca9d7b69be4a-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-ab0ac9-5b7888b747-pxq8l\" (UID: \"e8587ee7-1b21-409a-8c2d-ca9d7b69be4a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-ab0ac9-5b7888b747-pxq8l" Oct 31 00:11:14.076213 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:14.075947 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e8587ee7-1b21-409a-8c2d-ca9d7b69be4a-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-ab0ac9-5b7888b747-pxq8l\" (UID: \"e8587ee7-1b21-409a-8c2d-ca9d7b69be4a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-ab0ac9-5b7888b747-pxq8l" Oct 31 00:11:14.149809 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:14.149771 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:11:14.177245 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:14.177207 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e8587ee7-1b21-409a-8c2d-ca9d7b69be4a-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-ab0ac9-5b7888b747-pxq8l\" (UID: \"e8587ee7-1b21-409a-8c2d-ca9d7b69be4a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-ab0ac9-5b7888b747-pxq8l" Oct 31 00:11:14.177697 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:14.177297 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e8587ee7-1b21-409a-8c2d-ca9d7b69be4a-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-ab0ac9-5b7888b747-pxq8l\" (UID: \"e8587ee7-1b21-409a-8c2d-ca9d7b69be4a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-ab0ac9-5b7888b747-pxq8l" Oct 31 00:11:14.177953 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:14.177925 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lz649\" (UniqueName: \"kubernetes.io/projected/e8587ee7-1b21-409a-8c2d-ca9d7b69be4a-kube-api-access-lz649\") pod \"maas-api-e2e-shared-a-w3-ab0ac9-5b7888b747-pxq8l\" (UID: \"e8587ee7-1b21-409a-8c2d-ca9d7b69be4a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-ab0ac9-5b7888b747-pxq8l" Oct 31 00:11:14.178088 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:14.178028 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8587ee7-1b21-409a-8c2d-ca9d7b69be4a-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-ab0ac9-5b7888b747-pxq8l\" (UID: \"e8587ee7-1b21-409a-8c2d-ca9d7b69be4a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-ab0ac9-5b7888b747-pxq8l" Oct 31 00:11:14.179208 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:14.179183 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8587ee7-1b21-409a-8c2d-ca9d7b69be4a-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-ab0ac9-5b7888b747-pxq8l\" (UID: \"e8587ee7-1b21-409a-8c2d-ca9d7b69be4a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-ab0ac9-5b7888b747-pxq8l" Oct 31 00:11:14.180814 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:14.180793 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e8587ee7-1b21-409a-8c2d-ca9d7b69be4a-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-ab0ac9-5b7888b747-pxq8l\" (UID: \"e8587ee7-1b21-409a-8c2d-ca9d7b69be4a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-ab0ac9-5b7888b747-pxq8l" Oct 31 00:11:14.181092 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:14.181068 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e8587ee7-1b21-409a-8c2d-ca9d7b69be4a-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-ab0ac9-5b7888b747-pxq8l\" (UID: \"e8587ee7-1b21-409a-8c2d-ca9d7b69be4a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-ab0ac9-5b7888b747-pxq8l" Oct 31 00:11:14.242462 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:14.242381 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lz649\" (UniqueName: \"kubernetes.io/projected/e8587ee7-1b21-409a-8c2d-ca9d7b69be4a-kube-api-access-lz649\") pod \"maas-api-e2e-shared-a-w3-ab0ac9-5b7888b747-pxq8l\" (UID: \"e8587ee7-1b21-409a-8c2d-ca9d7b69be4a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-ab0ac9-5b7888b747-pxq8l" Oct 31 00:11:14.269897 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:14.269869 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-ab0ac9-5b7888b747-pxq8l" Oct 31 00:11:14.348528 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:14.333078 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:11:14.586798 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:14.586044 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:11:14.939680 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:14.935522 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-ab0ac9-5b7888b747-pxq8l"] Oct 31 00:11:14.942661 ip-10-0-141-180 kubenswrapper[4927]: W1031 00:11:14.942604 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8587ee7_1b21_409a_8c2d_ca9d7b69be4a.slice/crio-b31911c0a5f877bcec2d78aefd91e7a7321bea32bd24859b600c4a749ae8d7ba WatchSource:0}: Error finding container b31911c0a5f877bcec2d78aefd91e7a7321bea32bd24859b600c4a749ae8d7ba: Status 404 returned error can't find the container with id b31911c0a5f877bcec2d78aefd91e7a7321bea32bd24859b600c4a749ae8d7ba Oct 31 00:11:15.077275 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:15.077236 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-ab0ac9-5b7888b747-pxq8l" event={"ID":"e8587ee7-1b21-409a-8c2d-ca9d7b69be4a","Type":"ContainerStarted","Data":"b31911c0a5f877bcec2d78aefd91e7a7321bea32bd24859b600c4a749ae8d7ba"} Oct 31 00:11:16.087216 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:16.087182 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-ab0ac9-5b7888b747-pxq8l" event={"ID":"e8587ee7-1b21-409a-8c2d-ca9d7b69be4a","Type":"ContainerStarted","Data":"555021ce9e0b1d669136f3a2152a57aa9794f2ab916744e8f011247599f01083"} Oct 31 00:11:16.087631 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:16.087264 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-ab0ac9-5b7888b747-pxq8l" Oct 31 00:11:16.240238 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:16.240137 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-ab0ac9-5b7888b747-pxq8l" podStartSLOduration=2.247104368 podStartE2EDuration="3.240120592s" totalImagesPullingTime="993.016224ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:11:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:11:16.237777073 +0000 UTC m=+1789.274107925" watchObservedRunningTime="2026-10-31 00:11:16.240120592 +0000 UTC m=+1789.276451436" Oct 31 00:11:16.481471 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:16.481248 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:11:16.638664 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:16.638567 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:11:16.792251 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:16.792209 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:11:18.733111 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:18.733065 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:11:19.189194 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:19.189148 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:11:19.417641 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:19.417589 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:11:19.702408 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:11:19.702370 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sqv69" containerName="extractor" Oct 31 00:11:20.029872 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:20.029836 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w0-957019-openshift-default-59bc4d44f9-8xtcd"] Oct 31 00:11:20.052938 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:20.052909 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-957019-openshift-default-59bc4d44f9-8xtcd" Oct 31 00:11:20.112390 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:20.109575 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w0-957019-openshift-default-dockercfg-npgtt\"" Oct 31 00:11:20.133312 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:20.133278 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:11:20.139086 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:20.139057 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b623edb-855c-44b8-a0c1-abbd33925bd4-credential-socket\") pod \"e2e-shared-b-w0-957019-openshift-default-59bc4d44f9-8xtcd\" (UID: \"4b623edb-855c-44b8-a0c1-abbd33925bd4\") " pod="openshift-ingress/e2e-shared-b-w0-957019-openshift-default-59bc4d44f9-8xtcd" Oct 31 00:11:20.139240 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:20.139132 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b623edb-855c-44b8-a0c1-abbd33925bd4-istio-data\") pod \"e2e-shared-b-w0-957019-openshift-default-59bc4d44f9-8xtcd\" (UID: \"4b623edb-855c-44b8-a0c1-abbd33925bd4\") " pod="openshift-ingress/e2e-shared-b-w0-957019-openshift-default-59bc4d44f9-8xtcd" Oct 31 00:11:20.139240 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:20.139223 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b623edb-855c-44b8-a0c1-abbd33925bd4-istio-token\") pod \"e2e-shared-b-w0-957019-openshift-default-59bc4d44f9-8xtcd\" (UID: \"4b623edb-855c-44b8-a0c1-abbd33925bd4\") " pod="openshift-ingress/e2e-shared-b-w0-957019-openshift-default-59bc4d44f9-8xtcd" Oct 31 00:11:20.139343 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:20.139286 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vflf\" (UniqueName: \"kubernetes.io/projected/4b623edb-855c-44b8-a0c1-abbd33925bd4-kube-api-access-5vflf\") pod \"e2e-shared-b-w0-957019-openshift-default-59bc4d44f9-8xtcd\" (UID: \"4b623edb-855c-44b8-a0c1-abbd33925bd4\") " pod="openshift-ingress/e2e-shared-b-w0-957019-openshift-default-59bc4d44f9-8xtcd" Oct 31 00:11:20.139443 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:20.139420 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b623edb-855c-44b8-a0c1-abbd33925bd4-istiod-ca-cert\") pod \"e2e-shared-b-w0-957019-openshift-default-59bc4d44f9-8xtcd\" (UID: \"4b623edb-855c-44b8-a0c1-abbd33925bd4\") " pod="openshift-ingress/e2e-shared-b-w0-957019-openshift-default-59bc4d44f9-8xtcd" Oct 31 00:11:20.139598 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:20.139578 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b623edb-855c-44b8-a0c1-abbd33925bd4-istio-podinfo\") pod \"e2e-shared-b-w0-957019-openshift-default-59bc4d44f9-8xtcd\" (UID: \"4b623edb-855c-44b8-a0c1-abbd33925bd4\") " pod="openshift-ingress/e2e-shared-b-w0-957019-openshift-default-59bc4d44f9-8xtcd" Oct 31 00:11:20.139684 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:20.139620 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b623edb-855c-44b8-a0c1-abbd33925bd4-workload-socket\") pod \"e2e-shared-b-w0-957019-openshift-default-59bc4d44f9-8xtcd\" (UID: \"4b623edb-855c-44b8-a0c1-abbd33925bd4\") " pod="openshift-ingress/e2e-shared-b-w0-957019-openshift-default-59bc4d44f9-8xtcd" Oct 31 00:11:20.139753 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:20.139720 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b623edb-855c-44b8-a0c1-abbd33925bd4-istio-envoy\") pod \"e2e-shared-b-w0-957019-openshift-default-59bc4d44f9-8xtcd\" (UID: \"4b623edb-855c-44b8-a0c1-abbd33925bd4\") " pod="openshift-ingress/e2e-shared-b-w0-957019-openshift-default-59bc4d44f9-8xtcd" Oct 31 00:11:20.139817 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:20.139763 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b623edb-855c-44b8-a0c1-abbd33925bd4-workload-certs\") pod \"e2e-shared-b-w0-957019-openshift-default-59bc4d44f9-8xtcd\" (UID: \"4b623edb-855c-44b8-a0c1-abbd33925bd4\") " pod="openshift-ingress/e2e-shared-b-w0-957019-openshift-default-59bc4d44f9-8xtcd" Oct 31 00:11:20.165028 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:20.164990 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-957019-openshift-default-59bc4d44f9-8xtcd"] Oct 31 00:11:20.243394 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:20.241573 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b623edb-855c-44b8-a0c1-abbd33925bd4-istio-podinfo\") pod \"e2e-shared-b-w0-957019-openshift-default-59bc4d44f9-8xtcd\" (UID: \"4b623edb-855c-44b8-a0c1-abbd33925bd4\") " pod="openshift-ingress/e2e-shared-b-w0-957019-openshift-default-59bc4d44f9-8xtcd" Oct 31 00:11:20.243394 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:20.241634 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b623edb-855c-44b8-a0c1-abbd33925bd4-workload-socket\") pod \"e2e-shared-b-w0-957019-openshift-default-59bc4d44f9-8xtcd\" (UID: \"4b623edb-855c-44b8-a0c1-abbd33925bd4\") " pod="openshift-ingress/e2e-shared-b-w0-957019-openshift-default-59bc4d44f9-8xtcd" Oct 31 00:11:20.243394 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:20.241699 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b623edb-855c-44b8-a0c1-abbd33925bd4-istio-envoy\") pod \"e2e-shared-b-w0-957019-openshift-default-59bc4d44f9-8xtcd\" (UID: \"4b623edb-855c-44b8-a0c1-abbd33925bd4\") " pod="openshift-ingress/e2e-shared-b-w0-957019-openshift-default-59bc4d44f9-8xtcd" Oct 31 00:11:20.243394 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:20.241737 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b623edb-855c-44b8-a0c1-abbd33925bd4-workload-certs\") pod \"e2e-shared-b-w0-957019-openshift-default-59bc4d44f9-8xtcd\" (UID: \"4b623edb-855c-44b8-a0c1-abbd33925bd4\") " pod="openshift-ingress/e2e-shared-b-w0-957019-openshift-default-59bc4d44f9-8xtcd" Oct 31 00:11:20.243394 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:20.241844 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b623edb-855c-44b8-a0c1-abbd33925bd4-credential-socket\") pod \"e2e-shared-b-w0-957019-openshift-default-59bc4d44f9-8xtcd\" (UID: \"4b623edb-855c-44b8-a0c1-abbd33925bd4\") " pod="openshift-ingress/e2e-shared-b-w0-957019-openshift-default-59bc4d44f9-8xtcd" Oct 31 00:11:20.243394 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:20.241925 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b623edb-855c-44b8-a0c1-abbd33925bd4-istio-data\") pod \"e2e-shared-b-w0-957019-openshift-default-59bc4d44f9-8xtcd\" (UID: \"4b623edb-855c-44b8-a0c1-abbd33925bd4\") " pod="openshift-ingress/e2e-shared-b-w0-957019-openshift-default-59bc4d44f9-8xtcd" Oct 31 00:11:20.243394 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:20.241956 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b623edb-855c-44b8-a0c1-abbd33925bd4-istio-token\") pod \"e2e-shared-b-w0-957019-openshift-default-59bc4d44f9-8xtcd\" (UID: \"4b623edb-855c-44b8-a0c1-abbd33925bd4\") " pod="openshift-ingress/e2e-shared-b-w0-957019-openshift-default-59bc4d44f9-8xtcd" Oct 31 00:11:20.243394 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:20.242004 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vflf\" (UniqueName: \"kubernetes.io/projected/4b623edb-855c-44b8-a0c1-abbd33925bd4-kube-api-access-5vflf\") pod \"e2e-shared-b-w0-957019-openshift-default-59bc4d44f9-8xtcd\" (UID: \"4b623edb-855c-44b8-a0c1-abbd33925bd4\") " pod="openshift-ingress/e2e-shared-b-w0-957019-openshift-default-59bc4d44f9-8xtcd" Oct 31 00:11:20.243394 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:20.242081 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b623edb-855c-44b8-a0c1-abbd33925bd4-istiod-ca-cert\") pod \"e2e-shared-b-w0-957019-openshift-default-59bc4d44f9-8xtcd\" (UID: \"4b623edb-855c-44b8-a0c1-abbd33925bd4\") " pod="openshift-ingress/e2e-shared-b-w0-957019-openshift-default-59bc4d44f9-8xtcd" Oct 31 00:11:20.243394 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:20.243028 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b623edb-855c-44b8-a0c1-abbd33925bd4-istiod-ca-cert\") pod \"e2e-shared-b-w0-957019-openshift-default-59bc4d44f9-8xtcd\" (UID: \"4b623edb-855c-44b8-a0c1-abbd33925bd4\") " pod="openshift-ingress/e2e-shared-b-w0-957019-openshift-default-59bc4d44f9-8xtcd" Oct 31 00:11:20.297455 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:20.277167 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b623edb-855c-44b8-a0c1-abbd33925bd4-credential-socket\") pod \"e2e-shared-b-w0-957019-openshift-default-59bc4d44f9-8xtcd\" (UID: \"4b623edb-855c-44b8-a0c1-abbd33925bd4\") " pod="openshift-ingress/e2e-shared-b-w0-957019-openshift-default-59bc4d44f9-8xtcd" Oct 31 00:11:20.297455 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:20.277519 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b623edb-855c-44b8-a0c1-abbd33925bd4-workload-socket\") pod \"e2e-shared-b-w0-957019-openshift-default-59bc4d44f9-8xtcd\" (UID: \"4b623edb-855c-44b8-a0c1-abbd33925bd4\") " pod="openshift-ingress/e2e-shared-b-w0-957019-openshift-default-59bc4d44f9-8xtcd" Oct 31 00:11:20.297455 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:20.278530 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b623edb-855c-44b8-a0c1-abbd33925bd4-workload-certs\") pod \"e2e-shared-b-w0-957019-openshift-default-59bc4d44f9-8xtcd\" (UID: \"4b623edb-855c-44b8-a0c1-abbd33925bd4\") " pod="openshift-ingress/e2e-shared-b-w0-957019-openshift-default-59bc4d44f9-8xtcd" Oct 31 00:11:20.297455 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:20.281386 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b623edb-855c-44b8-a0c1-abbd33925bd4-istio-data\") pod \"e2e-shared-b-w0-957019-openshift-default-59bc4d44f9-8xtcd\" (UID: \"4b623edb-855c-44b8-a0c1-abbd33925bd4\") " pod="openshift-ingress/e2e-shared-b-w0-957019-openshift-default-59bc4d44f9-8xtcd" Oct 31 00:11:20.323359 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:20.311177 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b623edb-855c-44b8-a0c1-abbd33925bd4-istio-envoy\") pod \"e2e-shared-b-w0-957019-openshift-default-59bc4d44f9-8xtcd\" (UID: \"4b623edb-855c-44b8-a0c1-abbd33925bd4\") " pod="openshift-ingress/e2e-shared-b-w0-957019-openshift-default-59bc4d44f9-8xtcd" Oct 31 00:11:20.323359 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:20.313489 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b623edb-855c-44b8-a0c1-abbd33925bd4-istio-podinfo\") pod \"e2e-shared-b-w0-957019-openshift-default-59bc4d44f9-8xtcd\" (UID: \"4b623edb-855c-44b8-a0c1-abbd33925bd4\") " pod="openshift-ingress/e2e-shared-b-w0-957019-openshift-default-59bc4d44f9-8xtcd" Oct 31 00:11:20.344955 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:20.344276 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b623edb-855c-44b8-a0c1-abbd33925bd4-istio-token\") pod \"e2e-shared-b-w0-957019-openshift-default-59bc4d44f9-8xtcd\" (UID: \"4b623edb-855c-44b8-a0c1-abbd33925bd4\") " pod="openshift-ingress/e2e-shared-b-w0-957019-openshift-default-59bc4d44f9-8xtcd" Oct 31 00:11:20.344955 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:20.344366 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vflf\" (UniqueName: \"kubernetes.io/projected/4b623edb-855c-44b8-a0c1-abbd33925bd4-kube-api-access-5vflf\") pod \"e2e-shared-b-w0-957019-openshift-default-59bc4d44f9-8xtcd\" (UID: \"4b623edb-855c-44b8-a0c1-abbd33925bd4\") " pod="openshift-ingress/e2e-shared-b-w0-957019-openshift-default-59bc4d44f9-8xtcd" Oct 31 00:11:20.375169 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:20.375125 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:11:20.385679 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:20.385649 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-957019-openshift-default-59bc4d44f9-8xtcd" Oct 31 00:11:20.864555 ip-10-0-141-180 kubenswrapper[4927]: W1031 00:11:20.864521 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b623edb_855c_44b8_a0c1_abbd33925bd4.slice/crio-0b96aaed85980a6afd72a71d0a227b0d09800adf26b12ab03c9982553e0767bb WatchSource:0}: Error finding container 0b96aaed85980a6afd72a71d0a227b0d09800adf26b12ab03c9982553e0767bb: Status 404 returned error can't find the container with id 0b96aaed85980a6afd72a71d0a227b0d09800adf26b12ab03c9982553e0767bb Oct 31 00:11:20.865097 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:20.865071 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-957019-openshift-default-59bc4d44f9-8xtcd"] Oct 31 00:11:20.867671 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:20.867639 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:11:20.867757 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:20.867705 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:11:20.867757 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:20.867735 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:11:21.320210 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:21.320167 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-957019-openshift-default-59bc4d44f9-8xtcd" event={"ID":"4b623edb-855c-44b8-a0c1-abbd33925bd4","Type":"ContainerStarted","Data":"cdcb32181d652df3402395552e0ae0898b41240322995bcf0f51e7ff004c2707"} Oct 31 00:11:21.320718 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:21.320240 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-957019-openshift-default-59bc4d44f9-8xtcd" event={"ID":"4b623edb-855c-44b8-a0c1-abbd33925bd4","Type":"ContainerStarted","Data":"0b96aaed85980a6afd72a71d0a227b0d09800adf26b12ab03c9982553e0767bb"} Oct 31 00:11:21.386879 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:21.386845 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-957019-openshift-default-59bc4d44f9-8xtcd" Oct 31 00:11:21.389305 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:21.389277 4927 patch_prober.go:29] interesting pod/e2e-shared-b-w0-957019-openshift-default-59bc4d44f9-8xtcd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" start-of-body= Oct 31 00:11:21.389473 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:21.389344 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-957019-openshift-default-59bc4d44f9-8xtcd" podUID="4b623edb-855c-44b8-a0c1-abbd33925bd4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" Oct 31 00:11:21.445853 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:21.445783 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w0-957019-openshift-default-59bc4d44f9-8xtcd" podStartSLOduration=2.445763604 podStartE2EDuration="2.445763604s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:11:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:11:21.442603404 +0000 UTC m=+1794.478934251" watchObservedRunningTime="2026-10-31 00:11:21.445763604 +0000 UTC m=+1794.482094431" Oct 31 00:11:21.597381 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:21.597283 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:11:21.771369 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:21.750227 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:11:22.046423 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:22.046385 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-03c703-68ccf588f9-957fw"] Oct 31 00:11:22.066416 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:22.066388 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-03c703-68ccf588f9-957fw" Oct 31 00:11:22.093541 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:22.093505 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-03c703-68ccf588f9-957fw"] Oct 31 00:11:22.107406 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:22.107376 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:11:22.130413 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:22.124653 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-03c703-dockercfg-cxlm2\"" Oct 31 00:11:22.133675 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:22.133614 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-03c703\"" Oct 31 00:11:22.157380 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:22.136427 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7afae12-c562-4a37-adf2-506d8780f93f-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-03c703-68ccf588f9-957fw\" (UID: \"a7afae12-c562-4a37-adf2-506d8780f93f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-03c703-68ccf588f9-957fw" Oct 31 00:11:22.157380 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:22.136552 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmsql\" (UniqueName: \"kubernetes.io/projected/a7afae12-c562-4a37-adf2-506d8780f93f-kube-api-access-cmsql\") pod \"payload-pre-processing-e2e-inf-a-03c703-68ccf588f9-957fw\" (UID: \"a7afae12-c562-4a37-adf2-506d8780f93f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-03c703-68ccf588f9-957fw" Oct 31 00:11:22.227917 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:22.198740 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-ab0ac9-5b7888b747-pxq8l" Oct 31 00:11:22.240786 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:22.238223 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmsql\" (UniqueName: \"kubernetes.io/projected/a7afae12-c562-4a37-adf2-506d8780f93f-kube-api-access-cmsql\") pod \"payload-pre-processing-e2e-inf-a-03c703-68ccf588f9-957fw\" (UID: \"a7afae12-c562-4a37-adf2-506d8780f93f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-03c703-68ccf588f9-957fw" Oct 31 00:11:22.240786 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:22.238474 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7afae12-c562-4a37-adf2-506d8780f93f-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-03c703-68ccf588f9-957fw\" (UID: \"a7afae12-c562-4a37-adf2-506d8780f93f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-03c703-68ccf588f9-957fw" Oct 31 00:11:22.259191 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:22.259157 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7afae12-c562-4a37-adf2-506d8780f93f-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-03c703-68ccf588f9-957fw\" (UID: \"a7afae12-c562-4a37-adf2-506d8780f93f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-03c703-68ccf588f9-957fw" Oct 31 00:11:22.265375 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:22.263381 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-03c703-587b89bbdd-vzfkm"] Oct 31 00:11:22.313772 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:22.300713 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-03c703-587b89bbdd-vzfkm" Oct 31 00:11:22.385535 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:22.385491 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-03c703-587b89bbdd-vzfkm"] Oct 31 00:11:22.459319 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:22.459257 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmsql\" (UniqueName: \"kubernetes.io/projected/a7afae12-c562-4a37-adf2-506d8780f93f-kube-api-access-cmsql\") pod \"payload-pre-processing-e2e-inf-a-03c703-68ccf588f9-957fw\" (UID: \"a7afae12-c562-4a37-adf2-506d8780f93f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-03c703-68ccf588f9-957fw" Oct 31 00:11:22.495895 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:22.495865 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-957019-openshift-default-59bc4d44f9-8xtcd" Oct 31 00:11:22.497084 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:22.497059 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9106a2b-acad-4af9-af60-757055c5d135-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-03c703-587b89bbdd-vzfkm\" (UID: \"b9106a2b-acad-4af9-af60-757055c5d135\") " pod="openshift-ingress/payload-processing-e2e-inf-a-03c703-587b89bbdd-vzfkm" Oct 31 00:11:22.497443 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:22.497423 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhdff\" (UniqueName: \"kubernetes.io/projected/b9106a2b-acad-4af9-af60-757055c5d135-kube-api-access-dhdff\") pod \"payload-processing-e2e-inf-a-03c703-587b89bbdd-vzfkm\" (UID: \"b9106a2b-acad-4af9-af60-757055c5d135\") " pod="openshift-ingress/payload-processing-e2e-inf-a-03c703-587b89bbdd-vzfkm" Oct 31 00:11:22.499772 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:22.499744 4927 generic.go:356] "Generic (PLEG): container finished" podID="85098bb1-0c3e-4ecb-a771-3f57503a7cd8" containerID="7b5c45f7b944ca39203a0d1ceca2181c2fe8985765f8f818ff71500e3dbcfc04" exitCode=0 Oct 31 00:11:22.499918 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:22.499899 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-54ad8c-openshift-default-575c545fdf-gvfbr" event={"ID":"85098bb1-0c3e-4ecb-a771-3f57503a7cd8","Type":"ContainerDied","Data":"7b5c45f7b944ca39203a0d1ceca2181c2fe8985765f8f818ff71500e3dbcfc04"} Oct 31 00:11:22.500237 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:22.500222 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-957019-openshift-default-59bc4d44f9-8xtcd" Oct 31 00:11:22.501485 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:22.500956 4927 scope.go:122] "RemoveContainer" containerID="7b5c45f7b944ca39203a0d1ceca2181c2fe8985765f8f818ff71500e3dbcfc04" Oct 31 00:11:22.502542 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:22.501931 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-957019-openshift-default-59bc4d44f9-8xtcd" Oct 31 00:11:22.503211 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:22.503174 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:11:22.503312 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:22.503264 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:11:22.503418 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:22.503315 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:11:22.599647 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:22.599622 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9106a2b-acad-4af9-af60-757055c5d135-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-03c703-587b89bbdd-vzfkm\" (UID: \"b9106a2b-acad-4af9-af60-757055c5d135\") " pod="openshift-ingress/payload-processing-e2e-inf-a-03c703-587b89bbdd-vzfkm" Oct 31 00:11:22.600127 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:22.600104 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhdff\" (UniqueName: \"kubernetes.io/projected/b9106a2b-acad-4af9-af60-757055c5d135-kube-api-access-dhdff\") pod \"payload-processing-e2e-inf-a-03c703-587b89bbdd-vzfkm\" (UID: \"b9106a2b-acad-4af9-af60-757055c5d135\") " pod="openshift-ingress/payload-processing-e2e-inf-a-03c703-587b89bbdd-vzfkm" Oct 31 00:11:22.600424 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:22.600406 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9106a2b-acad-4af9-af60-757055c5d135-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-03c703-587b89bbdd-vzfkm\" (UID: \"b9106a2b-acad-4af9-af60-757055c5d135\") " pod="openshift-ingress/payload-processing-e2e-inf-a-03c703-587b89bbdd-vzfkm" Oct 31 00:11:22.670378 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:22.662914 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhdff\" (UniqueName: \"kubernetes.io/projected/b9106a2b-acad-4af9-af60-757055c5d135-kube-api-access-dhdff\") pod \"payload-processing-e2e-inf-a-03c703-587b89bbdd-vzfkm\" (UID: \"b9106a2b-acad-4af9-af60-757055c5d135\") " pod="openshift-ingress/payload-processing-e2e-inf-a-03c703-587b89bbdd-vzfkm" Oct 31 00:11:22.723283 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:22.723241 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-03c703-68ccf588f9-957fw" Oct 31 00:11:22.799165 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:22.798655 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-03c703-587b89bbdd-vzfkm" Oct 31 00:11:23.129560 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:23.129529 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-03c703-68ccf588f9-957fw"] Oct 31 00:11:23.129780 ip-10-0-141-180 kubenswrapper[4927]: W1031 00:11:23.129756 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7afae12_c562_4a37_adf2_506d8780f93f.slice/crio-4f4d6d409c0326af1d7af95c57711f2a7a20eab8a6389eb2e7302a7f192b5c61 WatchSource:0}: Error finding container 4f4d6d409c0326af1d7af95c57711f2a7a20eab8a6389eb2e7302a7f192b5c61: Status 404 returned error can't find the container with id 4f4d6d409c0326af1d7af95c57711f2a7a20eab8a6389eb2e7302a7f192b5c61 Oct 31 00:11:23.246630 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:23.246592 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-03c703-587b89bbdd-vzfkm"] Oct 31 00:11:23.533674 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:23.533632 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-03c703-68ccf588f9-957fw" event={"ID":"a7afae12-c562-4a37-adf2-506d8780f93f","Type":"ContainerStarted","Data":"f85af1aee7b46a0f95e7cf09c0f45dccb852bc978228e81c8a84f8915cf3b05e"} Oct 31 00:11:23.534099 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:23.533699 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-03c703-68ccf588f9-957fw" event={"ID":"a7afae12-c562-4a37-adf2-506d8780f93f","Type":"ContainerStarted","Data":"4f4d6d409c0326af1d7af95c57711f2a7a20eab8a6389eb2e7302a7f192b5c61"} Oct 31 00:11:23.534099 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:23.533752 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-03c703-68ccf588f9-957fw" Oct 31 00:11:23.536083 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:23.536052 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-54ad8c-openshift-default-575c545fdf-gvfbr" event={"ID":"85098bb1-0c3e-4ecb-a771-3f57503a7cd8","Type":"ContainerStarted","Data":"efb5ceaf3b35dc1ac4f015f54c759b4866359273451e2abac9bdbfd626bbca2a"} Oct 31 00:11:23.537811 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:23.537784 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-03c703-587b89bbdd-vzfkm" event={"ID":"b9106a2b-acad-4af9-af60-757055c5d135","Type":"ContainerStarted","Data":"b3f8af41a9ca14245a70502788a0cb1772c90d0d3ebb633769bb736c20ec3d75"} Oct 31 00:11:23.537942 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:23.537826 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-03c703-587b89bbdd-vzfkm" event={"ID":"b9106a2b-acad-4af9-af60-757055c5d135","Type":"ContainerStarted","Data":"273fbf1ef2735f96a5606cac324fd3de7024c8e612275e2862eaf0b78b0d5971"} Oct 31 00:11:23.537942 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:23.537874 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-03c703-587b89bbdd-vzfkm" Oct 31 00:11:23.654067 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:23.654011 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-03c703-68ccf588f9-957fw" podStartSLOduration=2.653997631 podStartE2EDuration="2.653997631s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:11:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:11:23.650965672 +0000 UTC m=+1796.687296513" watchObservedRunningTime="2026-10-31 00:11:23.653997631 +0000 UTC m=+1796.690328453" Oct 31 00:11:23.741286 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:23.741249 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-54ad8c-openshift-default-575c545fdf-gvfbr" Oct 31 00:11:23.742909 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:23.742884 4927 patch_prober.go:29] interesting pod/e2e-shared-a-w4-54ad8c-openshift-default-575c545fdf-gvfbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body= Oct 31 00:11:23.742995 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:23.742935 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-54ad8c-openshift-default-575c545fdf-gvfbr" podUID="85098bb1-0c3e-4ecb-a771-3f57503a7cd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" Oct 31 00:11:23.785168 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:23.785041 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-03c703-587b89bbdd-vzfkm" podStartSLOduration=1.7850208969999999 podStartE2EDuration="1.785020897s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:11:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:11:23.770736809 +0000 UTC m=+1796.807067651" watchObservedRunningTime="2026-10-31 00:11:23.785020897 +0000 UTC m=+1796.821351721" Oct 31 00:11:24.259752 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:24.259714 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:11:24.476366 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:24.476017 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:11:24.641476 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:24.640938 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:11:24.741377 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:24.741311 4927 patch_prober.go:29] interesting pod/e2e-shared-a-w4-54ad8c-openshift-default-575c545fdf-gvfbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body= Oct 31 00:11:24.741566 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:24.741400 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-54ad8c-openshift-default-575c545fdf-gvfbr" podUID="85098bb1-0c3e-4ecb-a771-3f57503a7cd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" Oct 31 00:11:25.741276 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:25.741238 4927 patch_prober.go:29] interesting pod/e2e-shared-a-w4-54ad8c-openshift-default-575c545fdf-gvfbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body= Oct 31 00:11:25.741719 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:25.741303 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-54ad8c-openshift-default-575c545fdf-gvfbr" podUID="85098bb1-0c3e-4ecb-a771-3f57503a7cd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" Oct 31 00:11:26.732357 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:26.730430 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-54ad8c-openshift-default-575c545fdf-gvfbr" Oct 31 00:11:26.742031 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:26.742000 4927 patch_prober.go:29] interesting pod/e2e-shared-a-w4-54ad8c-openshift-default-575c545fdf-gvfbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body= Oct 31 00:11:26.742429 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:26.742053 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-54ad8c-openshift-default-575c545fdf-gvfbr" podUID="85098bb1-0c3e-4ecb-a771-3f57503a7cd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" Oct 31 00:11:27.742021 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:27.741979 4927 patch_prober.go:29] interesting pod/e2e-shared-a-w4-54ad8c-openshift-default-575c545fdf-gvfbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body= Oct 31 00:11:27.742212 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:27.742042 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-54ad8c-openshift-default-575c545fdf-gvfbr" podUID="85098bb1-0c3e-4ecb-a771-3f57503a7cd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" Oct 31 00:11:27.973225 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:27.973188 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:11:28.280850 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:28.280816 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:11:28.479255 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:28.479221 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:11:28.501881 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:28.501852 4927 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-pqt2c_6382dfb5-a363-47a2-bc79-0b96f7bec0c3/console-operator/3.log" Oct 31 00:11:28.665022 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:28.664992 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:11:28.759257 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:28.759218 4927 patch_prober.go:29] interesting pod/e2e-shared-a-w4-54ad8c-openshift-default-575c545fdf-gvfbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body= Oct 31 00:11:28.760041 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:28.760009 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-54ad8c-openshift-default-575c545fdf-gvfbr" podUID="85098bb1-0c3e-4ecb-a771-3f57503a7cd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" Oct 31 00:11:28.783366 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:28.780641 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-957019-76d548dcf-bbqz5"] Oct 31 00:11:28.814651 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:28.814618 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-957019-76d548dcf-bbqz5" Oct 31 00:11:28.861041 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:28.818387 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-957019-76d548dcf-bbqz5"] Oct 31 00:11:28.982410 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:28.951647 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/358e61e1-8b72-4d73-b3ec-9016ebe766df-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-957019-76d548dcf-bbqz5\" (UID: \"358e61e1-8b72-4d73-b3ec-9016ebe766df\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-957019-76d548dcf-bbqz5" Oct 31 00:11:28.995417 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:28.995374 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pndm9\" (UniqueName: \"kubernetes.io/projected/358e61e1-8b72-4d73-b3ec-9016ebe766df-kube-api-access-pndm9\") pod \"payload-pre-processing-e2e-shared-b-w0-957019-76d548dcf-bbqz5\" (UID: \"358e61e1-8b72-4d73-b3ec-9016ebe766df\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-957019-76d548dcf-bbqz5" Oct 31 00:11:29.038361 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:29.015754 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w0-957019\"" Oct 31 00:11:29.038361 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:29.025898 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w0-957019-dockercfg-tf2hw\"" Oct 31 00:11:29.139356 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:29.131218 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pndm9\" (UniqueName: \"kubernetes.io/projected/358e61e1-8b72-4d73-b3ec-9016ebe766df-kube-api-access-pndm9\") pod \"payload-pre-processing-e2e-shared-b-w0-957019-76d548dcf-bbqz5\" (UID: \"358e61e1-8b72-4d73-b3ec-9016ebe766df\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-957019-76d548dcf-bbqz5" Oct 31 00:11:29.139356 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:29.131521 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/358e61e1-8b72-4d73-b3ec-9016ebe766df-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-957019-76d548dcf-bbqz5\" (UID: \"358e61e1-8b72-4d73-b3ec-9016ebe766df\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-957019-76d548dcf-bbqz5" Oct 31 00:11:29.157986 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:29.157945 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/358e61e1-8b72-4d73-b3ec-9016ebe766df-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-957019-76d548dcf-bbqz5\" (UID: \"358e61e1-8b72-4d73-b3ec-9016ebe766df\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-957019-76d548dcf-bbqz5" Oct 31 00:11:29.227364 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:29.226713 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pndm9\" (UniqueName: \"kubernetes.io/projected/358e61e1-8b72-4d73-b3ec-9016ebe766df-kube-api-access-pndm9\") pod \"payload-pre-processing-e2e-shared-b-w0-957019-76d548dcf-bbqz5\" (UID: \"358e61e1-8b72-4d73-b3ec-9016ebe766df\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-957019-76d548dcf-bbqz5" Oct 31 00:11:29.237446 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:29.228419 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-957019-76d548dcf-bbqz5" Oct 31 00:11:29.259357 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:29.259263 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:11:29.268599 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:29.268129 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-957019-788c577f9b-2shs7"] Oct 31 00:11:29.276257 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:29.276196 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-957019-788c577f9b-2shs7"] Oct 31 00:11:29.276388 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:29.276310 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-957019-788c577f9b-2shs7" Oct 31 00:11:29.390309 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:29.375032 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fb8gp\" (UniqueName: \"kubernetes.io/projected/ed77f29b-ed8c-428a-a8f7-1c55892d64a8-kube-api-access-fb8gp\") pod \"payload-processing-e2e-shared-b-w0-957019-788c577f9b-2shs7\" (UID: \"ed77f29b-ed8c-428a-a8f7-1c55892d64a8\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-957019-788c577f9b-2shs7" Oct 31 00:11:29.390953 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:29.390905 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed77f29b-ed8c-428a-a8f7-1c55892d64a8-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-957019-788c577f9b-2shs7\" (UID: \"ed77f29b-ed8c-428a-a8f7-1c55892d64a8\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-957019-788c577f9b-2shs7" Oct 31 00:11:29.421849 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:29.420422 4927 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-pqt2c_6382dfb5-a363-47a2-bc79-0b96f7bec0c3/console-operator/3.log" Oct 31 00:11:29.508622 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:29.506100 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed77f29b-ed8c-428a-a8f7-1c55892d64a8-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-957019-788c577f9b-2shs7\" (UID: \"ed77f29b-ed8c-428a-a8f7-1c55892d64a8\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-957019-788c577f9b-2shs7" Oct 31 00:11:29.508881 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:29.508857 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fb8gp\" (UniqueName: \"kubernetes.io/projected/ed77f29b-ed8c-428a-a8f7-1c55892d64a8-kube-api-access-fb8gp\") pod \"payload-processing-e2e-shared-b-w0-957019-788c577f9b-2shs7\" (UID: \"ed77f29b-ed8c-428a-a8f7-1c55892d64a8\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-957019-788c577f9b-2shs7" Oct 31 00:11:29.525136 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:29.525094 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed77f29b-ed8c-428a-a8f7-1c55892d64a8-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-957019-788c577f9b-2shs7\" (UID: \"ed77f29b-ed8c-428a-a8f7-1c55892d64a8\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-957019-788c577f9b-2shs7" Oct 31 00:11:29.578317 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:29.578279 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fb8gp\" (UniqueName: \"kubernetes.io/projected/ed77f29b-ed8c-428a-a8f7-1c55892d64a8-kube-api-access-fb8gp\") pod \"payload-processing-e2e-shared-b-w0-957019-788c577f9b-2shs7\" (UID: \"ed77f29b-ed8c-428a-a8f7-1c55892d64a8\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-957019-788c577f9b-2shs7" Oct 31 00:11:29.746879 ip-10-0-141-180 kubenswrapper[4927]: W1031 00:11:29.746845 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod358e61e1_8b72_4d73_b3ec_9016ebe766df.slice/crio-5dd73fd4e823bab43538738cb06e02c89e1bac3dcaf347b874a023d039689873 WatchSource:0}: Error finding container 5dd73fd4e823bab43538738cb06e02c89e1bac3dcaf347b874a023d039689873: Status 404 returned error can't find the container with id 5dd73fd4e823bab43538738cb06e02c89e1bac3dcaf347b874a023d039689873 Oct 31 00:11:29.748292 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:29.748261 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-957019-76d548dcf-bbqz5"] Oct 31 00:11:29.763471 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:29.763398 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-54ad8c-openshift-default-575c545fdf-gvfbr" Oct 31 00:11:29.770044 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:29.769934 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-54ad8c-openshift-default-575c545fdf-gvfbr" Oct 31 00:11:29.808550 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:29.808505 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-957019-788c577f9b-2shs7" Oct 31 00:11:30.204494 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:30.204463 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-957019-788c577f9b-2shs7"] Oct 31 00:11:30.204632 ip-10-0-141-180 kubenswrapper[4927]: W1031 00:11:30.204470 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded77f29b_ed8c_428a_a8f7_1c55892d64a8.slice/crio-488c18389ffd9db13a0da965bee820f1572087eff0c93eabd0fd584d8290b9c0 WatchSource:0}: Error finding container 488c18389ffd9db13a0da965bee820f1572087eff0c93eabd0fd584d8290b9c0: Status 404 returned error can't find the container with id 488c18389ffd9db13a0da965bee820f1572087eff0c93eabd0fd584d8290b9c0 Oct 31 00:11:30.506899 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:30.506787 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:11:30.658262 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:30.658227 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:11:30.699398 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:30.699361 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-957019-788c577f9b-2shs7" event={"ID":"ed77f29b-ed8c-428a-a8f7-1c55892d64a8","Type":"ContainerStarted","Data":"99501923286cd113e2da0d7532e8dde4c29e085e88863f546ad571f09c2db9fd"} Oct 31 00:11:30.699591 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:30.699424 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-957019-788c577f9b-2shs7" event={"ID":"ed77f29b-ed8c-428a-a8f7-1c55892d64a8","Type":"ContainerStarted","Data":"488c18389ffd9db13a0da965bee820f1572087eff0c93eabd0fd584d8290b9c0"} Oct 31 00:11:30.699591 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:30.699453 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-957019-788c577f9b-2shs7" Oct 31 00:11:30.701211 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:30.701182 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-957019-76d548dcf-bbqz5" event={"ID":"358e61e1-8b72-4d73-b3ec-9016ebe766df","Type":"ContainerStarted","Data":"ac2ec1840a36d20ada7e088ed1f12b11a9b7fbdfd9a9ef229d46de9154756250"} Oct 31 00:11:30.701380 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:30.701225 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-957019-76d548dcf-bbqz5" event={"ID":"358e61e1-8b72-4d73-b3ec-9016ebe766df","Type":"ContainerStarted","Data":"5dd73fd4e823bab43538738cb06e02c89e1bac3dcaf347b874a023d039689873"} Oct 31 00:11:30.701380 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:30.701273 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-957019-76d548dcf-bbqz5" Oct 31 00:11:30.818114 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:30.818010 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:11:30.945618 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:30.945552 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-957019-788c577f9b-2shs7" podStartSLOduration=2.9455317770000002 podStartE2EDuration="2.945531777s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:11:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:11:30.822680341 +0000 UTC m=+1803.859011175" watchObservedRunningTime="2026-10-31 00:11:30.945531777 +0000 UTC m=+1803.981862600" Oct 31 00:11:31.197711 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:31.197677 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:11:31.729117 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:31.729062 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-957019-76d548dcf-bbqz5" podStartSLOduration=3.729043801 podStartE2EDuration="3.729043801s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:11:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:11:30.958663727 +0000 UTC m=+1803.994994549" watchObservedRunningTime="2026-10-31 00:11:31.729043801 +0000 UTC m=+1804.765374625" Oct 31 00:11:31.730528 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:31.730505 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-4a3dba-openshift-default-dc9664475-zxb5s"] Oct 31 00:11:31.735485 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:31.735467 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-4a3dba-openshift-default-dc9664475-zxb5s" Oct 31 00:11:31.746919 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:31.746892 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b1c447d-a95f-4edb-9c9d-8987eea5eaf4-istio-envoy\") pod \"e2e-shared-b-w4-4a3dba-openshift-default-dc9664475-zxb5s\" (UID: \"1b1c447d-a95f-4edb-9c9d-8987eea5eaf4\") " pod="openshift-ingress/e2e-shared-b-w4-4a3dba-openshift-default-dc9664475-zxb5s" Oct 31 00:11:31.747118 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:31.747085 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j64rz\" (UniqueName: \"kubernetes.io/projected/1b1c447d-a95f-4edb-9c9d-8987eea5eaf4-kube-api-access-j64rz\") pod \"e2e-shared-b-w4-4a3dba-openshift-default-dc9664475-zxb5s\" (UID: \"1b1c447d-a95f-4edb-9c9d-8987eea5eaf4\") " pod="openshift-ingress/e2e-shared-b-w4-4a3dba-openshift-default-dc9664475-zxb5s" Oct 31 00:11:31.747219 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:31.747142 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b1c447d-a95f-4edb-9c9d-8987eea5eaf4-workload-socket\") pod \"e2e-shared-b-w4-4a3dba-openshift-default-dc9664475-zxb5s\" (UID: \"1b1c447d-a95f-4edb-9c9d-8987eea5eaf4\") " pod="openshift-ingress/e2e-shared-b-w4-4a3dba-openshift-default-dc9664475-zxb5s" Oct 31 00:11:31.747287 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:31.747252 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b1c447d-a95f-4edb-9c9d-8987eea5eaf4-istio-podinfo\") pod \"e2e-shared-b-w4-4a3dba-openshift-default-dc9664475-zxb5s\" (UID: \"1b1c447d-a95f-4edb-9c9d-8987eea5eaf4\") " pod="openshift-ingress/e2e-shared-b-w4-4a3dba-openshift-default-dc9664475-zxb5s" Oct 31 00:11:31.747504 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:31.747484 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b1c447d-a95f-4edb-9c9d-8987eea5eaf4-istiod-ca-cert\") pod \"e2e-shared-b-w4-4a3dba-openshift-default-dc9664475-zxb5s\" (UID: \"1b1c447d-a95f-4edb-9c9d-8987eea5eaf4\") " pod="openshift-ingress/e2e-shared-b-w4-4a3dba-openshift-default-dc9664475-zxb5s" Oct 31 00:11:31.747589 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:31.747557 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b1c447d-a95f-4edb-9c9d-8987eea5eaf4-workload-certs\") pod \"e2e-shared-b-w4-4a3dba-openshift-default-dc9664475-zxb5s\" (UID: \"1b1c447d-a95f-4edb-9c9d-8987eea5eaf4\") " pod="openshift-ingress/e2e-shared-b-w4-4a3dba-openshift-default-dc9664475-zxb5s" Oct 31 00:11:31.747675 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:31.747661 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b1c447d-a95f-4edb-9c9d-8987eea5eaf4-istio-token\") pod \"e2e-shared-b-w4-4a3dba-openshift-default-dc9664475-zxb5s\" (UID: \"1b1c447d-a95f-4edb-9c9d-8987eea5eaf4\") " pod="openshift-ingress/e2e-shared-b-w4-4a3dba-openshift-default-dc9664475-zxb5s" Oct 31 00:11:31.747717 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:31.747702 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b1c447d-a95f-4edb-9c9d-8987eea5eaf4-credential-socket\") pod \"e2e-shared-b-w4-4a3dba-openshift-default-dc9664475-zxb5s\" (UID: \"1b1c447d-a95f-4edb-9c9d-8987eea5eaf4\") " pod="openshift-ingress/e2e-shared-b-w4-4a3dba-openshift-default-dc9664475-zxb5s" Oct 31 00:11:31.747757 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:31.747738 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b1c447d-a95f-4edb-9c9d-8987eea5eaf4-istio-data\") pod \"e2e-shared-b-w4-4a3dba-openshift-default-dc9664475-zxb5s\" (UID: \"1b1c447d-a95f-4edb-9c9d-8987eea5eaf4\") " pod="openshift-ingress/e2e-shared-b-w4-4a3dba-openshift-default-dc9664475-zxb5s" Oct 31 00:11:32.045405 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:31.993760 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-4a3dba-openshift-default-dc9664475-zxb5s"] Oct 31 00:11:32.045405 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:31.994060 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w4-4a3dba-openshift-default-dockercfg-cmjd6\"" Oct 31 00:11:32.045405 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:32.003758 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:11:32.065463 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:32.062612 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b1c447d-a95f-4edb-9c9d-8987eea5eaf4-istiod-ca-cert\") pod \"e2e-shared-b-w4-4a3dba-openshift-default-dc9664475-zxb5s\" (UID: \"1b1c447d-a95f-4edb-9c9d-8987eea5eaf4\") " pod="openshift-ingress/e2e-shared-b-w4-4a3dba-openshift-default-dc9664475-zxb5s" Oct 31 00:11:32.065463 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:32.062717 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b1c447d-a95f-4edb-9c9d-8987eea5eaf4-workload-certs\") pod \"e2e-shared-b-w4-4a3dba-openshift-default-dc9664475-zxb5s\" (UID: \"1b1c447d-a95f-4edb-9c9d-8987eea5eaf4\") " pod="openshift-ingress/e2e-shared-b-w4-4a3dba-openshift-default-dc9664475-zxb5s" Oct 31 00:11:32.065463 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:32.062927 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b1c447d-a95f-4edb-9c9d-8987eea5eaf4-istio-token\") pod \"e2e-shared-b-w4-4a3dba-openshift-default-dc9664475-zxb5s\" (UID: \"1b1c447d-a95f-4edb-9c9d-8987eea5eaf4\") " pod="openshift-ingress/e2e-shared-b-w4-4a3dba-openshift-default-dc9664475-zxb5s" Oct 31 00:11:32.065463 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:32.062972 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b1c447d-a95f-4edb-9c9d-8987eea5eaf4-credential-socket\") pod \"e2e-shared-b-w4-4a3dba-openshift-default-dc9664475-zxb5s\" (UID: \"1b1c447d-a95f-4edb-9c9d-8987eea5eaf4\") " pod="openshift-ingress/e2e-shared-b-w4-4a3dba-openshift-default-dc9664475-zxb5s" Oct 31 00:11:32.065463 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:32.063049 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b1c447d-a95f-4edb-9c9d-8987eea5eaf4-istio-data\") pod \"e2e-shared-b-w4-4a3dba-openshift-default-dc9664475-zxb5s\" (UID: \"1b1c447d-a95f-4edb-9c9d-8987eea5eaf4\") " pod="openshift-ingress/e2e-shared-b-w4-4a3dba-openshift-default-dc9664475-zxb5s" Oct 31 00:11:32.065463 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:32.063105 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b1c447d-a95f-4edb-9c9d-8987eea5eaf4-istio-envoy\") pod \"e2e-shared-b-w4-4a3dba-openshift-default-dc9664475-zxb5s\" (UID: \"1b1c447d-a95f-4edb-9c9d-8987eea5eaf4\") " pod="openshift-ingress/e2e-shared-b-w4-4a3dba-openshift-default-dc9664475-zxb5s" Oct 31 00:11:32.065463 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:32.063133 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j64rz\" (UniqueName: \"kubernetes.io/projected/1b1c447d-a95f-4edb-9c9d-8987eea5eaf4-kube-api-access-j64rz\") pod \"e2e-shared-b-w4-4a3dba-openshift-default-dc9664475-zxb5s\" (UID: \"1b1c447d-a95f-4edb-9c9d-8987eea5eaf4\") " pod="openshift-ingress/e2e-shared-b-w4-4a3dba-openshift-default-dc9664475-zxb5s" Oct 31 00:11:32.065463 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:32.063180 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b1c447d-a95f-4edb-9c9d-8987eea5eaf4-workload-socket\") pod \"e2e-shared-b-w4-4a3dba-openshift-default-dc9664475-zxb5s\" (UID: \"1b1c447d-a95f-4edb-9c9d-8987eea5eaf4\") " pod="openshift-ingress/e2e-shared-b-w4-4a3dba-openshift-default-dc9664475-zxb5s" Oct 31 00:11:32.065463 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:32.063283 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b1c447d-a95f-4edb-9c9d-8987eea5eaf4-istio-podinfo\") pod \"e2e-shared-b-w4-4a3dba-openshift-default-dc9664475-zxb5s\" (UID: \"1b1c447d-a95f-4edb-9c9d-8987eea5eaf4\") " pod="openshift-ingress/e2e-shared-b-w4-4a3dba-openshift-default-dc9664475-zxb5s" Oct 31 00:11:32.065463 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:32.063903 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b1c447d-a95f-4edb-9c9d-8987eea5eaf4-credential-socket\") pod \"e2e-shared-b-w4-4a3dba-openshift-default-dc9664475-zxb5s\" (UID: \"1b1c447d-a95f-4edb-9c9d-8987eea5eaf4\") " pod="openshift-ingress/e2e-shared-b-w4-4a3dba-openshift-default-dc9664475-zxb5s" Oct 31 00:11:32.065463 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:32.064657 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b1c447d-a95f-4edb-9c9d-8987eea5eaf4-istiod-ca-cert\") pod \"e2e-shared-b-w4-4a3dba-openshift-default-dc9664475-zxb5s\" (UID: \"1b1c447d-a95f-4edb-9c9d-8987eea5eaf4\") " pod="openshift-ingress/e2e-shared-b-w4-4a3dba-openshift-default-dc9664475-zxb5s" Oct 31 00:11:32.065463 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:32.064927 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b1c447d-a95f-4edb-9c9d-8987eea5eaf4-workload-certs\") pod \"e2e-shared-b-w4-4a3dba-openshift-default-dc9664475-zxb5s\" (UID: \"1b1c447d-a95f-4edb-9c9d-8987eea5eaf4\") " pod="openshift-ingress/e2e-shared-b-w4-4a3dba-openshift-default-dc9664475-zxb5s" Oct 31 00:11:32.066266 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:32.065904 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b1c447d-a95f-4edb-9c9d-8987eea5eaf4-istio-data\") pod \"e2e-shared-b-w4-4a3dba-openshift-default-dc9664475-zxb5s\" (UID: \"1b1c447d-a95f-4edb-9c9d-8987eea5eaf4\") " pod="openshift-ingress/e2e-shared-b-w4-4a3dba-openshift-default-dc9664475-zxb5s" Oct 31 00:11:32.066498 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:32.066471 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b1c447d-a95f-4edb-9c9d-8987eea5eaf4-workload-socket\") pod \"e2e-shared-b-w4-4a3dba-openshift-default-dc9664475-zxb5s\" (UID: \"1b1c447d-a95f-4edb-9c9d-8987eea5eaf4\") " pod="openshift-ingress/e2e-shared-b-w4-4a3dba-openshift-default-dc9664475-zxb5s" Oct 31 00:11:32.067389 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:32.067359 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b1c447d-a95f-4edb-9c9d-8987eea5eaf4-istio-podinfo\") pod \"e2e-shared-b-w4-4a3dba-openshift-default-dc9664475-zxb5s\" (UID: \"1b1c447d-a95f-4edb-9c9d-8987eea5eaf4\") " pod="openshift-ingress/e2e-shared-b-w4-4a3dba-openshift-default-dc9664475-zxb5s" Oct 31 00:11:32.068717 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:32.068694 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b1c447d-a95f-4edb-9c9d-8987eea5eaf4-istio-envoy\") pod \"e2e-shared-b-w4-4a3dba-openshift-default-dc9664475-zxb5s\" (UID: \"1b1c447d-a95f-4edb-9c9d-8987eea5eaf4\") " pod="openshift-ingress/e2e-shared-b-w4-4a3dba-openshift-default-dc9664475-zxb5s" Oct 31 00:11:32.149354 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:32.149040 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b1c447d-a95f-4edb-9c9d-8987eea5eaf4-istio-token\") pod \"e2e-shared-b-w4-4a3dba-openshift-default-dc9664475-zxb5s\" (UID: \"1b1c447d-a95f-4edb-9c9d-8987eea5eaf4\") " pod="openshift-ingress/e2e-shared-b-w4-4a3dba-openshift-default-dc9664475-zxb5s" Oct 31 00:11:32.168102 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:32.168069 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j64rz\" (UniqueName: \"kubernetes.io/projected/1b1c447d-a95f-4edb-9c9d-8987eea5eaf4-kube-api-access-j64rz\") pod \"e2e-shared-b-w4-4a3dba-openshift-default-dc9664475-zxb5s\" (UID: \"1b1c447d-a95f-4edb-9c9d-8987eea5eaf4\") " pod="openshift-ingress/e2e-shared-b-w4-4a3dba-openshift-default-dc9664475-zxb5s" Oct 31 00:11:32.269359 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:32.265898 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:11:32.336261 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:32.335012 4927 scope.go:122] "RemoveContainer" containerID="4db49853d5571d577dd419949769b96ec9ce49fb1cc21f1eb223367eb3987675" Oct 31 00:11:32.353165 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:32.353082 4927 scope.go:122] "RemoveContainer" containerID="e81e24db8cd912deac0573ac2b43848a00c994307a2cb1d92d30ffc0f9d80dd3" Oct 31 00:11:32.357836 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:32.357643 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-4a3dba-openshift-default-dc9664475-zxb5s" Oct 31 00:11:32.364428 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:32.364326 4927 scope.go:122] "RemoveContainer" containerID="61d254c0a0eb124fd07bfad8fe7dfc6b785f25c5411d961cdf3f23e322e6b7db" Oct 31 00:11:32.815127 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:32.815102 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-4a3dba-openshift-default-dc9664475-zxb5s"] Oct 31 00:11:32.817827 ip-10-0-141-180 kubenswrapper[4927]: W1031 00:11:32.817803 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b1c447d_a95f_4edb_9c9d_8987eea5eaf4.slice/crio-422ea89a0d08389286efbc4d7e02c55e7ade5829834719ea8e7c68849e422dff WatchSource:0}: Error finding container 422ea89a0d08389286efbc4d7e02c55e7ade5829834719ea8e7c68849e422dff: Status 404 returned error can't find the container with id 422ea89a0d08389286efbc4d7e02c55e7ade5829834719ea8e7c68849e422dff Oct 31 00:11:32.821461 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:32.821419 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:11:32.821564 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:32.821521 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:11:32.821675 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:32.821580 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:11:33.735514 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:33.735472 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-4a3dba-openshift-default-dc9664475-zxb5s" event={"ID":"1b1c447d-a95f-4edb-9c9d-8987eea5eaf4","Type":"ContainerStarted","Data":"d750bd6d9be7f3a0cecd96781cede0a31a4438375a563f22bd141188270130da"} Oct 31 00:11:33.735899 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:33.735534 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-4a3dba-openshift-default-dc9664475-zxb5s" event={"ID":"1b1c447d-a95f-4edb-9c9d-8987eea5eaf4","Type":"ContainerStarted","Data":"422ea89a0d08389286efbc4d7e02c55e7ade5829834719ea8e7c68849e422dff"} Oct 31 00:11:33.918843 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:33.918774 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-4a3dba-openshift-default-dc9664475-zxb5s" podStartSLOduration=2.918754706 podStartE2EDuration="2.918754706s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:11:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:11:33.913893704 +0000 UTC m=+1806.950224525" watchObservedRunningTime="2026-10-31 00:11:33.918754706 +0000 UTC m=+1806.955085529" Oct 31 00:11:34.200647 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:34.200610 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:11:34.358633 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:34.358599 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-4a3dba-openshift-default-dc9664475-zxb5s" Oct 31 00:11:34.365048 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:34.365019 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-4a3dba-openshift-default-dc9664475-zxb5s" Oct 31 00:11:34.563312 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:34.563236 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-03c703-68ccf588f9-957fw" Oct 31 00:11:34.563312 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:34.563304 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-03c703-587b89bbdd-vzfkm" Oct 31 00:11:34.747755 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:34.747717 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-4a3dba-openshift-default-dc9664475-zxb5s" Oct 31 00:11:34.749096 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:34.749072 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-4a3dba-openshift-default-dc9664475-zxb5s" Oct 31 00:11:36.798614 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:36.798579 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:11:36.959097 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:36.959057 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:11:39.124180 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:39.124145 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:11:39.821034 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:39.820995 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:11:40.053615 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:40.053574 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:11:41.749355 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:41.747394 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-957019-76d548dcf-bbqz5" Oct 31 00:11:41.778358 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:41.750106 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-957019-788c577f9b-2shs7" Oct 31 00:11:42.110485 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:42.099947 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:11:42.132253 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:42.132212 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a380d1e-c9kqq"] Oct 31 00:11:42.204189 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:42.204133 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a380d1e-c9kqq"] Oct 31 00:11:42.410606 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:42.410568 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:11:43.088037 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:43.087998 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c127cf0-aa5c-4d3b-bdb5-704bf0931c92" path="/var/lib/kubelet/pods/1c127cf0-aa5c-4d3b-bdb5-704bf0931c92/volumes" Oct 31 00:11:44.092735 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:44.092701 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:11:49.115863 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:49.115781 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:11:49.681449 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:11:49.681412 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sqv69" containerName="extractor" Oct 31 00:11:50.074839 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:50.074767 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-j2j46"] Oct 31 00:11:50.075049 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:50.075025 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-j2j46" podUID="a0f800b8-0d6f-484a-bfa8-e71b15ce18e5" containerName="istio-proxy" containerID="cri-o://5829cc9dd4c653da767410b7e21bd6de3da2f4239d4b4bc49bc4b779f2871f79" gracePeriod=30 Oct 31 00:11:50.894486 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:50.894440 4927 patch_prober.go:29] interesting pod/e2e-shared-a-w3-ab0ac9-openshift-default-54c6546bdf-lxwwq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.197:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:11:50.894988 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:50.894555 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w3-ab0ac9-openshift-default-54c6546bdf-lxwwq" podUID="3b669d7f-c767-4c4e-a524-6e774fc69034" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.197:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:11:52.928251 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:52.928208 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:11:53.281155 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:53.281045 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:11:53.813566 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:53.813479 4927 patch_prober.go:29] interesting pod/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-j2j46 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 00:11:53.813566 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:53.813548 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-j2j46" podUID="a0f800b8-0d6f-484a-bfa8-e71b15ce18e5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:11:53.919785 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:53.919745 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:11:54.028805 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:54.028755 4927 generic.go:356] "Generic (PLEG): container finished" podID="3b669d7f-c767-4c4e-a524-6e774fc69034" containerID="e2cdeea856504f4f745d34fae7ed87897794b11dec004edbaf555fde8f4bb24e" exitCode=0 Oct 31 00:11:54.029270 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:54.028820 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-ab0ac9-openshift-default-54c6546bdf-lxwwq" event={"ID":"3b669d7f-c767-4c4e-a524-6e774fc69034","Type":"ContainerDied","Data":"e2cdeea856504f4f745d34fae7ed87897794b11dec004edbaf555fde8f4bb24e"} Oct 31 00:11:54.029317 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:54.029285 4927 scope.go:122] "RemoveContainer" containerID="e2cdeea856504f4f745d34fae7ed87897794b11dec004edbaf555fde8f4bb24e" Oct 31 00:11:54.031931 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:54.031896 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:11:54.032019 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:54.031990 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:11:54.032068 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:54.032051 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:11:55.050589 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:55.050545 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-ab0ac9-openshift-default-54c6546bdf-lxwwq" event={"ID":"3b669d7f-c767-4c4e-a524-6e774fc69034","Type":"ContainerStarted","Data":"e8dc89e23fffca41c6652fcb3bfbab07c7fbf700202efaf16e98f640624ae2ba"} Oct 31 00:11:55.057266 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:55.057238 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-ab0ac9-openshift-default-54c6546bdf-lxwwq" Oct 31 00:11:55.057475 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:55.057446 4927 patch_prober.go:29] interesting pod/e2e-shared-a-w3-ab0ac9-openshift-default-54c6546bdf-lxwwq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused" start-of-body= Oct 31 00:11:55.057565 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:55.057510 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-ab0ac9-openshift-default-54c6546bdf-lxwwq" podUID="3b669d7f-c767-4c4e-a524-6e774fc69034" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused" Oct 31 00:11:55.457013 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:55.456988 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-j2j46" Oct 31 00:11:55.498347 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:55.498303 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0f800b8-0d6f-484a-bfa8-e71b15ce18e5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0f800b8-0d6f-484a-bfa8-e71b15ce18e5-workload-certs\") pod \"a0f800b8-0d6f-484a-bfa8-e71b15ce18e5\" (UID: \"a0f800b8-0d6f-484a-bfa8-e71b15ce18e5\") " Oct 31 00:11:55.498511 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:55.498409 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0f800b8-0d6f-484a-bfa8-e71b15ce18e5-kube-api-access-jtdzk\" (UniqueName: \"kubernetes.io/projected/a0f800b8-0d6f-484a-bfa8-e71b15ce18e5-kube-api-access-jtdzk\") pod \"a0f800b8-0d6f-484a-bfa8-e71b15ce18e5\" (UID: \"a0f800b8-0d6f-484a-bfa8-e71b15ce18e5\") " Oct 31 00:11:55.498511 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:55.498447 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a0f800b8-0d6f-484a-bfa8-e71b15ce18e5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0f800b8-0d6f-484a-bfa8-e71b15ce18e5-istiod-ca-cert\") pod \"a0f800b8-0d6f-484a-bfa8-e71b15ce18e5\" (UID: \"a0f800b8-0d6f-484a-bfa8-e71b15ce18e5\") " Oct 31 00:11:55.498511 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:55.498469 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0f800b8-0d6f-484a-bfa8-e71b15ce18e5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0f800b8-0d6f-484a-bfa8-e71b15ce18e5-istio-data\") pod \"a0f800b8-0d6f-484a-bfa8-e71b15ce18e5\" (UID: \"a0f800b8-0d6f-484a-bfa8-e71b15ce18e5\") " Oct 31 00:11:55.498692 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:55.498556 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a0f800b8-0d6f-484a-bfa8-e71b15ce18e5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0f800b8-0d6f-484a-bfa8-e71b15ce18e5-istio-podinfo\") pod \"a0f800b8-0d6f-484a-bfa8-e71b15ce18e5\" (UID: \"a0f800b8-0d6f-484a-bfa8-e71b15ce18e5\") " Oct 31 00:11:55.498692 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:55.498575 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0f800b8-0d6f-484a-bfa8-e71b15ce18e5-workload-certs" pod "a0f800b8-0d6f-484a-bfa8-e71b15ce18e5" (UID: "a0f800b8-0d6f-484a-bfa8-e71b15ce18e5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:11:55.498808 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:55.498730 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0f800b8-0d6f-484a-bfa8-e71b15ce18e5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0f800b8-0d6f-484a-bfa8-e71b15ce18e5-credential-socket\") pod \"a0f800b8-0d6f-484a-bfa8-e71b15ce18e5\" (UID: \"a0f800b8-0d6f-484a-bfa8-e71b15ce18e5\") " Oct 31 00:11:55.498868 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:55.498820 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0f800b8-0d6f-484a-bfa8-e71b15ce18e5-istio-data" pod "a0f800b8-0d6f-484a-bfa8-e71b15ce18e5" (UID: "a0f800b8-0d6f-484a-bfa8-e71b15ce18e5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:11:55.498868 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:55.498832 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0f800b8-0d6f-484a-bfa8-e71b15ce18e5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0f800b8-0d6f-484a-bfa8-e71b15ce18e5-istio-envoy\") pod \"a0f800b8-0d6f-484a-bfa8-e71b15ce18e5\" (UID: \"a0f800b8-0d6f-484a-bfa8-e71b15ce18e5\") " Oct 31 00:11:55.498868 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:55.498860 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0f800b8-0d6f-484a-bfa8-e71b15ce18e5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0f800b8-0d6f-484a-bfa8-e71b15ce18e5-workload-socket\") pod \"a0f800b8-0d6f-484a-bfa8-e71b15ce18e5\" (UID: \"a0f800b8-0d6f-484a-bfa8-e71b15ce18e5\") " Oct 31 00:11:55.499024 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:55.498887 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0f800b8-0d6f-484a-bfa8-e71b15ce18e5-istiod-ca-cert" pod "a0f800b8-0d6f-484a-bfa8-e71b15ce18e5" (UID: "a0f800b8-0d6f-484a-bfa8-e71b15ce18e5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:11:55.499024 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:55.498904 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0f800b8-0d6f-484a-bfa8-e71b15ce18e5-istio-token\" (UniqueName: \"kubernetes.io/projected/a0f800b8-0d6f-484a-bfa8-e71b15ce18e5-istio-token\") pod \"a0f800b8-0d6f-484a-bfa8-e71b15ce18e5\" (UID: \"a0f800b8-0d6f-484a-bfa8-e71b15ce18e5\") " Oct 31 00:11:55.499094 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:55.499076 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0f800b8-0d6f-484a-bfa8-e71b15ce18e5-workload-socket" pod "a0f800b8-0d6f-484a-bfa8-e71b15ce18e5" (UID: "a0f800b8-0d6f-484a-bfa8-e71b15ce18e5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:11:55.499224 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:55.499201 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0f800b8-0d6f-484a-bfa8-e71b15ce18e5-credential-socket" pod "a0f800b8-0d6f-484a-bfa8-e71b15ce18e5" (UID: "a0f800b8-0d6f-484a-bfa8-e71b15ce18e5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:11:55.499660 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:55.499635 4927 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0f800b8-0d6f-484a-bfa8-e71b15ce18e5-workload-certs\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:11:55.499785 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:55.499662 4927 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0f800b8-0d6f-484a-bfa8-e71b15ce18e5-istiod-ca-cert\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:11:55.499785 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:55.499677 4927 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0f800b8-0d6f-484a-bfa8-e71b15ce18e5-istio-data\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:11:55.499785 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:55.499692 4927 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0f800b8-0d6f-484a-bfa8-e71b15ce18e5-credential-socket\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:11:55.499785 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:55.499706 4927 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0f800b8-0d6f-484a-bfa8-e71b15ce18e5-workload-socket\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:11:55.501043 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:55.501016 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0f800b8-0d6f-484a-bfa8-e71b15ce18e5-istio-envoy" pod "a0f800b8-0d6f-484a-bfa8-e71b15ce18e5" (UID: "a0f800b8-0d6f-484a-bfa8-e71b15ce18e5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:11:55.501168 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:55.501059 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a0f800b8-0d6f-484a-bfa8-e71b15ce18e5-istio-podinfo" pod "a0f800b8-0d6f-484a-bfa8-e71b15ce18e5" (UID: "a0f800b8-0d6f-484a-bfa8-e71b15ce18e5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:11:55.501226 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:55.501168 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0f800b8-0d6f-484a-bfa8-e71b15ce18e5-istio-token" pod "a0f800b8-0d6f-484a-bfa8-e71b15ce18e5" (UID: "a0f800b8-0d6f-484a-bfa8-e71b15ce18e5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:11:55.501226 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:55.501195 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0f800b8-0d6f-484a-bfa8-e71b15ce18e5-kube-api-access-jtdzk" pod "a0f800b8-0d6f-484a-bfa8-e71b15ce18e5" (UID: "a0f800b8-0d6f-484a-bfa8-e71b15ce18e5"). InnerVolumeSpecName "kube-api-access-jtdzk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:11:55.600819 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:55.600785 4927 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0f800b8-0d6f-484a-bfa8-e71b15ce18e5-istio-envoy\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:11:55.600819 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:55.600816 4927 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0f800b8-0d6f-484a-bfa8-e71b15ce18e5-istio-token\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:11:55.601034 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:55.600829 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jtdzk\" (UniqueName: \"kubernetes.io/projected/a0f800b8-0d6f-484a-bfa8-e71b15ce18e5-kube-api-access-jtdzk\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:11:55.601034 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:55.600839 4927 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0f800b8-0d6f-484a-bfa8-e71b15ce18e5-istio-podinfo\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:11:56.058282 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:56.058240 4927 patch_prober.go:29] interesting pod/e2e-shared-a-w3-ab0ac9-openshift-default-54c6546bdf-lxwwq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused" start-of-body= Oct 31 00:11:56.058858 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:56.058303 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-ab0ac9-openshift-default-54c6546bdf-lxwwq" podUID="3b669d7f-c767-4c4e-a524-6e774fc69034" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused" Oct 31 00:11:56.062737 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:56.062709 4927 generic.go:356] "Generic (PLEG): container finished" podID="a0f800b8-0d6f-484a-bfa8-e71b15ce18e5" containerID="5829cc9dd4c653da767410b7e21bd6de3da2f4239d4b4bc49bc4b779f2871f79" exitCode=0 Oct 31 00:11:56.062898 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:56.062779 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-j2j46" Oct 31 00:11:56.062898 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:56.062804 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-j2j46" event={"ID":"a0f800b8-0d6f-484a-bfa8-e71b15ce18e5","Type":"ContainerDied","Data":"5829cc9dd4c653da767410b7e21bd6de3da2f4239d4b4bc49bc4b779f2871f79"} Oct 31 00:11:56.063010 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:56.062902 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-j2j46" event={"ID":"a0f800b8-0d6f-484a-bfa8-e71b15ce18e5","Type":"ContainerDied","Data":"30a7626bc54dd27455f33b97c63d4bc335b342a47367a0999567799083941a75"} Oct 31 00:11:56.063010 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:56.062936 4927 scope.go:122] "RemoveContainer" containerID="5829cc9dd4c653da767410b7e21bd6de3da2f4239d4b4bc49bc4b779f2871f79" Oct 31 00:11:56.076155 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:56.076138 4927 scope.go:122] "RemoveContainer" containerID="5829cc9dd4c653da767410b7e21bd6de3da2f4239d4b4bc49bc4b779f2871f79" Oct 31 00:11:56.076459 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:11:56.076437 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5829cc9dd4c653da767410b7e21bd6de3da2f4239d4b4bc49bc4b779f2871f79\": container with ID starting with 5829cc9dd4c653da767410b7e21bd6de3da2f4239d4b4bc49bc4b779f2871f79 not found: ID does not exist" containerID="5829cc9dd4c653da767410b7e21bd6de3da2f4239d4b4bc49bc4b779f2871f79" Oct 31 00:11:56.076544 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:56.076467 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5829cc9dd4c653da767410b7e21bd6de3da2f4239d4b4bc49bc4b779f2871f79"} err="failed to get container status \"5829cc9dd4c653da767410b7e21bd6de3da2f4239d4b4bc49bc4b779f2871f79\": rpc error: code = NotFound desc = could not find container \"5829cc9dd4c653da767410b7e21bd6de3da2f4239d4b4bc49bc4b779f2871f79\": container with ID starting with 5829cc9dd4c653da767410b7e21bd6de3da2f4239d4b4bc49bc4b779f2871f79 not found: ID does not exist" Oct 31 00:11:56.248743 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:56.248706 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-j2j46"] Oct 31 00:11:56.254132 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:56.254103 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-j2j46"] Oct 31 00:11:57.058423 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:57.058386 4927 patch_prober.go:29] interesting pod/e2e-shared-a-w3-ab0ac9-openshift-default-54c6546bdf-lxwwq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused" start-of-body= Oct 31 00:11:57.058862 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:57.058445 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-ab0ac9-openshift-default-54c6546bdf-lxwwq" podUID="3b669d7f-c767-4c4e-a524-6e774fc69034" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused" Oct 31 00:11:57.089253 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:57.089202 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a0f800b8-0d6f-484a-bfa8-e71b15ce18e5" path="/var/lib/kubelet/pods/a0f800b8-0d6f-484a-bfa8-e71b15ce18e5/volumes" Oct 31 00:11:58.058135 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:58.058093 4927 patch_prober.go:29] interesting pod/e2e-shared-a-w3-ab0ac9-openshift-default-54c6546bdf-lxwwq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused" start-of-body= Oct 31 00:11:58.058345 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:58.058155 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-ab0ac9-openshift-default-54c6546bdf-lxwwq" podUID="3b669d7f-c767-4c4e-a524-6e774fc69034" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused" Oct 31 00:11:58.479047 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:58.479009 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:11:59.058515 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:59.058479 4927 patch_prober.go:29] interesting pod/e2e-shared-a-w3-ab0ac9-openshift-default-54c6546bdf-lxwwq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused" start-of-body= Oct 31 00:11:59.058743 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:11:59.058542 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-ab0ac9-openshift-default-54c6546bdf-lxwwq" podUID="3b669d7f-c767-4c4e-a524-6e774fc69034" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused" Oct 31 00:12:00.058171 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:12:00.058128 4927 patch_prober.go:29] interesting pod/e2e-shared-a-w3-ab0ac9-openshift-default-54c6546bdf-lxwwq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused" start-of-body= Oct 31 00:12:00.058620 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:12:00.058200 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-ab0ac9-openshift-default-54c6546bdf-lxwwq" podUID="3b669d7f-c767-4c4e-a524-6e774fc69034" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused" Oct 31 00:12:01.062442 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:12:01.062409 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-ab0ac9-openshift-default-54c6546bdf-lxwwq" Oct 31 00:12:01.062936 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:12:01.062803 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-ab0ac9-openshift-default-54c6546bdf-lxwwq" Oct 31 00:12:01.063421 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:12:01.063400 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-ab0ac9-openshift-default-54c6546bdf-lxwwq" Oct 31 00:12:03.164593 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:12:03.164558 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:12:07.778398 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:12:07.778298 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-39e3a8-d9cc779c5-zxtvc"] Oct 31 00:12:07.779517 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:12:07.779483 4927 cpu_manager.go:414] "containerMap: removing container" podUID="a0f800b8-0d6f-484a-bfa8-e71b15ce18e5" containerName="istio-proxy" Oct 31 00:12:07.779517 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:12:07.779508 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0f800b8-0d6f-484a-bfa8-e71b15ce18e5" containerName="istio-proxy" Oct 31 00:12:07.779772 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:12:07.779755 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0f800b8-0d6f-484a-bfa8-e71b15ce18e5" containerName="istio-proxy" Oct 31 00:12:07.783581 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:12:07.783561 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-39e3a8-d9cc779c5-zxtvc" Oct 31 00:12:07.805668 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:12:07.805633 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/334b65b1-c1d7-48a8-8837-6f0268fbef2a-metrics-certs\") pod \"maas-api-e2e-api-b-39e3a8-d9cc779c5-zxtvc\" (UID: \"334b65b1-c1d7-48a8-8837-6f0268fbef2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-39e3a8-d9cc779c5-zxtvc" Oct 31 00:12:07.805839 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:12:07.805715 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7kpjb\" (UniqueName: \"kubernetes.io/projected/334b65b1-c1d7-48a8-8837-6f0268fbef2a-kube-api-access-7kpjb\") pod \"maas-api-e2e-api-b-39e3a8-d9cc779c5-zxtvc\" (UID: \"334b65b1-c1d7-48a8-8837-6f0268fbef2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-39e3a8-d9cc779c5-zxtvc" Oct 31 00:12:07.805931 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:12:07.805906 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/334b65b1-c1d7-48a8-8837-6f0268fbef2a-maas-api-tls\") pod \"maas-api-e2e-api-b-39e3a8-d9cc779c5-zxtvc\" (UID: \"334b65b1-c1d7-48a8-8837-6f0268fbef2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-39e3a8-d9cc779c5-zxtvc" Oct 31 00:12:07.806075 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:12:07.805944 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/334b65b1-c1d7-48a8-8837-6f0268fbef2a-openshift-service-ca\") pod \"maas-api-e2e-api-b-39e3a8-d9cc779c5-zxtvc\" (UID: \"334b65b1-c1d7-48a8-8837-6f0268fbef2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-39e3a8-d9cc779c5-zxtvc" Oct 31 00:12:07.841392 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:12:07.841360 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-39e3a8-d9cc779c5-zxtvc"] Oct 31 00:12:07.886717 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:12:07.886674 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-39e3a8-serving-cert\"" Oct 31 00:12:07.906910 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:12:07.906874 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/334b65b1-c1d7-48a8-8837-6f0268fbef2a-metrics-certs\") pod \"maas-api-e2e-api-b-39e3a8-d9cc779c5-zxtvc\" (UID: \"334b65b1-c1d7-48a8-8837-6f0268fbef2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-39e3a8-d9cc779c5-zxtvc" Oct 31 00:12:07.907082 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:12:07.906990 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7kpjb\" (UniqueName: \"kubernetes.io/projected/334b65b1-c1d7-48a8-8837-6f0268fbef2a-kube-api-access-7kpjb\") pod \"maas-api-e2e-api-b-39e3a8-d9cc779c5-zxtvc\" (UID: \"334b65b1-c1d7-48a8-8837-6f0268fbef2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-39e3a8-d9cc779c5-zxtvc" Oct 31 00:12:07.907167 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:12:07.907150 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/334b65b1-c1d7-48a8-8837-6f0268fbef2a-maas-api-tls\") pod \"maas-api-e2e-api-b-39e3a8-d9cc779c5-zxtvc\" (UID: \"334b65b1-c1d7-48a8-8837-6f0268fbef2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-39e3a8-d9cc779c5-zxtvc" Oct 31 00:12:07.907229 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:12:07.907182 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/334b65b1-c1d7-48a8-8837-6f0268fbef2a-openshift-service-ca\") pod \"maas-api-e2e-api-b-39e3a8-d9cc779c5-zxtvc\" (UID: \"334b65b1-c1d7-48a8-8837-6f0268fbef2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-39e3a8-d9cc779c5-zxtvc" Oct 31 00:12:07.907303 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:12:07.907279 4927 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-39e3a8-serving-cert: secret "maas-api-e2e-api-b-39e3a8-serving-cert" not found Oct 31 00:12:07.907408 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:12:07.907394 4927 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/334b65b1-c1d7-48a8-8837-6f0268fbef2a-maas-api-tls podName:334b65b1-c1d7-48a8-8837-6f0268fbef2a nodeName:}" failed. No retries permitted until 2026-10-31 00:12:08.407375527 +0000 UTC m=+1841.443706331 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/334b65b1-c1d7-48a8-8837-6f0268fbef2a-maas-api-tls") pod "maas-api-e2e-api-b-39e3a8-d9cc779c5-zxtvc" (UID: "334b65b1-c1d7-48a8-8837-6f0268fbef2a") : secret "maas-api-e2e-api-b-39e3a8-serving-cert" not found Oct 31 00:12:07.907860 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:12:07.907837 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/334b65b1-c1d7-48a8-8837-6f0268fbef2a-openshift-service-ca\") pod \"maas-api-e2e-api-b-39e3a8-d9cc779c5-zxtvc\" (UID: \"334b65b1-c1d7-48a8-8837-6f0268fbef2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-39e3a8-d9cc779c5-zxtvc" Oct 31 00:12:07.909308 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:12:07.909289 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/334b65b1-c1d7-48a8-8837-6f0268fbef2a-metrics-certs\") pod \"maas-api-e2e-api-b-39e3a8-d9cc779c5-zxtvc\" (UID: \"334b65b1-c1d7-48a8-8837-6f0268fbef2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-39e3a8-d9cc779c5-zxtvc" Oct 31 00:12:07.995947 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:12:07.995919 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7kpjb\" (UniqueName: \"kubernetes.io/projected/334b65b1-c1d7-48a8-8837-6f0268fbef2a-kube-api-access-7kpjb\") pod \"maas-api-e2e-api-b-39e3a8-d9cc779c5-zxtvc\" (UID: \"334b65b1-c1d7-48a8-8837-6f0268fbef2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-39e3a8-d9cc779c5-zxtvc" Oct 31 00:12:08.012399 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:12:08.012368 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:12:08.167209 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:12:08.166282 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:12:08.315269 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:12:08.315232 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:12:08.419866 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:12:08.419775 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/334b65b1-c1d7-48a8-8837-6f0268fbef2a-maas-api-tls\") pod \"maas-api-e2e-api-b-39e3a8-d9cc779c5-zxtvc\" (UID: \"334b65b1-c1d7-48a8-8837-6f0268fbef2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-39e3a8-d9cc779c5-zxtvc" Oct 31 00:12:08.422315 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:12:08.422285 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/334b65b1-c1d7-48a8-8837-6f0268fbef2a-maas-api-tls\") pod \"maas-api-e2e-api-b-39e3a8-d9cc779c5-zxtvc\" (UID: \"334b65b1-c1d7-48a8-8837-6f0268fbef2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-39e3a8-d9cc779c5-zxtvc" Oct 31 00:12:08.699396 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:12:08.699272 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-39e3a8-d9cc779c5-zxtvc" Oct 31 00:12:09.112148 ip-10-0-141-180 kubenswrapper[4927]: W1031 00:12:09.112111 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod334b65b1_c1d7_48a8_8837_6f0268fbef2a.slice/crio-1d3b6365ee107d8c4cd88624c629dd879b0e7e7cf901fc4b813a789106cfa4c1 WatchSource:0}: Error finding container 1d3b6365ee107d8c4cd88624c629dd879b0e7e7cf901fc4b813a789106cfa4c1: Status 404 returned error can't find the container with id 1d3b6365ee107d8c4cd88624c629dd879b0e7e7cf901fc4b813a789106cfa4c1 Oct 31 00:12:09.122306 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:12:09.122279 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-39e3a8-d9cc779c5-zxtvc"] Oct 31 00:12:09.187405 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:12:09.187327 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-39e3a8-d9cc779c5-zxtvc" event={"ID":"334b65b1-c1d7-48a8-8837-6f0268fbef2a","Type":"ContainerStarted","Data":"1d3b6365ee107d8c4cd88624c629dd879b0e7e7cf901fc4b813a789106cfa4c1"} Oct 31 00:12:10.200626 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:12:10.200588 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-39e3a8-d9cc779c5-zxtvc" event={"ID":"334b65b1-c1d7-48a8-8837-6f0268fbef2a","Type":"ContainerStarted","Data":"184cfba3d3e5360b6dd2bf23e9e2050a30009de770a4de69c2464924731ca392"} Oct 31 00:12:10.201035 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:12:10.200659 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-39e3a8-d9cc779c5-zxtvc" Oct 31 00:12:10.326263 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:12:10.326158 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-39e3a8-d9cc779c5-zxtvc" podStartSLOduration=2.380573888 podStartE2EDuration="3.326142294s" totalImagesPullingTime="945.568406ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:12:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:12:10.324564462 +0000 UTC m=+1843.360895309" watchObservedRunningTime="2026-10-31 00:12:10.326142294 +0000 UTC m=+1843.362473140" Oct 31 00:12:12.765792 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:12:12.765745 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:12:14.365618 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:12:14.365574 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-a251fdfb-6fb6bf7d6b-xmmgr"] Oct 31 00:12:14.464502 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:12:14.464465 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-a251fdfb-6f8487684c-6qzl9"] Oct 31 00:12:14.695028 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:12:14.694993 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:12:14.848942 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:12:14.848908 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:12:16.216857 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:12:16.216825 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-39e3a8-d9cc779c5-zxtvc" Oct 31 00:12:17.352858 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:12:17.352825 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:12:19.366426 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:12:19.366371 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-a251fdfb-6fb6bf7d6b-xmmgr" podUID="c690fe76-022e-44e3-97f3-485dacb13fb9" containerName="payload-processing" containerID="cri-o://43283d1910c02712d40fa345f0f5087fcb4b39bb083fea208a8fb941db2246fe" gracePeriod=25 Oct 31 00:12:19.465757 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:12:19.465674 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-a251fdfb-6f8487684c-6qzl9" podUID="f742e7e8-ad0d-4693-8c12-5b2940957133" containerName="payload-pre-processing" containerID="cri-o://ec6b17c2ceea177eb3f8efca903aaec07945a781ce20aa3a572eb4610861da8e" gracePeriod=25 Oct 31 00:12:19.680103 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:12:19.680069 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sqv69" containerName="extractor" Oct 31 00:12:20.826563 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:12:20.826522 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:12:21.986421 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:12:21.986362 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:12:25.352649 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:12:25.352610 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-a251fdfb-6fb6bf7d6b-xmmgr" podUID="c690fe76-022e-44e3-97f3-485dacb13fb9" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 00:12:25.353059 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:12:25.352659 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-a251fdfb-6f8487684c-6qzl9" podUID="f742e7e8-ad0d-4693-8c12-5b2940957133" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 00:12:26.420347 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:12:26.420295 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:12:26.592125 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:12:26.592084 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:12:26.763224 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:12:26.763131 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:12:28.348387 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:12:28.348323 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:12:31.163038 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:12:31.162993 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:12:32.434827 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:12:32.434792 4927 scope.go:122] "RemoveContainer" containerID="45a7cfbae4b2f81551d80e5ce3b2b56559aa77672d1ed322ad5e28483bbfa6d9" Oct 31 00:12:35.354604 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:12:35.354232 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-a251fdfb-6fb6bf7d6b-xmmgr" podUID="c690fe76-022e-44e3-97f3-485dacb13fb9" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 00:12:35.354604 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:12:35.354434 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-a251fdfb-6f8487684c-6qzl9" podUID="f742e7e8-ad0d-4693-8c12-5b2940957133" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 00:12:35.823295 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:12:35.823153 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:12:38.162256 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:12:38.160631 4927 patch_prober.go:29] interesting pod/e2e-api-b-39e3a8-openshift-default-6b6658797-ghqtx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.202:15021/healthz/ready\": read tcp 10.132.0.2:49362->10.132.0.202:15021: read: connection reset by peer" start-of-body= Oct 31 00:12:38.162256 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:12:38.160692 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-39e3a8-openshift-default-6b6658797-ghqtx" podUID="7a19c11a-45ef-434e-9667-e75e17898834" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.202:15021/healthz/ready\": read tcp 10.132.0.2:49362->10.132.0.202:15021: read: connection reset by peer" Oct 31 00:12:38.636669 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:12:38.636571 4927 generic.go:356] "Generic (PLEG): container finished" podID="7a19c11a-45ef-434e-9667-e75e17898834" containerID="77edaee9d31170d4678826952d723309b37e54f7f38e8893aa8fc44583812e4e" exitCode=0 Oct 31 00:12:38.636669 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:12:38.636649 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-39e3a8-openshift-default-6b6658797-ghqtx" event={"ID":"7a19c11a-45ef-434e-9667-e75e17898834","Type":"ContainerDied","Data":"77edaee9d31170d4678826952d723309b37e54f7f38e8893aa8fc44583812e4e"} Oct 31 00:12:38.637160 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:12:38.637136 4927 scope.go:122] "RemoveContainer" containerID="77edaee9d31170d4678826952d723309b37e54f7f38e8893aa8fc44583812e4e" Oct 31 00:12:38.639016 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:12:38.638984 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:12:38.639128 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:12:38.639073 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:12:38.639200 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:12:38.639122 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:12:39.538417 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:12:39.538382 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:12:39.662924 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:12:39.662871 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-39e3a8-openshift-default-6b6658797-ghqtx" event={"ID":"7a19c11a-45ef-434e-9667-e75e17898834","Type":"ContainerStarted","Data":"95c0c344e1207a0e409bc745275f504fef3764d367fd47a27a853fe7cb5c598e"} Oct 31 00:12:39.666200 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:12:39.666168 4927 generic.go:356] "Generic (PLEG): container finished" podID="f742e7e8-ad0d-4693-8c12-5b2940957133" containerID="ec6b17c2ceea177eb3f8efca903aaec07945a781ce20aa3a572eb4610861da8e" exitCode=0 Oct 31 00:12:39.666871 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:12:39.666520 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-a251fdfb-6f8487684c-6qzl9" event={"ID":"f742e7e8-ad0d-4693-8c12-5b2940957133","Type":"ContainerDied","Data":"ec6b17c2ceea177eb3f8efca903aaec07945a781ce20aa3a572eb4610861da8e"} Oct 31 00:12:39.673967 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:12:39.673930 4927 generic.go:356] "Generic (PLEG): container finished" podID="c690fe76-022e-44e3-97f3-485dacb13fb9" containerID="43283d1910c02712d40fa345f0f5087fcb4b39bb083fea208a8fb941db2246fe" exitCode=0 Oct 31 00:12:39.674133 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:12:39.674117 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-a251fdfb-6fb6bf7d6b-xmmgr" event={"ID":"c690fe76-022e-44e3-97f3-485dacb13fb9","Type":"ContainerDied","Data":"43283d1910c02712d40fa345f0f5087fcb4b39bb083fea208a8fb941db2246fe"} Oct 31 00:12:39.690610 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:12:39.690225 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-39e3a8-openshift-default-6b6658797-ghqtx" Oct 31 00:12:39.690610 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:12:39.690319 4927 patch_prober.go:29] interesting pod/e2e-api-b-39e3a8-openshift-default-6b6658797-ghqtx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" start-of-body= Oct 31 00:12:39.690610 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:12:39.690408 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-39e3a8-openshift-default-6b6658797-ghqtx" podUID="7a19c11a-45ef-434e-9667-e75e17898834" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" Oct 31 00:12:39.757146 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:12:39.757098 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-a251fdfb-6fb6bf7d6b-xmmgr" Oct 31 00:12:39.893102 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:12:39.893038 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-a251fdfb-6f8487684c-6qzl9" Oct 31 00:12:39.950039 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:12:39.950002 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c690fe76-022e-44e3-97f3-485dacb13fb9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c690fe76-022e-44e3-97f3-485dacb13fb9-plugins-config-volume\") pod \"c690fe76-022e-44e3-97f3-485dacb13fb9\" (UID: \"c690fe76-022e-44e3-97f3-485dacb13fb9\") " Oct 31 00:12:39.950535 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:12:39.950507 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c690fe76-022e-44e3-97f3-485dacb13fb9-kube-api-access-wpt64\" (UniqueName: \"kubernetes.io/projected/c690fe76-022e-44e3-97f3-485dacb13fb9-kube-api-access-wpt64\") pod \"c690fe76-022e-44e3-97f3-485dacb13fb9\" (UID: \"c690fe76-022e-44e3-97f3-485dacb13fb9\") " Oct 31 00:12:39.962743 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:12:39.962692 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c690fe76-022e-44e3-97f3-485dacb13fb9-kube-api-access-wpt64" pod "c690fe76-022e-44e3-97f3-485dacb13fb9" (UID: "c690fe76-022e-44e3-97f3-485dacb13fb9"). InnerVolumeSpecName "kube-api-access-wpt64". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:12:39.965612 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:12:39.964480 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c690fe76-022e-44e3-97f3-485dacb13fb9-plugins-config-volume" pod "c690fe76-022e-44e3-97f3-485dacb13fb9" (UID: "c690fe76-022e-44e3-97f3-485dacb13fb9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:12:39.993770 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:12:39.993738 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:12:40.052041 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:12:40.051992 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f742e7e8-ad0d-4693-8c12-5b2940957133-kube-api-access-skxk5\" (UniqueName: \"kubernetes.io/projected/f742e7e8-ad0d-4693-8c12-5b2940957133-kube-api-access-skxk5\") pod \"f742e7e8-ad0d-4693-8c12-5b2940957133\" (UID: \"f742e7e8-ad0d-4693-8c12-5b2940957133\") " Oct 31 00:12:40.052400 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:12:40.052378 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f742e7e8-ad0d-4693-8c12-5b2940957133-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f742e7e8-ad0d-4693-8c12-5b2940957133-plugins-config-volume\") pod \"f742e7e8-ad0d-4693-8c12-5b2940957133\" (UID: \"f742e7e8-ad0d-4693-8c12-5b2940957133\") " Oct 31 00:12:40.054075 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:12:40.054044 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f742e7e8-ad0d-4693-8c12-5b2940957133-plugins-config-volume" pod "f742e7e8-ad0d-4693-8c12-5b2940957133" (UID: "f742e7e8-ad0d-4693-8c12-5b2940957133"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:12:40.054272 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:12:40.054249 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wpt64\" (UniqueName: \"kubernetes.io/projected/c690fe76-022e-44e3-97f3-485dacb13fb9-kube-api-access-wpt64\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:12:40.054375 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:12:40.054295 4927 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f742e7e8-ad0d-4693-8c12-5b2940957133-plugins-config-volume\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:12:40.054375 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:12:40.054313 4927 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c690fe76-022e-44e3-97f3-485dacb13fb9-plugins-config-volume\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:12:40.055679 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:12:40.055643 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f742e7e8-ad0d-4693-8c12-5b2940957133-kube-api-access-skxk5" pod "f742e7e8-ad0d-4693-8c12-5b2940957133" (UID: "f742e7e8-ad0d-4693-8c12-5b2940957133"). InnerVolumeSpecName "kube-api-access-skxk5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:12:40.156281 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:12:40.156178 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-skxk5\" (UniqueName: \"kubernetes.io/projected/f742e7e8-ad0d-4693-8c12-5b2940957133-kube-api-access-skxk5\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:12:40.353714 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:12:40.353131 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-957019-77d447ff94-88797"] Oct 31 00:12:40.356626 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:12:40.355590 4927 cpu_manager.go:414] "containerMap: removing container" podUID="f742e7e8-ad0d-4693-8c12-5b2940957133" containerName="payload-pre-processing" Oct 31 00:12:40.356626 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:12:40.355617 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f742e7e8-ad0d-4693-8c12-5b2940957133" containerName="payload-pre-processing" Oct 31 00:12:40.356626 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:12:40.355670 4927 cpu_manager.go:414] "containerMap: removing container" podUID="c690fe76-022e-44e3-97f3-485dacb13fb9" containerName="payload-processing" Oct 31 00:12:40.356626 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:12:40.355678 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c690fe76-022e-44e3-97f3-485dacb13fb9" containerName="payload-processing" Oct 31 00:12:40.356626 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:12:40.355940 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="c690fe76-022e-44e3-97f3-485dacb13fb9" containerName="payload-processing" Oct 31 00:12:40.356626 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:12:40.355965 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="f742e7e8-ad0d-4693-8c12-5b2940957133" containerName="payload-pre-processing" Oct 31 00:12:40.360505 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:12:40.360477 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-957019-77d447ff94-88797" Oct 31 00:12:40.361881 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:12:40.361848 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:12:40.431440 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:12:40.431407 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w0-957019-serving-cert\"" Oct 31 00:12:40.478140 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:12:40.478095 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f5pvj\" (UniqueName: \"kubernetes.io/projected/08cf4b69-cb92-439b-bd2f-66734dc9f641-kube-api-access-f5pvj\") pod \"maas-api-e2e-shared-b-w0-957019-77d447ff94-88797\" (UID: \"08cf4b69-cb92-439b-bd2f-66734dc9f641\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-957019-77d447ff94-88797" Oct 31 00:12:40.478640 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:12:40.478611 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/08cf4b69-cb92-439b-bd2f-66734dc9f641-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-957019-77d447ff94-88797\" (UID: \"08cf4b69-cb92-439b-bd2f-66734dc9f641\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-957019-77d447ff94-88797" Oct 31 00:12:40.493616 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:12:40.493582 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/08cf4b69-cb92-439b-bd2f-66734dc9f641-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-957019-77d447ff94-88797\" (UID: \"08cf4b69-cb92-439b-bd2f-66734dc9f641\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-957019-77d447ff94-88797" Oct 31 00:12:40.493973 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:12:40.493952 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/08cf4b69-cb92-439b-bd2f-66734dc9f641-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-957019-77d447ff94-88797\" (UID: \"08cf4b69-cb92-439b-bd2f-66734dc9f641\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-957019-77d447ff94-88797" Oct 31 00:12:40.561451 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:12:40.561417 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-957019-77d447ff94-88797"] Oct 31 00:12:40.564673 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:12:40.564638 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:12:40.680438 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:12:40.656291 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/08cf4b69-cb92-439b-bd2f-66734dc9f641-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-957019-77d447ff94-88797\" (UID: \"08cf4b69-cb92-439b-bd2f-66734dc9f641\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-957019-77d447ff94-88797" Oct 31 00:12:40.681105 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:12:40.681060 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/08cf4b69-cb92-439b-bd2f-66734dc9f641-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-957019-77d447ff94-88797\" (UID: \"08cf4b69-cb92-439b-bd2f-66734dc9f641\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-957019-77d447ff94-88797" Oct 31 00:12:40.681428 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:12:40.681408 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/08cf4b69-cb92-439b-bd2f-66734dc9f641-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-957019-77d447ff94-88797\" (UID: \"08cf4b69-cb92-439b-bd2f-66734dc9f641\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-957019-77d447ff94-88797" Oct 31 00:12:40.681652 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:12:40.681606 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5pvj\" (UniqueName: \"kubernetes.io/projected/08cf4b69-cb92-439b-bd2f-66734dc9f641-kube-api-access-f5pvj\") pod \"maas-api-e2e-shared-b-w0-957019-77d447ff94-88797\" (UID: \"08cf4b69-cb92-439b-bd2f-66734dc9f641\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-957019-77d447ff94-88797" Oct 31 00:12:40.710362 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:12:40.694226 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/08cf4b69-cb92-439b-bd2f-66734dc9f641-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-957019-77d447ff94-88797\" (UID: \"08cf4b69-cb92-439b-bd2f-66734dc9f641\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-957019-77d447ff94-88797" Oct 31 00:12:40.711017 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:12:40.694700 4927 patch_prober.go:29] interesting pod/e2e-api-b-39e3a8-openshift-default-6b6658797-ghqtx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" start-of-body= Oct 31 00:12:40.711266 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:12:40.711236 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-39e3a8-openshift-default-6b6658797-ghqtx" podUID="7a19c11a-45ef-434e-9667-e75e17898834" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" Oct 31 00:12:40.812368 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:12:40.810944 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/08cf4b69-cb92-439b-bd2f-66734dc9f641-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-957019-77d447ff94-88797\" (UID: \"08cf4b69-cb92-439b-bd2f-66734dc9f641\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-957019-77d447ff94-88797" Oct 31 00:12:40.875356 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:12:40.874033 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/08cf4b69-cb92-439b-bd2f-66734dc9f641-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-957019-77d447ff94-88797\" (UID: \"08cf4b69-cb92-439b-bd2f-66734dc9f641\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-957019-77d447ff94-88797" Oct 31 00:12:40.982752 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:12:40.982613 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:12:40.985492 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:12:40.985287 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f5pvj\" (UniqueName: \"kubernetes.io/projected/08cf4b69-cb92-439b-bd2f-66734dc9f641-kube-api-access-f5pvj\") pod \"maas-api-e2e-shared-b-w0-957019-77d447ff94-88797\" (UID: \"08cf4b69-cb92-439b-bd2f-66734dc9f641\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-957019-77d447ff94-88797" Oct 31 00:12:41.001834 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:12:40.996156 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-a251fdfb-6f8487684c-6qzl9" event={"ID":"f742e7e8-ad0d-4693-8c12-5b2940957133","Type":"ContainerDied","Data":"e29884b7ba0c9ca298962593b469ca67522a6686a3ee1527b159d7e5ebde48e5"} Oct 31 00:12:41.001834 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:12:40.996256 4927 scope.go:122] "RemoveContainer" containerID="ec6b17c2ceea177eb3f8efca903aaec07945a781ce20aa3a572eb4610861da8e" Oct 31 00:12:41.001834 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:12:40.996537 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-a251fdfb-6f8487684c-6qzl9" Oct 31 00:12:41.032614 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:12:41.006866 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-a251fdfb-6fb6bf7d6b-xmmgr" Oct 31 00:12:41.090374 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:12:41.039704 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-a251fdfb-6fb6bf7d6b-xmmgr" event={"ID":"c690fe76-022e-44e3-97f3-485dacb13fb9","Type":"ContainerDied","Data":"b35d71a69c17001cbb8637ebf94c1624c7a936f4fa860a43a65da00b207d14e5"} Oct 31 00:12:41.282534 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:12:41.282178 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-957019-77d447ff94-88797" Oct 31 00:12:41.283702 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:12:41.283654 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:12:41.298974 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:12:41.286496 4927 scope.go:122] "RemoveContainer" containerID="43283d1910c02712d40fa345f0f5087fcb4b39bb083fea208a8fb941db2246fe" Oct 31 00:12:41.421353 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:12:41.420451 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-a251fdfb-6fb6bf7d6b-xmmgr"] Oct 31 00:12:41.449932 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:12:41.448535 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-a251fdfb-6fb6bf7d6b-xmmgr"] Oct 31 00:12:41.557792 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:12:41.557679 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-a251fdfb-6f8487684c-6qzl9"] Oct 31 00:12:41.559319 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:12:41.559207 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-a251fdfb-6f8487684c-6qzl9"] Oct 31 00:12:41.690651 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:12:41.690598 4927 patch_prober.go:29] interesting pod/e2e-api-b-39e3a8-openshift-default-6b6658797-ghqtx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" start-of-body= Oct 31 00:12:41.691086 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:12:41.690673 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-39e3a8-openshift-default-6b6658797-ghqtx" podUID="7a19c11a-45ef-434e-9667-e75e17898834" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" Oct 31 00:12:41.773954 ip-10-0-141-180 kubenswrapper[4927]: W1031 00:12:41.773924 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08cf4b69_cb92_439b_bd2f_66734dc9f641.slice/crio-2678ad28b757546d7dad3d62f58fb88745e4eff0b7ddabb21c5282d3c0222a12 WatchSource:0}: Error finding container 2678ad28b757546d7dad3d62f58fb88745e4eff0b7ddabb21c5282d3c0222a12: Status 404 returned error can't find the container with id 2678ad28b757546d7dad3d62f58fb88745e4eff0b7ddabb21c5282d3c0222a12 Oct 31 00:12:41.774252 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:12:41.774230 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-957019-77d447ff94-88797"] Oct 31 00:12:42.041366 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:12:42.039431 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-957019-77d447ff94-88797" event={"ID":"08cf4b69-cb92-439b-bd2f-66734dc9f641","Type":"ContainerStarted","Data":"2678ad28b757546d7dad3d62f58fb88745e4eff0b7ddabb21c5282d3c0222a12"} Oct 31 00:12:42.690172 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:12:42.690083 4927 patch_prober.go:29] interesting pod/e2e-api-b-39e3a8-openshift-default-6b6658797-ghqtx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" start-of-body= Oct 31 00:12:42.690172 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:12:42.690126 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-39e3a8-openshift-default-6b6658797-ghqtx" podUID="7a19c11a-45ef-434e-9667-e75e17898834" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" Oct 31 00:12:43.056110 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:12:43.055994 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-957019-77d447ff94-88797" event={"ID":"08cf4b69-cb92-439b-bd2f-66734dc9f641","Type":"ContainerStarted","Data":"3e33d5b602b753df19917df823673000655af48d843e6dcc6dbb908e9036f161"} Oct 31 00:12:43.058581 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:12:43.057013 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-957019-77d447ff94-88797" Oct 31 00:12:43.091405 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:12:43.091367 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c690fe76-022e-44e3-97f3-485dacb13fb9" path="/var/lib/kubelet/pods/c690fe76-022e-44e3-97f3-485dacb13fb9/volumes" Oct 31 00:12:43.092007 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:12:43.091984 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f742e7e8-ad0d-4693-8c12-5b2940957133" path="/var/lib/kubelet/pods/f742e7e8-ad0d-4693-8c12-5b2940957133/volumes" Oct 31 00:12:43.183240 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:12:43.183179 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-957019-77d447ff94-88797" podStartSLOduration=2.283250765 podStartE2EDuration="3.183165589s" totalImagesPullingTime="899.914824ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:12:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:12:43.179684804 +0000 UTC m=+1876.216015622" watchObservedRunningTime="2026-10-31 00:12:43.183165589 +0000 UTC m=+1876.219496410" Oct 31 00:12:43.689210 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:12:43.689174 4927 patch_prober.go:29] interesting pod/e2e-api-b-39e3a8-openshift-default-6b6658797-ghqtx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" start-of-body= Oct 31 00:12:43.689427 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:12:43.689233 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-39e3a8-openshift-default-6b6658797-ghqtx" podUID="7a19c11a-45ef-434e-9667-e75e17898834" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" Oct 31 00:12:44.689619 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:12:44.689582 4927 patch_prober.go:29] interesting pod/e2e-api-b-39e3a8-openshift-default-6b6658797-ghqtx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" start-of-body= Oct 31 00:12:44.690068 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:12:44.689634 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-39e3a8-openshift-default-6b6658797-ghqtx" podUID="7a19c11a-45ef-434e-9667-e75e17898834" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" Oct 31 00:12:44.989226 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:12:44.989140 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:12:45.197543 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:12:45.197501 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:12:45.585457 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:12:45.585411 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:12:45.696351 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:12:45.692703 4927 patch_prober.go:29] interesting pod/e2e-api-b-39e3a8-openshift-default-6b6658797-ghqtx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" start-of-body= Oct 31 00:12:45.696351 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:12:45.692785 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-39e3a8-openshift-default-6b6658797-ghqtx" podUID="7a19c11a-45ef-434e-9667-e75e17898834" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" Oct 31 00:12:46.695130 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:12:46.695097 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-39e3a8-openshift-default-6b6658797-ghqtx" Oct 31 00:12:46.695617 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:12:46.695598 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-39e3a8-openshift-default-6b6658797-ghqtx" Oct 31 00:12:46.696371 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:12:46.696319 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-39e3a8-openshift-default-6b6658797-ghqtx" Oct 31 00:12:48.576413 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:12:48.576369 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:12:48.762236 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:12:48.762045 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:12:49.332353 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:12:49.332075 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-d9e782-67987686-hwzmf"] Oct 31 00:12:49.384319 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:12:49.379064 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-d9e782-67987686-hwzmf"] Oct 31 00:12:49.384319 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:12:49.379502 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-d9e782-67987686-hwzmf" Oct 31 00:12:49.402916 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:12:49.402884 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b45aa3c1-e11b-4d72-b93c-2248b7916834-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-d9e782-67987686-hwzmf\" (UID: \"b45aa3c1-e11b-4d72-b93c-2248b7916834\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-d9e782-67987686-hwzmf" Oct 31 00:12:49.403058 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:12:49.403019 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8j9lw\" (UniqueName: \"kubernetes.io/projected/b45aa3c1-e11b-4d72-b93c-2248b7916834-kube-api-access-8j9lw\") pod \"payload-processing-e2e-shared-b-w3-d9e782-67987686-hwzmf\" (UID: \"b45aa3c1-e11b-4d72-b93c-2248b7916834\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-d9e782-67987686-hwzmf" Oct 31 00:12:49.437489 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:12:49.437456 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-d9e782\"" Oct 31 00:12:49.488544 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:12:49.488517 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-d9e782-dockercfg-jcwqr\"" Oct 31 00:12:49.504969 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:12:49.504932 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b45aa3c1-e11b-4d72-b93c-2248b7916834-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-d9e782-67987686-hwzmf\" (UID: \"b45aa3c1-e11b-4d72-b93c-2248b7916834\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-d9e782-67987686-hwzmf" Oct 31 00:12:49.505101 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:12:49.505070 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8j9lw\" (UniqueName: \"kubernetes.io/projected/b45aa3c1-e11b-4d72-b93c-2248b7916834-kube-api-access-8j9lw\") pod \"payload-processing-e2e-shared-b-w3-d9e782-67987686-hwzmf\" (UID: \"b45aa3c1-e11b-4d72-b93c-2248b7916834\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-d9e782-67987686-hwzmf" Oct 31 00:12:49.505748 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:12:49.505730 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b45aa3c1-e11b-4d72-b93c-2248b7916834-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-d9e782-67987686-hwzmf\" (UID: \"b45aa3c1-e11b-4d72-b93c-2248b7916834\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-d9e782-67987686-hwzmf" Oct 31 00:12:49.567673 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:12:49.567645 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8j9lw\" (UniqueName: \"kubernetes.io/projected/b45aa3c1-e11b-4d72-b93c-2248b7916834-kube-api-access-8j9lw\") pod \"payload-processing-e2e-shared-b-w3-d9e782-67987686-hwzmf\" (UID: \"b45aa3c1-e11b-4d72-b93c-2248b7916834\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-d9e782-67987686-hwzmf" Oct 31 00:12:49.680064 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:12:49.680021 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sqv69" containerName="extractor" Oct 31 00:12:49.702894 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:12:49.702867 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-d9e782-67987686-hwzmf" Oct 31 00:12:50.083999 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:12:50.083975 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-957019-77d447ff94-88797" Oct 31 00:12:50.087523 ip-10-0-141-180 kubenswrapper[4927]: W1031 00:12:50.087495 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb45aa3c1_e11b_4d72_b93c_2248b7916834.slice/crio-90c2c0e0dd845d2c1501a23ec861d40f5bf8f049804d2fb60c886145da412826 WatchSource:0}: Error finding container 90c2c0e0dd845d2c1501a23ec861d40f5bf8f049804d2fb60c886145da412826: Status 404 returned error can't find the container with id 90c2c0e0dd845d2c1501a23ec861d40f5bf8f049804d2fb60c886145da412826 Oct 31 00:12:50.087623 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:12:50.087523 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-d9e782-67987686-hwzmf"] Oct 31 00:12:50.333234 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:12:50.333193 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-d9e782-67987686-hwzmf" event={"ID":"b45aa3c1-e11b-4d72-b93c-2248b7916834","Type":"ContainerStarted","Data":"61958775a787dc49f2a0ae9730b259296d1852329d78b4640794111c16ae5d86"} Oct 31 00:12:50.333484 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:12:50.333257 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-d9e782-67987686-hwzmf" event={"ID":"b45aa3c1-e11b-4d72-b93c-2248b7916834","Type":"ContainerStarted","Data":"90c2c0e0dd845d2c1501a23ec861d40f5bf8f049804d2fb60c886145da412826"} Oct 31 00:12:50.333484 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:12:50.333450 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-d9e782-67987686-hwzmf" Oct 31 00:12:50.458607 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:12:50.458542 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-d9e782-67987686-hwzmf" podStartSLOduration=2.458525121 podStartE2EDuration="2.458525121s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:12:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:12:50.455031189 +0000 UTC m=+1883.491362028" watchObservedRunningTime="2026-10-31 00:12:50.458525121 +0000 UTC m=+1883.494855944" Oct 31 00:12:50.671954 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:12:50.671910 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:12:55.479307 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:12:55.479271 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:12:59.957121 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:12:59.957075 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-4a3dba-744d479576-fx5nx"] Oct 31 00:12:59.962203 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:12:59.962185 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-4a3dba-744d479576-fx5nx" Oct 31 00:13:00.023421 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:13:00.023225 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-4a3dba-744d479576-fx5nx"] Oct 31 00:13:00.066062 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:13:00.066021 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w4-4a3dba-serving-cert\"" Oct 31 00:13:00.072091 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:13:00.072055 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f0295476-7278-4b04-af13-d09dafe0b5b6-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-4a3dba-744d479576-fx5nx\" (UID: \"f0295476-7278-4b04-af13-d09dafe0b5b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-4a3dba-744d479576-fx5nx" Oct 31 00:13:00.072244 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:13:00.072197 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0295476-7278-4b04-af13-d09dafe0b5b6-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-4a3dba-744d479576-fx5nx\" (UID: \"f0295476-7278-4b04-af13-d09dafe0b5b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-4a3dba-744d479576-fx5nx" Oct 31 00:13:00.072362 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:13:00.072322 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6bt8r\" (UniqueName: \"kubernetes.io/projected/f0295476-7278-4b04-af13-d09dafe0b5b6-kube-api-access-6bt8r\") pod \"maas-api-e2e-shared-b-w4-4a3dba-744d479576-fx5nx\" (UID: \"f0295476-7278-4b04-af13-d09dafe0b5b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-4a3dba-744d479576-fx5nx" Oct 31 00:13:00.072446 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:13:00.072429 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f0295476-7278-4b04-af13-d09dafe0b5b6-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-4a3dba-744d479576-fx5nx\" (UID: \"f0295476-7278-4b04-af13-d09dafe0b5b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-4a3dba-744d479576-fx5nx" Oct 31 00:13:00.174244 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:13:00.174208 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f0295476-7278-4b04-af13-d09dafe0b5b6-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-4a3dba-744d479576-fx5nx\" (UID: \"f0295476-7278-4b04-af13-d09dafe0b5b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-4a3dba-744d479576-fx5nx" Oct 31 00:13:00.174455 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:13:00.174311 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0295476-7278-4b04-af13-d09dafe0b5b6-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-4a3dba-744d479576-fx5nx\" (UID: \"f0295476-7278-4b04-af13-d09dafe0b5b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-4a3dba-744d479576-fx5nx" Oct 31 00:13:00.174455 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:13:00.174385 4927 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-4a3dba-serving-cert: secret "maas-api-e2e-shared-b-w4-4a3dba-serving-cert" not found Oct 31 00:13:00.174455 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:13:00.174411 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bt8r\" (UniqueName: \"kubernetes.io/projected/f0295476-7278-4b04-af13-d09dafe0b5b6-kube-api-access-6bt8r\") pod \"maas-api-e2e-shared-b-w4-4a3dba-744d479576-fx5nx\" (UID: \"f0295476-7278-4b04-af13-d09dafe0b5b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-4a3dba-744d479576-fx5nx" Oct 31 00:13:00.174455 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:13:00.174456 4927 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f0295476-7278-4b04-af13-d09dafe0b5b6-maas-api-tls podName:f0295476-7278-4b04-af13-d09dafe0b5b6 nodeName:}" failed. No retries permitted until 2026-10-31 00:13:00.674439312 +0000 UTC m=+1893.710770131 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f0295476-7278-4b04-af13-d09dafe0b5b6-maas-api-tls") pod "maas-api-e2e-shared-b-w4-4a3dba-744d479576-fx5nx" (UID: "f0295476-7278-4b04-af13-d09dafe0b5b6") : secret "maas-api-e2e-shared-b-w4-4a3dba-serving-cert" not found Oct 31 00:13:00.174705 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:13:00.174505 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f0295476-7278-4b04-af13-d09dafe0b5b6-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-4a3dba-744d479576-fx5nx\" (UID: \"f0295476-7278-4b04-af13-d09dafe0b5b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-4a3dba-744d479576-fx5nx" Oct 31 00:13:00.175074 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:13:00.175048 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0295476-7278-4b04-af13-d09dafe0b5b6-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-4a3dba-744d479576-fx5nx\" (UID: \"f0295476-7278-4b04-af13-d09dafe0b5b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-4a3dba-744d479576-fx5nx" Oct 31 00:13:00.177085 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:13:00.177061 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f0295476-7278-4b04-af13-d09dafe0b5b6-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-4a3dba-744d479576-fx5nx\" (UID: \"f0295476-7278-4b04-af13-d09dafe0b5b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-4a3dba-744d479576-fx5nx" Oct 31 00:13:00.187445 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:13:00.187421 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:13:00.257414 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:13:00.256994 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bt8r\" (UniqueName: \"kubernetes.io/projected/f0295476-7278-4b04-af13-d09dafe0b5b6-kube-api-access-6bt8r\") pod \"maas-api-e2e-shared-b-w4-4a3dba-744d479576-fx5nx\" (UID: \"f0295476-7278-4b04-af13-d09dafe0b5b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-4a3dba-744d479576-fx5nx" Oct 31 00:13:00.384268 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:13:00.384233 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:13:00.548404 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:13:00.545425 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:13:00.699906 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:13:00.699870 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f0295476-7278-4b04-af13-d09dafe0b5b6-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-4a3dba-744d479576-fx5nx\" (UID: \"f0295476-7278-4b04-af13-d09dafe0b5b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-4a3dba-744d479576-fx5nx" Oct 31 00:13:00.702356 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:13:00.702313 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f0295476-7278-4b04-af13-d09dafe0b5b6-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-4a3dba-744d479576-fx5nx\" (UID: \"f0295476-7278-4b04-af13-d09dafe0b5b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-4a3dba-744d479576-fx5nx" Oct 31 00:13:00.876985 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:13:00.876898 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-4a3dba-744d479576-fx5nx" Oct 31 00:13:01.275362 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:13:01.275313 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-4a3dba-744d479576-fx5nx"] Oct 31 00:13:01.279097 ip-10-0-141-180 kubenswrapper[4927]: W1031 00:13:01.279069 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0295476_7278_4b04_af13_d09dafe0b5b6.slice/crio-00bbaf6706ea5db937ea575ce594963bf4252a95da2c55102e93e5840fd56f65 WatchSource:0}: Error finding container 00bbaf6706ea5db937ea575ce594963bf4252a95da2c55102e93e5840fd56f65: Status 404 returned error can't find the container with id 00bbaf6706ea5db937ea575ce594963bf4252a95da2c55102e93e5840fd56f65 Oct 31 00:13:01.346076 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:13:01.346045 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-d9e782-67987686-hwzmf" Oct 31 00:13:01.466078 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:13:01.466035 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-4a3dba-744d479576-fx5nx" event={"ID":"f0295476-7278-4b04-af13-d09dafe0b5b6","Type":"ContainerStarted","Data":"00bbaf6706ea5db937ea575ce594963bf4252a95da2c55102e93e5840fd56f65"} Oct 31 00:13:02.478481 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:13:02.477449 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-4a3dba-744d479576-fx5nx" event={"ID":"f0295476-7278-4b04-af13-d09dafe0b5b6","Type":"ContainerStarted","Data":"2d8ef2f89979a6d233b1d612e90d708121b6b44568cc97de08635e8ea03d4674"} Oct 31 00:13:02.478481 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:13:02.477720 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-4a3dba-744d479576-fx5nx" Oct 31 00:13:02.601794 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:13:02.601737 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-4a3dba-744d479576-fx5nx" podStartSLOduration=2.643592677 podStartE2EDuration="3.601722437s" totalImagesPullingTime="958.12976ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:12:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:13:02.59703633 +0000 UTC m=+1895.633367152" watchObservedRunningTime="2026-10-31 00:13:02.601722437 +0000 UTC m=+1895.638053298" Oct 31 00:13:05.264130 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:13:05.264100 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:13:08.493562 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:13:08.493531 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-4a3dba-744d479576-fx5nx" Oct 31 00:13:10.251458 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:13:10.251415 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:13:14.834820 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:13:14.834780 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:13:19.038299 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:13:19.038253 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:13:19.189830 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:13:19.189797 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:13:19.371095 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:13:19.371007 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:13:19.680376 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:13:19.680327 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sqv69" containerName="extractor" Oct 31 00:13:20.702889 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:13:20.702851 4927 generic.go:356] "Generic (PLEG): container finished" podID="9ee78c4f-f1a0-4cc0-8441-106def585fff" containerID="b203e94aec128fce45da60bb93ab9e2259f6808bbc342b332cb76a4071ddd011" exitCode=0 Oct 31 00:13:20.703355 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:13:20.702926 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-03c703-openshift-default-78bd587bff-5m2n8" event={"ID":"9ee78c4f-f1a0-4cc0-8441-106def585fff","Type":"ContainerDied","Data":"b203e94aec128fce45da60bb93ab9e2259f6808bbc342b332cb76a4071ddd011"} Oct 31 00:13:20.703408 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:13:20.703391 4927 scope.go:122] "RemoveContainer" containerID="b203e94aec128fce45da60bb93ab9e2259f6808bbc342b332cb76a4071ddd011" Oct 31 00:13:20.705198 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:13:20.705161 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:13:20.705313 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:13:20.705251 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:13:20.705313 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:13:20.705299 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:13:21.740551 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:13:21.740509 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-03c703-openshift-default-78bd587bff-5m2n8" event={"ID":"9ee78c4f-f1a0-4cc0-8441-106def585fff","Type":"ContainerStarted","Data":"3f3ceed141a812642c377a73790e461ebd99d22338f2aefc053f34bc04922e55"} Oct 31 00:13:22.221725 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:13:22.221691 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-03c703-openshift-default-78bd587bff-5m2n8" Oct 31 00:13:22.222097 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:13:22.222073 4927 patch_prober.go:29] interesting pod/e2e-inf-a-03c703-openshift-default-78bd587bff-5m2n8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused" start-of-body= Oct 31 00:13:22.222265 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:13:22.222238 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-03c703-openshift-default-78bd587bff-5m2n8" podUID="9ee78c4f-f1a0-4cc0-8441-106def585fff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused" Oct 31 00:13:23.221354 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:13:23.220667 4927 patch_prober.go:29] interesting pod/e2e-inf-a-03c703-openshift-default-78bd587bff-5m2n8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused" start-of-body= Oct 31 00:13:23.221354 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:13:23.220746 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-03c703-openshift-default-78bd587bff-5m2n8" podUID="9ee78c4f-f1a0-4cc0-8441-106def585fff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused" Oct 31 00:13:23.928611 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:13:23.928569 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:13:24.067197 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:13:24.067155 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-16364156-gx8fb"] Oct 31 00:13:24.106791 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:13:24.106753 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-16364156-gx8fb"] Oct 31 00:13:24.133356 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:13:24.132627 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c8d1a25-ns4dx"] Oct 31 00:13:24.141716 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:13:24.141685 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c8d1a25-ns4dx"] Oct 31 00:13:24.229778 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:13:24.229686 4927 patch_prober.go:29] interesting pod/e2e-inf-a-03c703-openshift-default-78bd587bff-5m2n8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused" start-of-body= Oct 31 00:13:24.230353 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:13:24.230300 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-03c703-openshift-default-78bd587bff-5m2n8" podUID="9ee78c4f-f1a0-4cc0-8441-106def585fff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused" Oct 31 00:13:25.088925 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:13:25.088891 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cec08908-5b14-4d8c-a101-174d1496e41c" path="/var/lib/kubelet/pods/cec08908-5b14-4d8c-a101-174d1496e41c/volumes" Oct 31 00:13:25.089390 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:13:25.089375 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f369330e-5800-40d6-929c-bc9996aaa3d9" path="/var/lib/kubelet/pods/f369330e-5800-40d6-929c-bc9996aaa3d9/volumes" Oct 31 00:13:25.221766 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:13:25.221725 4927 patch_prober.go:29] interesting pod/e2e-inf-a-03c703-openshift-default-78bd587bff-5m2n8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused" start-of-body= Oct 31 00:13:25.222063 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:13:25.222032 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-03c703-openshift-default-78bd587bff-5m2n8" podUID="9ee78c4f-f1a0-4cc0-8441-106def585fff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused" Oct 31 00:13:26.224356 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:13:26.224146 4927 patch_prober.go:29] interesting pod/e2e-inf-a-03c703-openshift-default-78bd587bff-5m2n8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused" start-of-body= Oct 31 00:13:26.224356 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:13:26.224214 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-03c703-openshift-default-78bd587bff-5m2n8" podUID="9ee78c4f-f1a0-4cc0-8441-106def585fff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused" Oct 31 00:13:27.223140 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:13:27.223106 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-03c703-openshift-default-78bd587bff-5m2n8" Oct 31 00:13:27.223589 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:13:27.223569 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-03c703-openshift-default-78bd587bff-5m2n8" Oct 31 00:13:27.224361 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:13:27.224309 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-03c703-openshift-default-78bd587bff-5m2n8" Oct 31 00:13:28.475085 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:13:28.475047 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:13:28.708391 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:13:28.708359 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:13:32.473963 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:13:32.473924 4927 scope.go:122] "RemoveContainer" containerID="c91807b9bc94a00f3405da15edff2caa442cc5056215fb0d07b9826453e94a78" Oct 31 00:13:32.485371 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:13:32.485342 4927 scope.go:122] "RemoveContainer" containerID="801fbd810ffe2c4a894321745973a5a789caed00ad1b177930dc6e6a9a01e689" Oct 31 00:13:33.197125 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:13:33.197075 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:13:37.902225 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:13:37.898723 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:13:42.671126 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:13:42.671091 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:13:42.826767 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:13:42.826714 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:13:45.201954 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:13:45.201921 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:13:46.402278 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:13:46.402125 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:13:46.701997 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:13:46.701963 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:13:47.795712 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:13:47.795672 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:13:49.068040 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:13:49.068001 4927 generic.go:356] "Generic (PLEG): container finished" podID="1b1c447d-a95f-4edb-9c9d-8987eea5eaf4" containerID="d750bd6d9be7f3a0cecd96781cede0a31a4438375a563f22bd141188270130da" exitCode=0 Oct 31 00:13:49.068597 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:13:49.068065 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-4a3dba-openshift-default-dc9664475-zxb5s" event={"ID":"1b1c447d-a95f-4edb-9c9d-8987eea5eaf4","Type":"ContainerDied","Data":"d750bd6d9be7f3a0cecd96781cede0a31a4438375a563f22bd141188270130da"} Oct 31 00:13:49.068672 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:13:49.068609 4927 scope.go:122] "RemoveContainer" containerID="d750bd6d9be7f3a0cecd96781cede0a31a4438375a563f22bd141188270130da" Oct 31 00:13:49.070466 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:13:49.070427 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:13:49.070574 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:13:49.070494 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:13:49.070574 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:13:49.070528 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:13:49.697021 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:13:49.696982 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sqv69" containerName="extractor" Oct 31 00:13:49.748046 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:13:49.748010 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-4a3dba-openshift-default-dc9664475-zxb5s" Oct 31 00:13:50.080475 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:13:50.080370 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-4a3dba-openshift-default-dc9664475-zxb5s" event={"ID":"1b1c447d-a95f-4edb-9c9d-8987eea5eaf4","Type":"ContainerStarted","Data":"963a5aa1c641ed6f16929af7e3016f135429161948921c103d3981e14fd8ab1d"} Oct 31 00:13:50.174287 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:13:50.174251 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:13:50.359002 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:13:50.358911 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-4a3dba-openshift-default-dc9664475-zxb5s" Oct 31 00:13:50.360405 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:13:50.360375 4927 patch_prober.go:29] interesting pod/e2e-shared-b-w4-4a3dba-openshift-default-dc9664475-zxb5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" start-of-body= Oct 31 00:13:50.360536 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:13:50.360424 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-4a3dba-openshift-default-dc9664475-zxb5s" podUID="1b1c447d-a95f-4edb-9c9d-8987eea5eaf4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" Oct 31 00:13:50.406727 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:13:50.406687 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:13:50.573499 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:13:50.573462 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:13:51.358866 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:13:51.358830 4927 patch_prober.go:29] interesting pod/e2e-shared-b-w4-4a3dba-openshift-default-dc9664475-zxb5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" start-of-body= Oct 31 00:13:51.359366 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:13:51.358883 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-4a3dba-openshift-default-dc9664475-zxb5s" podUID="1b1c447d-a95f-4edb-9c9d-8987eea5eaf4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" Oct 31 00:13:52.102158 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:13:52.102122 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:13:52.362404 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:13:52.358461 4927 patch_prober.go:29] interesting pod/e2e-shared-b-w4-4a3dba-openshift-default-dc9664475-zxb5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" start-of-body= Oct 31 00:13:52.362404 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:13:52.358570 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-4a3dba-openshift-default-dc9664475-zxb5s" podUID="1b1c447d-a95f-4edb-9c9d-8987eea5eaf4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" Oct 31 00:13:53.361773 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:13:53.361733 4927 patch_prober.go:29] interesting pod/e2e-shared-b-w4-4a3dba-openshift-default-dc9664475-zxb5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" start-of-body= Oct 31 00:13:53.361995 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:13:53.361791 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-4a3dba-openshift-default-dc9664475-zxb5s" podUID="1b1c447d-a95f-4edb-9c9d-8987eea5eaf4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" Oct 31 00:13:53.972085 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:13:53.972047 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:13:54.122190 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:13:54.122136 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:13:54.300029 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:13:54.299949 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:13:54.368354 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:13:54.363950 4927 patch_prober.go:29] interesting pod/e2e-shared-b-w4-4a3dba-openshift-default-dc9664475-zxb5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" start-of-body= Oct 31 00:13:54.368354 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:13:54.364047 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-4a3dba-openshift-default-dc9664475-zxb5s" podUID="1b1c447d-a95f-4edb-9c9d-8987eea5eaf4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" Oct 31 00:13:54.935231 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:13:54.935181 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:13:55.194401 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:13:55.189052 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:13:55.394355 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:13:55.359878 4927 patch_prober.go:29] interesting pod/e2e-shared-b-w4-4a3dba-openshift-default-dc9664475-zxb5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" start-of-body= Oct 31 00:13:55.394355 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:13:55.359961 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-4a3dba-openshift-default-dc9664475-zxb5s" podUID="1b1c447d-a95f-4edb-9c9d-8987eea5eaf4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" Oct 31 00:13:55.924293 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:13:55.924255 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-348a33-6987f947f-cqp2m"] Oct 31 00:13:55.931530 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:13:55.931500 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-348a33-6987f947f-cqp2m"] Oct 31 00:13:55.931738 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:13:55.931638 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-348a33-6987f947f-cqp2m" Oct 31 00:13:56.036635 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:13:56.036603 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-348a33-dockercfg-d57sz\"" Oct 31 00:13:56.036635 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:13:56.036623 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-348a33\"" Oct 31 00:13:56.100887 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:13:56.100850 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zzpt7\" (UniqueName: \"kubernetes.io/projected/31bc5ff3-08ec-470f-b1f9-713946258110-kube-api-access-zzpt7\") pod \"payload-pre-processing-e2e-inf-a-348a33-6987f947f-cqp2m\" (UID: \"31bc5ff3-08ec-470f-b1f9-713946258110\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-348a33-6987f947f-cqp2m" Oct 31 00:13:56.101135 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:13:56.100988 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31bc5ff3-08ec-470f-b1f9-713946258110-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-348a33-6987f947f-cqp2m\" (UID: \"31bc5ff3-08ec-470f-b1f9-713946258110\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-348a33-6987f947f-cqp2m" Oct 31 00:13:56.184557 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:13:56.184467 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-71b76a-64f77d669f-hm2nn"] Oct 31 00:13:56.202655 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:13:56.202611 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31bc5ff3-08ec-470f-b1f9-713946258110-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-348a33-6987f947f-cqp2m\" (UID: \"31bc5ff3-08ec-470f-b1f9-713946258110\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-348a33-6987f947f-cqp2m" Oct 31 00:13:56.203077 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:13:56.202850 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zzpt7\" (UniqueName: \"kubernetes.io/projected/31bc5ff3-08ec-470f-b1f9-713946258110-kube-api-access-zzpt7\") pod \"payload-pre-processing-e2e-inf-a-348a33-6987f947f-cqp2m\" (UID: \"31bc5ff3-08ec-470f-b1f9-713946258110\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-348a33-6987f947f-cqp2m" Oct 31 00:13:56.203372 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:13:56.203353 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31bc5ff3-08ec-470f-b1f9-713946258110-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-348a33-6987f947f-cqp2m\" (UID: \"31bc5ff3-08ec-470f-b1f9-713946258110\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-348a33-6987f947f-cqp2m" Oct 31 00:13:56.265972 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:13:56.265942 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zzpt7\" (UniqueName: \"kubernetes.io/projected/31bc5ff3-08ec-470f-b1f9-713946258110-kube-api-access-zzpt7\") pod \"payload-pre-processing-e2e-inf-a-348a33-6987f947f-cqp2m\" (UID: \"31bc5ff3-08ec-470f-b1f9-713946258110\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-348a33-6987f947f-cqp2m" Oct 31 00:13:56.367039 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:13:56.367010 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-4a3dba-openshift-default-dc9664475-zxb5s" Oct 31 00:13:56.367643 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:13:56.367615 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-4a3dba-openshift-default-dc9664475-zxb5s" Oct 31 00:13:56.369868 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:13:56.369844 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-4a3dba-openshift-default-dc9664475-zxb5s" Oct 31 00:13:56.438884 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:13:56.438791 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:13:56.551118 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:13:56.551081 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-348a33-6987f947f-cqp2m" Oct 31 00:13:56.594055 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:13:56.593794 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:13:56.987288 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:13:56.987258 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-348a33-6987f947f-cqp2m"] Oct 31 00:13:56.987825 ip-10-0-141-180 kubenswrapper[4927]: W1031 00:13:56.987796 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31bc5ff3_08ec_470f_b1f9_713946258110.slice/crio-a71611b508fd833c32f61a19f3fd2cf78d5c0b6f8306f59eaf052c8faf7cddc3 WatchSource:0}: Error finding container a71611b508fd833c32f61a19f3fd2cf78d5c0b6f8306f59eaf052c8faf7cddc3: Status 404 returned error can't find the container with id a71611b508fd833c32f61a19f3fd2cf78d5c0b6f8306f59eaf052c8faf7cddc3 Oct 31 00:13:58.060356 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:13:58.052414 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-348a33-6987f947f-cqp2m" event={"ID":"31bc5ff3-08ec-470f-b1f9-713946258110","Type":"ContainerStarted","Data":"77daa4a37e090eaed08df3549337816e2d553dd41bcefbf7a0a2eb08897d6743"} Oct 31 00:13:58.060356 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:13:58.052502 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-348a33-6987f947f-cqp2m" event={"ID":"31bc5ff3-08ec-470f-b1f9-713946258110","Type":"ContainerStarted","Data":"a71611b508fd833c32f61a19f3fd2cf78d5c0b6f8306f59eaf052c8faf7cddc3"} Oct 31 00:13:58.060356 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:13:58.052781 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-348a33-6987f947f-cqp2m" Oct 31 00:13:58.178293 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:13:58.178226 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-348a33-6987f947f-cqp2m" podStartSLOduration=3.178205918 podStartE2EDuration="3.178205918s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:13:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:13:58.173676394 +0000 UTC m=+1951.210007217" watchObservedRunningTime="2026-10-31 00:13:58.178205918 +0000 UTC m=+1951.214536741" Oct 31 00:13:58.589423 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:13:58.589385 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:13:59.015475 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:13:59.015437 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:13:59.948497 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:13:59.948463 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:14:00.401795 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:00.401697 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:14:01.007424 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:01.007386 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-460d4e-openshift-default-575d5fbfb7-ftjgz"] Oct 31 00:14:01.012532 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:01.012508 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-460d4e-openshift-default-575d5fbfb7-ftjgz" Oct 31 00:14:01.051012 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:01.050973 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d058674e-2706-41c4-8495-d7005ac1f562-istiod-ca-cert\") pod \"e2e-inf-b-460d4e-openshift-default-575d5fbfb7-ftjgz\" (UID: \"d058674e-2706-41c4-8495-d7005ac1f562\") " pod="openshift-ingress/e2e-inf-b-460d4e-openshift-default-575d5fbfb7-ftjgz" Oct 31 00:14:01.051854 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:01.051827 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d058674e-2706-41c4-8495-d7005ac1f562-credential-socket\") pod \"e2e-inf-b-460d4e-openshift-default-575d5fbfb7-ftjgz\" (UID: \"d058674e-2706-41c4-8495-d7005ac1f562\") " pod="openshift-ingress/e2e-inf-b-460d4e-openshift-default-575d5fbfb7-ftjgz" Oct 31 00:14:01.052208 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:01.052187 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f6rfl\" (UniqueName: \"kubernetes.io/projected/d058674e-2706-41c4-8495-d7005ac1f562-kube-api-access-f6rfl\") pod \"e2e-inf-b-460d4e-openshift-default-575d5fbfb7-ftjgz\" (UID: \"d058674e-2706-41c4-8495-d7005ac1f562\") " pod="openshift-ingress/e2e-inf-b-460d4e-openshift-default-575d5fbfb7-ftjgz" Oct 31 00:14:01.052439 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:01.052419 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d058674e-2706-41c4-8495-d7005ac1f562-istio-token\") pod \"e2e-inf-b-460d4e-openshift-default-575d5fbfb7-ftjgz\" (UID: \"d058674e-2706-41c4-8495-d7005ac1f562\") " pod="openshift-ingress/e2e-inf-b-460d4e-openshift-default-575d5fbfb7-ftjgz" Oct 31 00:14:01.052667 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:01.052652 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d058674e-2706-41c4-8495-d7005ac1f562-istio-data\") pod \"e2e-inf-b-460d4e-openshift-default-575d5fbfb7-ftjgz\" (UID: \"d058674e-2706-41c4-8495-d7005ac1f562\") " pod="openshift-ingress/e2e-inf-b-460d4e-openshift-default-575d5fbfb7-ftjgz" Oct 31 00:14:01.052893 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:01.052864 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d058674e-2706-41c4-8495-d7005ac1f562-istio-podinfo\") pod \"e2e-inf-b-460d4e-openshift-default-575d5fbfb7-ftjgz\" (UID: \"d058674e-2706-41c4-8495-d7005ac1f562\") " pod="openshift-ingress/e2e-inf-b-460d4e-openshift-default-575d5fbfb7-ftjgz" Oct 31 00:14:01.053080 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:01.053056 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d058674e-2706-41c4-8495-d7005ac1f562-workload-certs\") pod \"e2e-inf-b-460d4e-openshift-default-575d5fbfb7-ftjgz\" (UID: \"d058674e-2706-41c4-8495-d7005ac1f562\") " pod="openshift-ingress/e2e-inf-b-460d4e-openshift-default-575d5fbfb7-ftjgz" Oct 31 00:14:01.053248 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:01.053232 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d058674e-2706-41c4-8495-d7005ac1f562-workload-socket\") pod \"e2e-inf-b-460d4e-openshift-default-575d5fbfb7-ftjgz\" (UID: \"d058674e-2706-41c4-8495-d7005ac1f562\") " pod="openshift-ingress/e2e-inf-b-460d4e-openshift-default-575d5fbfb7-ftjgz" Oct 31 00:14:01.053431 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:01.053412 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d058674e-2706-41c4-8495-d7005ac1f562-istio-envoy\") pod \"e2e-inf-b-460d4e-openshift-default-575d5fbfb7-ftjgz\" (UID: \"d058674e-2706-41c4-8495-d7005ac1f562\") " pod="openshift-ingress/e2e-inf-b-460d4e-openshift-default-575d5fbfb7-ftjgz" Oct 31 00:14:01.070402 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:01.070368 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-460d4e-openshift-default-dockercfg-48mld\"" Oct 31 00:14:01.195831 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:01.195753 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-71b76a-64f77d669f-hm2nn" podUID="9612233d-027a-49e1-bb03-c9dc4f8ecb6b" containerName="payload-pre-processing" containerID="cri-o://3e1e344c5b15eadaedb6d60603ae37e296b562951294bc6606209b7eaa47b228" gracePeriod=25 Oct 31 00:14:01.216070 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:01.198159 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d058674e-2706-41c4-8495-d7005ac1f562-istiod-ca-cert\") pod \"e2e-inf-b-460d4e-openshift-default-575d5fbfb7-ftjgz\" (UID: \"d058674e-2706-41c4-8495-d7005ac1f562\") " pod="openshift-ingress/e2e-inf-b-460d4e-openshift-default-575d5fbfb7-ftjgz" Oct 31 00:14:01.354413 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:01.314442 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d058674e-2706-41c4-8495-d7005ac1f562-credential-socket\") pod \"e2e-inf-b-460d4e-openshift-default-575d5fbfb7-ftjgz\" (UID: \"d058674e-2706-41c4-8495-d7005ac1f562\") " pod="openshift-ingress/e2e-inf-b-460d4e-openshift-default-575d5fbfb7-ftjgz" Oct 31 00:14:01.354413 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:01.314564 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6rfl\" (UniqueName: \"kubernetes.io/projected/d058674e-2706-41c4-8495-d7005ac1f562-kube-api-access-f6rfl\") pod \"e2e-inf-b-460d4e-openshift-default-575d5fbfb7-ftjgz\" (UID: \"d058674e-2706-41c4-8495-d7005ac1f562\") " pod="openshift-ingress/e2e-inf-b-460d4e-openshift-default-575d5fbfb7-ftjgz" Oct 31 00:14:01.354413 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:01.314623 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d058674e-2706-41c4-8495-d7005ac1f562-istio-token\") pod \"e2e-inf-b-460d4e-openshift-default-575d5fbfb7-ftjgz\" (UID: \"d058674e-2706-41c4-8495-d7005ac1f562\") " pod="openshift-ingress/e2e-inf-b-460d4e-openshift-default-575d5fbfb7-ftjgz" Oct 31 00:14:01.354413 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:01.314730 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d058674e-2706-41c4-8495-d7005ac1f562-istio-data\") pod \"e2e-inf-b-460d4e-openshift-default-575d5fbfb7-ftjgz\" (UID: \"d058674e-2706-41c4-8495-d7005ac1f562\") " pod="openshift-ingress/e2e-inf-b-460d4e-openshift-default-575d5fbfb7-ftjgz" Oct 31 00:14:01.354413 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:01.314831 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d058674e-2706-41c4-8495-d7005ac1f562-istio-podinfo\") pod \"e2e-inf-b-460d4e-openshift-default-575d5fbfb7-ftjgz\" (UID: \"d058674e-2706-41c4-8495-d7005ac1f562\") " pod="openshift-ingress/e2e-inf-b-460d4e-openshift-default-575d5fbfb7-ftjgz" Oct 31 00:14:01.354413 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:01.314861 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d058674e-2706-41c4-8495-d7005ac1f562-workload-certs\") pod \"e2e-inf-b-460d4e-openshift-default-575d5fbfb7-ftjgz\" (UID: \"d058674e-2706-41c4-8495-d7005ac1f562\") " pod="openshift-ingress/e2e-inf-b-460d4e-openshift-default-575d5fbfb7-ftjgz" Oct 31 00:14:01.354413 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:01.314915 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d058674e-2706-41c4-8495-d7005ac1f562-workload-socket\") pod \"e2e-inf-b-460d4e-openshift-default-575d5fbfb7-ftjgz\" (UID: \"d058674e-2706-41c4-8495-d7005ac1f562\") " pod="openshift-ingress/e2e-inf-b-460d4e-openshift-default-575d5fbfb7-ftjgz" Oct 31 00:14:01.354413 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:01.314938 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d058674e-2706-41c4-8495-d7005ac1f562-istio-envoy\") pod \"e2e-inf-b-460d4e-openshift-default-575d5fbfb7-ftjgz\" (UID: \"d058674e-2706-41c4-8495-d7005ac1f562\") " pod="openshift-ingress/e2e-inf-b-460d4e-openshift-default-575d5fbfb7-ftjgz" Oct 31 00:14:01.354413 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:01.329665 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d058674e-2706-41c4-8495-d7005ac1f562-istio-data\") pod \"e2e-inf-b-460d4e-openshift-default-575d5fbfb7-ftjgz\" (UID: \"d058674e-2706-41c4-8495-d7005ac1f562\") " pod="openshift-ingress/e2e-inf-b-460d4e-openshift-default-575d5fbfb7-ftjgz" Oct 31 00:14:01.354413 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:01.214633 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d058674e-2706-41c4-8495-d7005ac1f562-istiod-ca-cert\") pod \"e2e-inf-b-460d4e-openshift-default-575d5fbfb7-ftjgz\" (UID: \"d058674e-2706-41c4-8495-d7005ac1f562\") " pod="openshift-ingress/e2e-inf-b-460d4e-openshift-default-575d5fbfb7-ftjgz" Oct 31 00:14:01.354413 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:01.330735 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d058674e-2706-41c4-8495-d7005ac1f562-credential-socket\") pod \"e2e-inf-b-460d4e-openshift-default-575d5fbfb7-ftjgz\" (UID: \"d058674e-2706-41c4-8495-d7005ac1f562\") " pod="openshift-ingress/e2e-inf-b-460d4e-openshift-default-575d5fbfb7-ftjgz" Oct 31 00:14:01.354413 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:01.224781 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:14:01.354413 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:01.332531 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d058674e-2706-41c4-8495-d7005ac1f562-workload-certs\") pod \"e2e-inf-b-460d4e-openshift-default-575d5fbfb7-ftjgz\" (UID: \"d058674e-2706-41c4-8495-d7005ac1f562\") " pod="openshift-ingress/e2e-inf-b-460d4e-openshift-default-575d5fbfb7-ftjgz" Oct 31 00:14:01.354413 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:01.332866 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d058674e-2706-41c4-8495-d7005ac1f562-workload-socket\") pod \"e2e-inf-b-460d4e-openshift-default-575d5fbfb7-ftjgz\" (UID: \"d058674e-2706-41c4-8495-d7005ac1f562\") " pod="openshift-ingress/e2e-inf-b-460d4e-openshift-default-575d5fbfb7-ftjgz" Oct 31 00:14:01.366625 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:01.366585 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-460d4e-openshift-default-575d5fbfb7-ftjgz"] Oct 31 00:14:01.423411 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:01.423043 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d058674e-2706-41c4-8495-d7005ac1f562-istio-envoy\") pod \"e2e-inf-b-460d4e-openshift-default-575d5fbfb7-ftjgz\" (UID: \"d058674e-2706-41c4-8495-d7005ac1f562\") " pod="openshift-ingress/e2e-inf-b-460d4e-openshift-default-575d5fbfb7-ftjgz" Oct 31 00:14:01.442316 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:01.442242 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6rfl\" (UniqueName: \"kubernetes.io/projected/d058674e-2706-41c4-8495-d7005ac1f562-kube-api-access-f6rfl\") pod \"e2e-inf-b-460d4e-openshift-default-575d5fbfb7-ftjgz\" (UID: \"d058674e-2706-41c4-8495-d7005ac1f562\") " pod="openshift-ingress/e2e-inf-b-460d4e-openshift-default-575d5fbfb7-ftjgz" Oct 31 00:14:01.592108 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:01.591969 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:14:01.596283 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:01.596192 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d058674e-2706-41c4-8495-d7005ac1f562-istio-podinfo\") pod \"e2e-inf-b-460d4e-openshift-default-575d5fbfb7-ftjgz\" (UID: \"d058674e-2706-41c4-8495-d7005ac1f562\") " pod="openshift-ingress/e2e-inf-b-460d4e-openshift-default-575d5fbfb7-ftjgz" Oct 31 00:14:01.612488 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:01.612446 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d058674e-2706-41c4-8495-d7005ac1f562-istio-token\") pod \"e2e-inf-b-460d4e-openshift-default-575d5fbfb7-ftjgz\" (UID: \"d058674e-2706-41c4-8495-d7005ac1f562\") " pod="openshift-ingress/e2e-inf-b-460d4e-openshift-default-575d5fbfb7-ftjgz" Oct 31 00:14:01.627824 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:01.627616 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-460d4e-openshift-default-575d5fbfb7-ftjgz" Oct 31 00:14:01.818148 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:01.818115 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-71b76a-64f77d669f-hm2nn" Oct 31 00:14:01.956535 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:01.956497 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9612233d-027a-49e1-bb03-c9dc4f8ecb6b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9612233d-027a-49e1-bb03-c9dc4f8ecb6b-plugins-config-volume\") pod \"9612233d-027a-49e1-bb03-c9dc4f8ecb6b\" (UID: \"9612233d-027a-49e1-bb03-c9dc4f8ecb6b\") " Oct 31 00:14:01.956741 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:01.956604 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9612233d-027a-49e1-bb03-c9dc4f8ecb6b-kube-api-access-48fdn\" (UniqueName: \"kubernetes.io/projected/9612233d-027a-49e1-bb03-c9dc4f8ecb6b-kube-api-access-48fdn\") pod \"9612233d-027a-49e1-bb03-c9dc4f8ecb6b\" (UID: \"9612233d-027a-49e1-bb03-c9dc4f8ecb6b\") " Oct 31 00:14:01.957008 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:01.956977 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9612233d-027a-49e1-bb03-c9dc4f8ecb6b-plugins-config-volume" pod "9612233d-027a-49e1-bb03-c9dc4f8ecb6b" (UID: "9612233d-027a-49e1-bb03-c9dc4f8ecb6b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:14:01.957224 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:01.957202 4927 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9612233d-027a-49e1-bb03-c9dc4f8ecb6b-plugins-config-volume\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:14:01.959311 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:01.959285 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9612233d-027a-49e1-bb03-c9dc4f8ecb6b-kube-api-access-48fdn" pod "9612233d-027a-49e1-bb03-c9dc4f8ecb6b" (UID: "9612233d-027a-49e1-bb03-c9dc4f8ecb6b"). InnerVolumeSpecName "kube-api-access-48fdn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:14:02.058234 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:02.058185 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-48fdn\" (UniqueName: \"kubernetes.io/projected/9612233d-027a-49e1-bb03-c9dc4f8ecb6b-kube-api-access-48fdn\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:14:02.112284 ip-10-0-141-180 kubenswrapper[4927]: W1031 00:14:02.112253 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd058674e_2706_41c4_8495_d7005ac1f562.slice/crio-95945db155b79d9dfff9785f9dd1e0b9c807ee1831c3f7c07214a4d2f5146fac WatchSource:0}: Error finding container 95945db155b79d9dfff9785f9dd1e0b9c807ee1831c3f7c07214a4d2f5146fac: Status 404 returned error can't find the container with id 95945db155b79d9dfff9785f9dd1e0b9c807ee1831c3f7c07214a4d2f5146fac Oct 31 00:14:02.112963 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:02.112931 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-460d4e-openshift-default-575d5fbfb7-ftjgz"] Oct 31 00:14:02.115130 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:02.115092 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:14:02.115226 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:02.115175 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:14:02.115226 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:02.115213 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:14:02.309976 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:02.309858 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:14:02.470843 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:02.470794 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:14:02.480851 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:02.480819 4927 generic.go:356] "Generic (PLEG): container finished" podID="9612233d-027a-49e1-bb03-c9dc4f8ecb6b" containerID="3e1e344c5b15eadaedb6d60603ae37e296b562951294bc6606209b7eaa47b228" exitCode=0 Oct 31 00:14:02.480994 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:02.480873 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-71b76a-64f77d669f-hm2nn" Oct 31 00:14:02.480994 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:02.480895 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-71b76a-64f77d669f-hm2nn" event={"ID":"9612233d-027a-49e1-bb03-c9dc4f8ecb6b","Type":"ContainerDied","Data":"3e1e344c5b15eadaedb6d60603ae37e296b562951294bc6606209b7eaa47b228"} Oct 31 00:14:02.480994 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:02.480951 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-71b76a-64f77d669f-hm2nn" event={"ID":"9612233d-027a-49e1-bb03-c9dc4f8ecb6b","Type":"ContainerDied","Data":"77344b1604184daed19b299ebe663a5508b2d1190c28b744609b466c64b4a6ff"} Oct 31 00:14:02.480994 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:02.480977 4927 scope.go:122] "RemoveContainer" containerID="3e1e344c5b15eadaedb6d60603ae37e296b562951294bc6606209b7eaa47b228" Oct 31 00:14:02.483160 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:02.483134 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-460d4e-openshift-default-575d5fbfb7-ftjgz" event={"ID":"d058674e-2706-41c4-8495-d7005ac1f562","Type":"ContainerStarted","Data":"7ff637e1e8bfb168ede519666b1271d01cd05dc174570cadd1fb02b573ac1d6d"} Oct 31 00:14:02.483292 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:02.483193 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-460d4e-openshift-default-575d5fbfb7-ftjgz" event={"ID":"d058674e-2706-41c4-8495-d7005ac1f562","Type":"ContainerStarted","Data":"95945db155b79d9dfff9785f9dd1e0b9c807ee1831c3f7c07214a4d2f5146fac"} Oct 31 00:14:02.493078 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:02.493057 4927 scope.go:122] "RemoveContainer" containerID="3e1e344c5b15eadaedb6d60603ae37e296b562951294bc6606209b7eaa47b228" Oct 31 00:14:02.493393 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:14:02.493372 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e1e344c5b15eadaedb6d60603ae37e296b562951294bc6606209b7eaa47b228\": container with ID starting with 3e1e344c5b15eadaedb6d60603ae37e296b562951294bc6606209b7eaa47b228 not found: ID does not exist" containerID="3e1e344c5b15eadaedb6d60603ae37e296b562951294bc6606209b7eaa47b228" Oct 31 00:14:02.493448 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:02.493400 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e1e344c5b15eadaedb6d60603ae37e296b562951294bc6606209b7eaa47b228"} err="failed to get container status \"3e1e344c5b15eadaedb6d60603ae37e296b562951294bc6606209b7eaa47b228\": rpc error: code = NotFound desc = could not find container \"3e1e344c5b15eadaedb6d60603ae37e296b562951294bc6606209b7eaa47b228\": container with ID starting with 3e1e344c5b15eadaedb6d60603ae37e296b562951294bc6606209b7eaa47b228 not found: ID does not exist" Oct 31 00:14:02.628083 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:02.627989 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-460d4e-openshift-default-575d5fbfb7-ftjgz" Oct 31 00:14:02.658934 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:02.658871 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-460d4e-openshift-default-575d5fbfb7-ftjgz" podStartSLOduration=2.658857224 podStartE2EDuration="2.658857224s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:14:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:14:02.657457112 +0000 UTC m=+1955.693787938" watchObservedRunningTime="2026-10-31 00:14:02.658857224 +0000 UTC m=+1955.695188123" Oct 31 00:14:02.845414 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:02.842229 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-71b76a-64f77d669f-hm2nn"] Oct 31 00:14:02.869357 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:02.868827 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-71b76a-64f77d669f-hm2nn"] Oct 31 00:14:03.088419 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:03.088386 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9612233d-027a-49e1-bb03-c9dc4f8ecb6b" path="/var/lib/kubelet/pods/9612233d-027a-49e1-bb03-c9dc4f8ecb6b/volumes" Oct 31 00:14:03.635102 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:03.635067 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-460d4e-openshift-default-575d5fbfb7-ftjgz" Oct 31 00:14:03.975731 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:03.975698 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:14:04.503473 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:04.503442 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-460d4e-openshift-default-575d5fbfb7-ftjgz" Oct 31 00:14:04.504554 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:04.504530 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-460d4e-openshift-default-575d5fbfb7-ftjgz" Oct 31 00:14:06.175315 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:06.175281 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:14:06.393277 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:06.393239 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:14:06.740627 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:06.726000 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:14:06.915355 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:06.909009 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:14:07.119021 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:07.118894 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:14:07.293200 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:07.292638 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:14:09.064387 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:09.064326 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:14:09.175882 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:09.175842 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-348a33-6987f947f-cqp2m" Oct 31 00:14:11.026857 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:11.026820 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:14:11.293407 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:11.285054 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:14:11.758353 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:11.749196 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-460d4e-6457b77684-dttcx"] Oct 31 00:14:11.758353 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:11.750675 4927 cpu_manager.go:414] "containerMap: removing container" podUID="9612233d-027a-49e1-bb03-c9dc4f8ecb6b" containerName="payload-pre-processing" Oct 31 00:14:11.758353 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:11.750699 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9612233d-027a-49e1-bb03-c9dc4f8ecb6b" containerName="payload-pre-processing" Oct 31 00:14:11.758353 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:11.751019 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="9612233d-027a-49e1-bb03-c9dc4f8ecb6b" containerName="payload-pre-processing" Oct 31 00:14:11.803619 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:11.803580 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-460d4e-6457b77684-dttcx"] Oct 31 00:14:11.803815 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:11.803725 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-460d4e-6457b77684-dttcx" Oct 31 00:14:11.858487 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:11.858461 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-460d4e\"" Oct 31 00:14:11.858657 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:11.858503 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-460d4e-dockercfg-wrxbk\"" Oct 31 00:14:12.002192 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:12.002156 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbe18c5f-bf04-448b-8385-e99804d6056d-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-460d4e-6457b77684-dttcx\" (UID: \"bbe18c5f-bf04-448b-8385-e99804d6056d\") " pod="openshift-ingress/payload-processing-e2e-inf-b-460d4e-6457b77684-dttcx" Oct 31 00:14:12.002421 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:12.002223 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wnx6x\" (UniqueName: \"kubernetes.io/projected/bbe18c5f-bf04-448b-8385-e99804d6056d-kube-api-access-wnx6x\") pod \"payload-processing-e2e-inf-b-460d4e-6457b77684-dttcx\" (UID: \"bbe18c5f-bf04-448b-8385-e99804d6056d\") " pod="openshift-ingress/payload-processing-e2e-inf-b-460d4e-6457b77684-dttcx" Oct 31 00:14:12.103725 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:12.103608 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbe18c5f-bf04-448b-8385-e99804d6056d-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-460d4e-6457b77684-dttcx\" (UID: \"bbe18c5f-bf04-448b-8385-e99804d6056d\") " pod="openshift-ingress/payload-processing-e2e-inf-b-460d4e-6457b77684-dttcx" Oct 31 00:14:12.103725 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:12.103715 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wnx6x\" (UniqueName: \"kubernetes.io/projected/bbe18c5f-bf04-448b-8385-e99804d6056d-kube-api-access-wnx6x\") pod \"payload-processing-e2e-inf-b-460d4e-6457b77684-dttcx\" (UID: \"bbe18c5f-bf04-448b-8385-e99804d6056d\") " pod="openshift-ingress/payload-processing-e2e-inf-b-460d4e-6457b77684-dttcx" Oct 31 00:14:12.104654 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:12.104587 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbe18c5f-bf04-448b-8385-e99804d6056d-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-460d4e-6457b77684-dttcx\" (UID: \"bbe18c5f-bf04-448b-8385-e99804d6056d\") " pod="openshift-ingress/payload-processing-e2e-inf-b-460d4e-6457b77684-dttcx" Oct 31 00:14:12.166405 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:12.166368 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wnx6x\" (UniqueName: \"kubernetes.io/projected/bbe18c5f-bf04-448b-8385-e99804d6056d-kube-api-access-wnx6x\") pod \"payload-processing-e2e-inf-b-460d4e-6457b77684-dttcx\" (UID: \"bbe18c5f-bf04-448b-8385-e99804d6056d\") " pod="openshift-ingress/payload-processing-e2e-inf-b-460d4e-6457b77684-dttcx" Oct 31 00:14:12.427409 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:12.427376 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-460d4e-6457b77684-dttcx" Oct 31 00:14:12.823314 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:12.823280 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-460d4e-6457b77684-dttcx"] Oct 31 00:14:12.823505 ip-10-0-141-180 kubenswrapper[4927]: W1031 00:14:12.823483 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbe18c5f_bf04_448b_8385_e99804d6056d.slice/crio-48c8ff945fac5dd851c8966db2edcdc777eeabab14bf7dfb09e3fae28f271d9d WatchSource:0}: Error finding container 48c8ff945fac5dd851c8966db2edcdc777eeabab14bf7dfb09e3fae28f271d9d: Status 404 returned error can't find the container with id 48c8ff945fac5dd851c8966db2edcdc777eeabab14bf7dfb09e3fae28f271d9d Oct 31 00:14:13.848590 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:13.848528 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-460d4e-6457b77684-dttcx" event={"ID":"bbe18c5f-bf04-448b-8385-e99804d6056d","Type":"ContainerStarted","Data":"afb90c351598ccb99abd88d34a8edf2a35aac81d8551d9cdd51003840808454b"} Oct 31 00:14:13.849091 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:13.848618 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-460d4e-6457b77684-dttcx" event={"ID":"bbe18c5f-bf04-448b-8385-e99804d6056d","Type":"ContainerStarted","Data":"48c8ff945fac5dd851c8966db2edcdc777eeabab14bf7dfb09e3fae28f271d9d"} Oct 31 00:14:13.849091 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:13.848663 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-460d4e-6457b77684-dttcx" Oct 31 00:14:13.937464 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:13.937401 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:14:14.004318 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:14.004241 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-460d4e-6457b77684-dttcx" podStartSLOduration=3.004216975 podStartE2EDuration="3.004216975s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:14:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:14:13.967115875 +0000 UTC m=+1967.003446723" watchObservedRunningTime="2026-10-31 00:14:14.004216975 +0000 UTC m=+1967.040547797" Oct 31 00:14:14.186015 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:14.185979 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:14:14.361283 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:14.361249 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:14:18.656184 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:18.656138 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:14:18.926900 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:18.926867 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:14:19.679291 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:14:19.679256 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sqv69" containerName="extractor" Oct 31 00:14:19.907171 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:19.907093 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:14:21.345347 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:21.345304 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-c9c21e-openshift-default-749bcfc96c-wjzsh"] Oct 31 00:14:21.351735 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:21.351706 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-c9c21e-openshift-default-749bcfc96c-wjzsh" Oct 31 00:14:21.358781 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:21.358748 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3898ce0c-61e4-4888-9e54-3185c86a5326-istio-podinfo\") pod \"e2e-api-a-c9c21e-openshift-default-749bcfc96c-wjzsh\" (UID: \"3898ce0c-61e4-4888-9e54-3185c86a5326\") " pod="openshift-ingress/e2e-api-a-c9c21e-openshift-default-749bcfc96c-wjzsh" Oct 31 00:14:21.358934 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:21.358801 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3898ce0c-61e4-4888-9e54-3185c86a5326-workload-socket\") pod \"e2e-api-a-c9c21e-openshift-default-749bcfc96c-wjzsh\" (UID: \"3898ce0c-61e4-4888-9e54-3185c86a5326\") " pod="openshift-ingress/e2e-api-a-c9c21e-openshift-default-749bcfc96c-wjzsh" Oct 31 00:14:21.358934 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:21.358832 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3898ce0c-61e4-4888-9e54-3185c86a5326-workload-certs\") pod \"e2e-api-a-c9c21e-openshift-default-749bcfc96c-wjzsh\" (UID: \"3898ce0c-61e4-4888-9e54-3185c86a5326\") " pod="openshift-ingress/e2e-api-a-c9c21e-openshift-default-749bcfc96c-wjzsh" Oct 31 00:14:21.358934 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:21.358897 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3898ce0c-61e4-4888-9e54-3185c86a5326-credential-socket\") pod \"e2e-api-a-c9c21e-openshift-default-749bcfc96c-wjzsh\" (UID: \"3898ce0c-61e4-4888-9e54-3185c86a5326\") " pod="openshift-ingress/e2e-api-a-c9c21e-openshift-default-749bcfc96c-wjzsh" Oct 31 00:14:21.358934 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:21.358920 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3898ce0c-61e4-4888-9e54-3185c86a5326-istio-token\") pod \"e2e-api-a-c9c21e-openshift-default-749bcfc96c-wjzsh\" (UID: \"3898ce0c-61e4-4888-9e54-3185c86a5326\") " pod="openshift-ingress/e2e-api-a-c9c21e-openshift-default-749bcfc96c-wjzsh" Oct 31 00:14:21.359160 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:21.359000 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2mlm\" (UniqueName: \"kubernetes.io/projected/3898ce0c-61e4-4888-9e54-3185c86a5326-kube-api-access-j2mlm\") pod \"e2e-api-a-c9c21e-openshift-default-749bcfc96c-wjzsh\" (UID: \"3898ce0c-61e4-4888-9e54-3185c86a5326\") " pod="openshift-ingress/e2e-api-a-c9c21e-openshift-default-749bcfc96c-wjzsh" Oct 31 00:14:21.359160 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:21.359044 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3898ce0c-61e4-4888-9e54-3185c86a5326-istiod-ca-cert\") pod \"e2e-api-a-c9c21e-openshift-default-749bcfc96c-wjzsh\" (UID: \"3898ce0c-61e4-4888-9e54-3185c86a5326\") " pod="openshift-ingress/e2e-api-a-c9c21e-openshift-default-749bcfc96c-wjzsh" Oct 31 00:14:21.359160 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:21.359096 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3898ce0c-61e4-4888-9e54-3185c86a5326-istio-envoy\") pod \"e2e-api-a-c9c21e-openshift-default-749bcfc96c-wjzsh\" (UID: \"3898ce0c-61e4-4888-9e54-3185c86a5326\") " pod="openshift-ingress/e2e-api-a-c9c21e-openshift-default-749bcfc96c-wjzsh" Oct 31 00:14:21.359160 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:21.359110 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3898ce0c-61e4-4888-9e54-3185c86a5326-istio-data\") pod \"e2e-api-a-c9c21e-openshift-default-749bcfc96c-wjzsh\" (UID: \"3898ce0c-61e4-4888-9e54-3185c86a5326\") " pod="openshift-ingress/e2e-api-a-c9c21e-openshift-default-749bcfc96c-wjzsh" Oct 31 00:14:21.405739 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:21.405704 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-c9c21e-openshift-default-dockercfg-4rql9\"" Oct 31 00:14:21.461443 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:21.460185 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3898ce0c-61e4-4888-9e54-3185c86a5326-istio-token\") pod \"e2e-api-a-c9c21e-openshift-default-749bcfc96c-wjzsh\" (UID: \"3898ce0c-61e4-4888-9e54-3185c86a5326\") " pod="openshift-ingress/e2e-api-a-c9c21e-openshift-default-749bcfc96c-wjzsh" Oct 31 00:14:21.461443 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:21.460316 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2mlm\" (UniqueName: \"kubernetes.io/projected/3898ce0c-61e4-4888-9e54-3185c86a5326-kube-api-access-j2mlm\") pod \"e2e-api-a-c9c21e-openshift-default-749bcfc96c-wjzsh\" (UID: \"3898ce0c-61e4-4888-9e54-3185c86a5326\") " pod="openshift-ingress/e2e-api-a-c9c21e-openshift-default-749bcfc96c-wjzsh" Oct 31 00:14:21.461443 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:21.460381 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3898ce0c-61e4-4888-9e54-3185c86a5326-istiod-ca-cert\") pod \"e2e-api-a-c9c21e-openshift-default-749bcfc96c-wjzsh\" (UID: \"3898ce0c-61e4-4888-9e54-3185c86a5326\") " pod="openshift-ingress/e2e-api-a-c9c21e-openshift-default-749bcfc96c-wjzsh" Oct 31 00:14:21.461443 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:21.460449 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3898ce0c-61e4-4888-9e54-3185c86a5326-istio-envoy\") pod \"e2e-api-a-c9c21e-openshift-default-749bcfc96c-wjzsh\" (UID: \"3898ce0c-61e4-4888-9e54-3185c86a5326\") " pod="openshift-ingress/e2e-api-a-c9c21e-openshift-default-749bcfc96c-wjzsh" Oct 31 00:14:21.461443 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:21.460471 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3898ce0c-61e4-4888-9e54-3185c86a5326-istio-data\") pod \"e2e-api-a-c9c21e-openshift-default-749bcfc96c-wjzsh\" (UID: \"3898ce0c-61e4-4888-9e54-3185c86a5326\") " pod="openshift-ingress/e2e-api-a-c9c21e-openshift-default-749bcfc96c-wjzsh" Oct 31 00:14:21.461443 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:21.460558 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3898ce0c-61e4-4888-9e54-3185c86a5326-istio-podinfo\") pod \"e2e-api-a-c9c21e-openshift-default-749bcfc96c-wjzsh\" (UID: \"3898ce0c-61e4-4888-9e54-3185c86a5326\") " pod="openshift-ingress/e2e-api-a-c9c21e-openshift-default-749bcfc96c-wjzsh" Oct 31 00:14:21.461443 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:21.460598 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3898ce0c-61e4-4888-9e54-3185c86a5326-workload-socket\") pod \"e2e-api-a-c9c21e-openshift-default-749bcfc96c-wjzsh\" (UID: \"3898ce0c-61e4-4888-9e54-3185c86a5326\") " pod="openshift-ingress/e2e-api-a-c9c21e-openshift-default-749bcfc96c-wjzsh" Oct 31 00:14:21.461443 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:21.460637 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3898ce0c-61e4-4888-9e54-3185c86a5326-workload-certs\") pod \"e2e-api-a-c9c21e-openshift-default-749bcfc96c-wjzsh\" (UID: \"3898ce0c-61e4-4888-9e54-3185c86a5326\") " pod="openshift-ingress/e2e-api-a-c9c21e-openshift-default-749bcfc96c-wjzsh" Oct 31 00:14:21.461443 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:21.460717 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3898ce0c-61e4-4888-9e54-3185c86a5326-credential-socket\") pod \"e2e-api-a-c9c21e-openshift-default-749bcfc96c-wjzsh\" (UID: \"3898ce0c-61e4-4888-9e54-3185c86a5326\") " pod="openshift-ingress/e2e-api-a-c9c21e-openshift-default-749bcfc96c-wjzsh" Oct 31 00:14:21.461443 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:21.461084 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3898ce0c-61e4-4888-9e54-3185c86a5326-credential-socket\") pod \"e2e-api-a-c9c21e-openshift-default-749bcfc96c-wjzsh\" (UID: \"3898ce0c-61e4-4888-9e54-3185c86a5326\") " pod="openshift-ingress/e2e-api-a-c9c21e-openshift-default-749bcfc96c-wjzsh" Oct 31 00:14:21.461443 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:21.461444 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3898ce0c-61e4-4888-9e54-3185c86a5326-istiod-ca-cert\") pod \"e2e-api-a-c9c21e-openshift-default-749bcfc96c-wjzsh\" (UID: \"3898ce0c-61e4-4888-9e54-3185c86a5326\") " pod="openshift-ingress/e2e-api-a-c9c21e-openshift-default-749bcfc96c-wjzsh" Oct 31 00:14:21.462693 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:21.462367 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3898ce0c-61e4-4888-9e54-3185c86a5326-workload-socket\") pod \"e2e-api-a-c9c21e-openshift-default-749bcfc96c-wjzsh\" (UID: \"3898ce0c-61e4-4888-9e54-3185c86a5326\") " pod="openshift-ingress/e2e-api-a-c9c21e-openshift-default-749bcfc96c-wjzsh" Oct 31 00:14:21.462693 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:21.462445 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3898ce0c-61e4-4888-9e54-3185c86a5326-workload-certs\") pod \"e2e-api-a-c9c21e-openshift-default-749bcfc96c-wjzsh\" (UID: \"3898ce0c-61e4-4888-9e54-3185c86a5326\") " pod="openshift-ingress/e2e-api-a-c9c21e-openshift-default-749bcfc96c-wjzsh" Oct 31 00:14:21.462693 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:21.462636 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3898ce0c-61e4-4888-9e54-3185c86a5326-istio-data\") pod \"e2e-api-a-c9c21e-openshift-default-749bcfc96c-wjzsh\" (UID: \"3898ce0c-61e4-4888-9e54-3185c86a5326\") " pod="openshift-ingress/e2e-api-a-c9c21e-openshift-default-749bcfc96c-wjzsh" Oct 31 00:14:21.465523 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:21.465481 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3898ce0c-61e4-4888-9e54-3185c86a5326-istio-envoy\") pod \"e2e-api-a-c9c21e-openshift-default-749bcfc96c-wjzsh\" (UID: \"3898ce0c-61e4-4888-9e54-3185c86a5326\") " pod="openshift-ingress/e2e-api-a-c9c21e-openshift-default-749bcfc96c-wjzsh" Oct 31 00:14:21.466523 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:21.466498 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3898ce0c-61e4-4888-9e54-3185c86a5326-istio-podinfo\") pod \"e2e-api-a-c9c21e-openshift-default-749bcfc96c-wjzsh\" (UID: \"3898ce0c-61e4-4888-9e54-3185c86a5326\") " pod="openshift-ingress/e2e-api-a-c9c21e-openshift-default-749bcfc96c-wjzsh" Oct 31 00:14:21.533508 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:21.533401 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-c9c21e-openshift-default-749bcfc96c-wjzsh"] Oct 31 00:14:21.539405 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:21.539117 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:14:21.634566 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:21.634114 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3898ce0c-61e4-4888-9e54-3185c86a5326-istio-token\") pod \"e2e-api-a-c9c21e-openshift-default-749bcfc96c-wjzsh\" (UID: \"3898ce0c-61e4-4888-9e54-3185c86a5326\") " pod="openshift-ingress/e2e-api-a-c9c21e-openshift-default-749bcfc96c-wjzsh" Oct 31 00:14:21.642110 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:21.642040 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2mlm\" (UniqueName: \"kubernetes.io/projected/3898ce0c-61e4-4888-9e54-3185c86a5326-kube-api-access-j2mlm\") pod \"e2e-api-a-c9c21e-openshift-default-749bcfc96c-wjzsh\" (UID: \"3898ce0c-61e4-4888-9e54-3185c86a5326\") " pod="openshift-ingress/e2e-api-a-c9c21e-openshift-default-749bcfc96c-wjzsh" Oct 31 00:14:21.686355 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:21.679251 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-c9c21e-openshift-default-749bcfc96c-wjzsh" Oct 31 00:14:21.771890 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:21.767591 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:14:22.228282 ip-10-0-141-180 kubenswrapper[4927]: W1031 00:14:22.228239 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3898ce0c_61e4_4888_9e54_3185c86a5326.slice/crio-5253d1b7b1a0d9094a0d5bf28ca0f7db9a2c1c023511f7f2f8b283547e72d521 WatchSource:0}: Error finding container 5253d1b7b1a0d9094a0d5bf28ca0f7db9a2c1c023511f7f2f8b283547e72d521: Status 404 returned error can't find the container with id 5253d1b7b1a0d9094a0d5bf28ca0f7db9a2c1c023511f7f2f8b283547e72d521 Oct 31 00:14:22.228994 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:22.228964 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-c9c21e-openshift-default-749bcfc96c-wjzsh"] Oct 31 00:14:22.231910 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:22.231864 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:14:22.232018 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:22.231953 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:14:22.232018 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:22.232000 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:14:23.016394 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:23.016317 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c9c21e-openshift-default-749bcfc96c-wjzsh" event={"ID":"3898ce0c-61e4-4888-9e54-3185c86a5326","Type":"ContainerStarted","Data":"c14ecd91a809659c013aabbf0dd11ba620bb7e763cd825cc1fbdf6357bc6bc56"} Oct 31 00:14:23.016885 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:23.016424 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c9c21e-openshift-default-749bcfc96c-wjzsh" event={"ID":"3898ce0c-61e4-4888-9e54-3185c86a5326","Type":"ContainerStarted","Data":"5253d1b7b1a0d9094a0d5bf28ca0f7db9a2c1c023511f7f2f8b283547e72d521"} Oct 31 00:14:23.194046 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:23.193983 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-c9c21e-openshift-default-749bcfc96c-wjzsh" podStartSLOduration=2.193965621 podStartE2EDuration="2.193965621s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:14:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:14:23.189825594 +0000 UTC m=+1976.226156444" watchObservedRunningTime="2026-10-31 00:14:23.193965621 +0000 UTC m=+1976.230296442" Oct 31 00:14:23.541512 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:23.541459 4927 patch_prober.go:29] interesting pod/e2e-shared-b-w0-957019-openshift-default-59bc4d44f9-8xtcd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.207:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:14:23.541706 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:23.541533 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w0-957019-openshift-default-59bc4d44f9-8xtcd" podUID="4b623edb-855c-44b8-a0c1-abbd33925bd4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.207:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:14:23.680412 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:23.680372 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-c9c21e-openshift-default-749bcfc96c-wjzsh" Oct 31 00:14:23.686968 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:23.686938 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-c9c21e-openshift-default-749bcfc96c-wjzsh" Oct 31 00:14:23.701479 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:23.701435 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:14:24.029189 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:24.029162 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-c9c21e-openshift-default-749bcfc96c-wjzsh" Oct 31 00:14:24.030228 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:24.030200 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-c9c21e-openshift-default-749bcfc96c-wjzsh" Oct 31 00:14:24.862047 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:24.862020 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-460d4e-6457b77684-dttcx" Oct 31 00:14:25.057479 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:25.051825 4927 generic.go:356] "Generic (PLEG): container finished" podID="4b623edb-855c-44b8-a0c1-abbd33925bd4" containerID="cdcb32181d652df3402395552e0ae0898b41240322995bcf0f51e7ff004c2707" exitCode=0 Oct 31 00:14:25.057479 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:25.053047 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-957019-openshift-default-59bc4d44f9-8xtcd" event={"ID":"4b623edb-855c-44b8-a0c1-abbd33925bd4","Type":"ContainerDied","Data":"cdcb32181d652df3402395552e0ae0898b41240322995bcf0f51e7ff004c2707"} Oct 31 00:14:25.057479 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:25.053547 4927 scope.go:122] "RemoveContainer" containerID="cdcb32181d652df3402395552e0ae0898b41240322995bcf0f51e7ff004c2707" Oct 31 00:14:25.057479 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:25.056534 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:14:25.057479 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:25.056618 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:14:25.057479 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:25.056664 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:14:26.067121 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:26.067081 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-957019-openshift-default-59bc4d44f9-8xtcd" event={"ID":"4b623edb-855c-44b8-a0c1-abbd33925bd4","Type":"ContainerStarted","Data":"83c89c92e6e55d4905b445efe29c501f62cd664f6c65d4ea0bcba420518d3191"} Oct 31 00:14:26.292683 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:26.292647 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:14:26.386604 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:26.386533 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-957019-openshift-default-59bc4d44f9-8xtcd" Oct 31 00:14:26.386755 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:26.386696 4927 patch_prober.go:29] interesting pod/e2e-shared-b-w0-957019-openshift-default-59bc4d44f9-8xtcd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" start-of-body= Oct 31 00:14:26.386755 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:26.386744 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-957019-openshift-default-59bc4d44f9-8xtcd" podUID="4b623edb-855c-44b8-a0c1-abbd33925bd4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" Oct 31 00:14:26.504732 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:26.504693 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:14:27.387180 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:27.387148 4927 patch_prober.go:29] interesting pod/e2e-shared-b-w0-957019-openshift-default-59bc4d44f9-8xtcd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" start-of-body= Oct 31 00:14:27.387606 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:27.387201 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-957019-openshift-default-59bc4d44f9-8xtcd" podUID="4b623edb-855c-44b8-a0c1-abbd33925bd4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" Oct 31 00:14:28.386907 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:28.386864 4927 patch_prober.go:29] interesting pod/e2e-shared-b-w0-957019-openshift-default-59bc4d44f9-8xtcd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" start-of-body= Oct 31 00:14:28.387096 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:28.386935 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-957019-openshift-default-59bc4d44f9-8xtcd" podUID="4b623edb-855c-44b8-a0c1-abbd33925bd4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" Oct 31 00:14:28.628357 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:28.627371 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:14:29.176884 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:29.176844 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:14:29.374825 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:29.374789 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:14:29.387278 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:29.387244 4927 patch_prober.go:29] interesting pod/e2e-shared-b-w0-957019-openshift-default-59bc4d44f9-8xtcd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" start-of-body= Oct 31 00:14:29.387452 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:29.387315 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-957019-openshift-default-59bc4d44f9-8xtcd" podUID="4b623edb-855c-44b8-a0c1-abbd33925bd4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" Oct 31 00:14:29.535451 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:29.534877 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:14:29.742445 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:29.739857 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:14:30.457567 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:30.457529 4927 patch_prober.go:29] interesting pod/e2e-shared-b-w0-957019-openshift-default-59bc4d44f9-8xtcd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" start-of-body= Oct 31 00:14:30.458223 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:30.458090 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-957019-openshift-default-59bc4d44f9-8xtcd" podUID="4b623edb-855c-44b8-a0c1-abbd33925bd4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" Oct 31 00:14:31.386489 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:31.386455 4927 patch_prober.go:29] interesting pod/e2e-shared-b-w0-957019-openshift-default-59bc4d44f9-8xtcd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" start-of-body= Oct 31 00:14:31.386908 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:31.386510 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-957019-openshift-default-59bc4d44f9-8xtcd" podUID="4b623edb-855c-44b8-a0c1-abbd33925bd4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" Oct 31 00:14:32.391352 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:32.391312 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-957019-openshift-default-59bc4d44f9-8xtcd" Oct 31 00:14:32.391796 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:32.391778 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-957019-openshift-default-59bc4d44f9-8xtcd" Oct 31 00:14:32.392840 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:32.392815 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-957019-openshift-default-59bc4d44f9-8xtcd" Oct 31 00:14:32.540203 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:32.540166 4927 scope.go:122] "RemoveContainer" containerID="3d26b0780840c981c1c50310e82bbea3d89cae5e5e35ee90328360c736e85ced" Oct 31 00:14:32.553219 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:32.553194 4927 scope.go:122] "RemoveContainer" containerID="a8bf905ff0c4c6178d3399e1f4d9e62a66d247667635d1123edd168d82f9162e" Oct 31 00:14:32.563722 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:32.563696 4927 scope.go:122] "RemoveContainer" containerID="09df47c3c1ba9856824b0c38fa11576d0ca01e8f24f94c845879102727695432" Oct 31 00:14:32.573536 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:32.573516 4927 scope.go:122] "RemoveContainer" containerID="c0300d5394dd3988178b39aa232395473496f27563c61f79ff1baca9b0574276" Oct 31 00:14:33.600760 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:33.600728 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:14:34.983942 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:34.983909 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-t4ms2"] Oct 31 00:14:34.984488 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:34.984159 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-t4ms2" podUID="1822bec6-cd20-4045-b362-d83766f3337b" containerName="istio-proxy" containerID="cri-o://52c84a877ba688ff497a00fa0c370da126a9e699c2f21bd029c9a50e48ec50cf" gracePeriod=30 Oct 31 00:14:38.865521 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:38.865481 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:14:40.285270 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:40.285240 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-t4ms2" Oct 31 00:14:40.452315 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:40.452256 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1822bec6-cd20-4045-b362-d83766f3337b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1822bec6-cd20-4045-b362-d83766f3337b-credential-socket\") pod \"1822bec6-cd20-4045-b362-d83766f3337b\" (UID: \"1822bec6-cd20-4045-b362-d83766f3337b\") " Oct 31 00:14:40.453373 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:40.452391 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1822bec6-cd20-4045-b362-d83766f3337b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1822bec6-cd20-4045-b362-d83766f3337b-workload-certs\") pod \"1822bec6-cd20-4045-b362-d83766f3337b\" (UID: \"1822bec6-cd20-4045-b362-d83766f3337b\") " Oct 31 00:14:40.453373 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:40.452498 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1822bec6-cd20-4045-b362-d83766f3337b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1822bec6-cd20-4045-b362-d83766f3337b-istiod-ca-cert\") pod \"1822bec6-cd20-4045-b362-d83766f3337b\" (UID: \"1822bec6-cd20-4045-b362-d83766f3337b\") " Oct 31 00:14:40.453373 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:40.452643 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1822bec6-cd20-4045-b362-d83766f3337b-kube-api-access-9jvrk\" (UniqueName: \"kubernetes.io/projected/1822bec6-cd20-4045-b362-d83766f3337b-kube-api-access-9jvrk\") pod \"1822bec6-cd20-4045-b362-d83766f3337b\" (UID: \"1822bec6-cd20-4045-b362-d83766f3337b\") " Oct 31 00:14:40.453373 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:40.452685 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1822bec6-cd20-4045-b362-d83766f3337b-istio-token\" (UniqueName: \"kubernetes.io/projected/1822bec6-cd20-4045-b362-d83766f3337b-istio-token\") pod \"1822bec6-cd20-4045-b362-d83766f3337b\" (UID: \"1822bec6-cd20-4045-b362-d83766f3337b\") " Oct 31 00:14:40.453373 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:40.452731 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1822bec6-cd20-4045-b362-d83766f3337b-credential-socket" pod "1822bec6-cd20-4045-b362-d83766f3337b" (UID: "1822bec6-cd20-4045-b362-d83766f3337b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:14:40.453373 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:40.452733 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1822bec6-cd20-4045-b362-d83766f3337b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1822bec6-cd20-4045-b362-d83766f3337b-istio-envoy\") pod \"1822bec6-cd20-4045-b362-d83766f3337b\" (UID: \"1822bec6-cd20-4045-b362-d83766f3337b\") " Oct 31 00:14:40.453373 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:40.452835 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1822bec6-cd20-4045-b362-d83766f3337b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1822bec6-cd20-4045-b362-d83766f3337b-workload-socket\") pod \"1822bec6-cd20-4045-b362-d83766f3337b\" (UID: \"1822bec6-cd20-4045-b362-d83766f3337b\") " Oct 31 00:14:40.453373 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:40.452884 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1822bec6-cd20-4045-b362-d83766f3337b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1822bec6-cd20-4045-b362-d83766f3337b-istio-podinfo\") pod \"1822bec6-cd20-4045-b362-d83766f3337b\" (UID: \"1822bec6-cd20-4045-b362-d83766f3337b\") " Oct 31 00:14:40.453373 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:40.453002 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1822bec6-cd20-4045-b362-d83766f3337b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1822bec6-cd20-4045-b362-d83766f3337b-istio-data\") pod \"1822bec6-cd20-4045-b362-d83766f3337b\" (UID: \"1822bec6-cd20-4045-b362-d83766f3337b\") " Oct 31 00:14:40.453373 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:40.453289 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1822bec6-cd20-4045-b362-d83766f3337b-workload-socket" pod "1822bec6-cd20-4045-b362-d83766f3337b" (UID: "1822bec6-cd20-4045-b362-d83766f3337b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:14:40.453981 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:40.453676 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1822bec6-cd20-4045-b362-d83766f3337b-istiod-ca-cert" pod "1822bec6-cd20-4045-b362-d83766f3337b" (UID: "1822bec6-cd20-4045-b362-d83766f3337b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:14:40.453981 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:40.453949 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1822bec6-cd20-4045-b362-d83766f3337b-workload-certs" pod "1822bec6-cd20-4045-b362-d83766f3337b" (UID: "1822bec6-cd20-4045-b362-d83766f3337b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:14:40.454719 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:40.454638 4927 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1822bec6-cd20-4045-b362-d83766f3337b-istiod-ca-cert\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:14:40.454719 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:40.454667 4927 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1822bec6-cd20-4045-b362-d83766f3337b-workload-socket\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:14:40.454719 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:40.454681 4927 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1822bec6-cd20-4045-b362-d83766f3337b-credential-socket\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:14:40.454719 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:40.454694 4927 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1822bec6-cd20-4045-b362-d83766f3337b-workload-certs\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:14:40.454998 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:40.454887 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1822bec6-cd20-4045-b362-d83766f3337b-istio-data" pod "1822bec6-cd20-4045-b362-d83766f3337b" (UID: "1822bec6-cd20-4045-b362-d83766f3337b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:14:40.456756 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:40.456727 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1822bec6-cd20-4045-b362-d83766f3337b-istio-envoy" pod "1822bec6-cd20-4045-b362-d83766f3337b" (UID: "1822bec6-cd20-4045-b362-d83766f3337b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:14:40.457231 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:40.457203 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1822bec6-cd20-4045-b362-d83766f3337b-istio-podinfo" pod "1822bec6-cd20-4045-b362-d83766f3337b" (UID: "1822bec6-cd20-4045-b362-d83766f3337b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:14:40.457510 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:40.457474 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1822bec6-cd20-4045-b362-d83766f3337b-kube-api-access-9jvrk" pod "1822bec6-cd20-4045-b362-d83766f3337b" (UID: "1822bec6-cd20-4045-b362-d83766f3337b"). InnerVolumeSpecName "kube-api-access-9jvrk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:14:40.457683 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:40.457663 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1822bec6-cd20-4045-b362-d83766f3337b-istio-token" pod "1822bec6-cd20-4045-b362-d83766f3337b" (UID: "1822bec6-cd20-4045-b362-d83766f3337b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:14:40.499478 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:40.479303 4927 generic.go:356] "Generic (PLEG): container finished" podID="1822bec6-cd20-4045-b362-d83766f3337b" containerID="52c84a877ba688ff497a00fa0c370da126a9e699c2f21bd029c9a50e48ec50cf" exitCode=0 Oct 31 00:14:40.499478 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:40.479627 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-t4ms2" event={"ID":"1822bec6-cd20-4045-b362-d83766f3337b","Type":"ContainerDied","Data":"52c84a877ba688ff497a00fa0c370da126a9e699c2f21bd029c9a50e48ec50cf"} Oct 31 00:14:40.499478 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:40.479684 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-t4ms2" event={"ID":"1822bec6-cd20-4045-b362-d83766f3337b","Type":"ContainerDied","Data":"56b5348455bc2c26be8b05bf7068d2b8cb5276653ea8f9a7b2cb8abd0de7ee6d"} Oct 31 00:14:40.499478 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:40.479715 4927 scope.go:122] "RemoveContainer" containerID="52c84a877ba688ff497a00fa0c370da126a9e699c2f21bd029c9a50e48ec50cf" Oct 31 00:14:40.499478 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:40.479883 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-t4ms2" Oct 31 00:14:40.523136 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:40.523097 4927 scope.go:122] "RemoveContainer" containerID="52c84a877ba688ff497a00fa0c370da126a9e699c2f21bd029c9a50e48ec50cf" Oct 31 00:14:40.523732 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:14:40.523703 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"52c84a877ba688ff497a00fa0c370da126a9e699c2f21bd029c9a50e48ec50cf\": container with ID starting with 52c84a877ba688ff497a00fa0c370da126a9e699c2f21bd029c9a50e48ec50cf not found: ID does not exist" containerID="52c84a877ba688ff497a00fa0c370da126a9e699c2f21bd029c9a50e48ec50cf" Oct 31 00:14:40.523899 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:40.523874 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52c84a877ba688ff497a00fa0c370da126a9e699c2f21bd029c9a50e48ec50cf"} err="failed to get container status \"52c84a877ba688ff497a00fa0c370da126a9e699c2f21bd029c9a50e48ec50cf\": rpc error: code = NotFound desc = could not find container \"52c84a877ba688ff497a00fa0c370da126a9e699c2f21bd029c9a50e48ec50cf\": container with ID starting with 52c84a877ba688ff497a00fa0c370da126a9e699c2f21bd029c9a50e48ec50cf not found: ID does not exist" Oct 31 00:14:40.558410 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:40.557731 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9jvrk\" (UniqueName: \"kubernetes.io/projected/1822bec6-cd20-4045-b362-d83766f3337b-kube-api-access-9jvrk\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:14:40.558410 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:40.557782 4927 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1822bec6-cd20-4045-b362-d83766f3337b-istio-token\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:14:40.558410 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:40.557796 4927 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1822bec6-cd20-4045-b362-d83766f3337b-istio-envoy\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:14:40.558410 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:40.557810 4927 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1822bec6-cd20-4045-b362-d83766f3337b-istio-podinfo\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:14:40.558410 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:40.557833 4927 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1822bec6-cd20-4045-b362-d83766f3337b-istio-data\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:14:40.656586 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:40.656547 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-t4ms2"] Oct 31 00:14:40.661163 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:40.661128 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-t4ms2"] Oct 31 00:14:41.088785 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:41.088750 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1822bec6-cd20-4045-b362-d83766f3337b" path="/var/lib/kubelet/pods/1822bec6-cd20-4045-b362-d83766f3337b/volumes" Oct 31 00:14:43.902603 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:43.902327 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:14:47.510446 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:47.510406 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-54ad8c-7c679ffcbb-kmx4b"] Oct 31 00:14:47.621669 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:47.621619 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-54ad8c-7b7f75c84c-nntg5"] Oct 31 00:14:47.863129 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:47.863046 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:14:48.040971 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:48.040898 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:14:48.593801 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:48.593759 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:14:49.680442 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:14:49.680399 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sqv69" containerName="extractor" Oct 31 00:14:52.510873 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:52.510813 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-54ad8c-7c679ffcbb-kmx4b" podUID="9dccc739-8fcc-4f45-b236-58338e7fcd05" containerName="payload-processing" containerID="cri-o://fb433bc4bcfdb57d32597b60f9be58977ea893e817dae7964a189cf70f59d542" gracePeriod=25 Oct 31 00:14:52.622521 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:52.622460 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-54ad8c-7b7f75c84c-nntg5" podUID="777b44d3-70ae-4988-b2bc-63c7a7830461" containerName="payload-pre-processing" containerID="cri-o://584615f03324988b3dc4b6b05211cca4612b3463d8c06fcd3d3318e41f6ae4c5" gracePeriod=25 Oct 31 00:14:52.848232 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:52.847838 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-54ad8c-7c679ffcbb-kmx4b" Oct 31 00:14:52.954269 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:52.954244 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-54ad8c-7b7f75c84c-nntg5" Oct 31 00:14:53.024069 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:53.024030 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9dccc739-8fcc-4f45-b236-58338e7fcd05-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9dccc739-8fcc-4f45-b236-58338e7fcd05-plugins-config-volume\") pod \"9dccc739-8fcc-4f45-b236-58338e7fcd05\" (UID: \"9dccc739-8fcc-4f45-b236-58338e7fcd05\") " Oct 31 00:14:53.024262 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:53.024093 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9dccc739-8fcc-4f45-b236-58338e7fcd05-kube-api-access-zhxnx\" (UniqueName: \"kubernetes.io/projected/9dccc739-8fcc-4f45-b236-58338e7fcd05-kube-api-access-zhxnx\") pod \"9dccc739-8fcc-4f45-b236-58338e7fcd05\" (UID: \"9dccc739-8fcc-4f45-b236-58338e7fcd05\") " Oct 31 00:14:53.025212 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:53.025169 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9dccc739-8fcc-4f45-b236-58338e7fcd05-plugins-config-volume" pod "9dccc739-8fcc-4f45-b236-58338e7fcd05" (UID: "9dccc739-8fcc-4f45-b236-58338e7fcd05"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:14:53.031348 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:53.027827 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9dccc739-8fcc-4f45-b236-58338e7fcd05-kube-api-access-zhxnx" pod "9dccc739-8fcc-4f45-b236-58338e7fcd05" (UID: "9dccc739-8fcc-4f45-b236-58338e7fcd05"). InnerVolumeSpecName "kube-api-access-zhxnx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:14:53.127019 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:53.126984 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/777b44d3-70ae-4988-b2bc-63c7a7830461-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/777b44d3-70ae-4988-b2bc-63c7a7830461-plugins-config-volume\") pod \"777b44d3-70ae-4988-b2bc-63c7a7830461\" (UID: \"777b44d3-70ae-4988-b2bc-63c7a7830461\") " Oct 31 00:14:53.127209 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:53.127072 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/777b44d3-70ae-4988-b2bc-63c7a7830461-kube-api-access-txfjg\" (UniqueName: \"kubernetes.io/projected/777b44d3-70ae-4988-b2bc-63c7a7830461-kube-api-access-txfjg\") pod \"777b44d3-70ae-4988-b2bc-63c7a7830461\" (UID: \"777b44d3-70ae-4988-b2bc-63c7a7830461\") " Oct 31 00:14:53.127858 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:53.127828 4927 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9dccc739-8fcc-4f45-b236-58338e7fcd05-plugins-config-volume\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:14:53.127973 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:53.127862 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zhxnx\" (UniqueName: \"kubernetes.io/projected/9dccc739-8fcc-4f45-b236-58338e7fcd05-kube-api-access-zhxnx\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:14:53.128900 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:53.128822 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/777b44d3-70ae-4988-b2bc-63c7a7830461-plugins-config-volume" pod "777b44d3-70ae-4988-b2bc-63c7a7830461" (UID: "777b44d3-70ae-4988-b2bc-63c7a7830461"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:14:53.132610 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:53.132576 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/777b44d3-70ae-4988-b2bc-63c7a7830461-kube-api-access-txfjg" pod "777b44d3-70ae-4988-b2bc-63c7a7830461" (UID: "777b44d3-70ae-4988-b2bc-63c7a7830461"). InnerVolumeSpecName "kube-api-access-txfjg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:14:53.201454 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:53.201399 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:14:53.229387 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:53.229349 4927 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/777b44d3-70ae-4988-b2bc-63c7a7830461-plugins-config-volume\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:14:53.229387 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:53.229380 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-txfjg\" (UniqueName: \"kubernetes.io/projected/777b44d3-70ae-4988-b2bc-63c7a7830461-kube-api-access-txfjg\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:14:53.604215 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:53.604183 4927 generic.go:356] "Generic (PLEG): container finished" podID="777b44d3-70ae-4988-b2bc-63c7a7830461" containerID="584615f03324988b3dc4b6b05211cca4612b3463d8c06fcd3d3318e41f6ae4c5" exitCode=0 Oct 31 00:14:53.604680 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:53.604222 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-54ad8c-7b7f75c84c-nntg5" event={"ID":"777b44d3-70ae-4988-b2bc-63c7a7830461","Type":"ContainerDied","Data":"584615f03324988b3dc4b6b05211cca4612b3463d8c06fcd3d3318e41f6ae4c5"} Oct 31 00:14:53.604680 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:53.604280 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-54ad8c-7b7f75c84c-nntg5" event={"ID":"777b44d3-70ae-4988-b2bc-63c7a7830461","Type":"ContainerDied","Data":"f7ad08296104b1bc872732f88788a04b729eb4507f835fa67d561d838234ba0e"} Oct 31 00:14:53.604680 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:53.604305 4927 scope.go:122] "RemoveContainer" containerID="584615f03324988b3dc4b6b05211cca4612b3463d8c06fcd3d3318e41f6ae4c5" Oct 31 00:14:53.604680 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:53.604234 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-54ad8c-7b7f75c84c-nntg5" Oct 31 00:14:53.606684 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:53.606652 4927 generic.go:356] "Generic (PLEG): container finished" podID="9dccc739-8fcc-4f45-b236-58338e7fcd05" containerID="fb433bc4bcfdb57d32597b60f9be58977ea893e817dae7964a189cf70f59d542" exitCode=0 Oct 31 00:14:53.606838 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:53.606708 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-54ad8c-7c679ffcbb-kmx4b" event={"ID":"9dccc739-8fcc-4f45-b236-58338e7fcd05","Type":"ContainerDied","Data":"fb433bc4bcfdb57d32597b60f9be58977ea893e817dae7964a189cf70f59d542"} Oct 31 00:14:53.606838 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:53.606805 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-54ad8c-7c679ffcbb-kmx4b" event={"ID":"9dccc739-8fcc-4f45-b236-58338e7fcd05","Type":"ContainerDied","Data":"f8f41f88675a55cec5360f521f57b72aae11f6f3969f7e63fb388b36097a9454"} Oct 31 00:14:53.606955 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:53.606901 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-54ad8c-7c679ffcbb-kmx4b" Oct 31 00:14:53.635645 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:53.635621 4927 scope.go:122] "RemoveContainer" containerID="584615f03324988b3dc4b6b05211cca4612b3463d8c06fcd3d3318e41f6ae4c5" Oct 31 00:14:53.636064 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:14:53.636038 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"584615f03324988b3dc4b6b05211cca4612b3463d8c06fcd3d3318e41f6ae4c5\": container with ID starting with 584615f03324988b3dc4b6b05211cca4612b3463d8c06fcd3d3318e41f6ae4c5 not found: ID does not exist" containerID="584615f03324988b3dc4b6b05211cca4612b3463d8c06fcd3d3318e41f6ae4c5" Oct 31 00:14:53.636155 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:53.636072 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"584615f03324988b3dc4b6b05211cca4612b3463d8c06fcd3d3318e41f6ae4c5"} err="failed to get container status \"584615f03324988b3dc4b6b05211cca4612b3463d8c06fcd3d3318e41f6ae4c5\": rpc error: code = NotFound desc = could not find container \"584615f03324988b3dc4b6b05211cca4612b3463d8c06fcd3d3318e41f6ae4c5\": container with ID starting with 584615f03324988b3dc4b6b05211cca4612b3463d8c06fcd3d3318e41f6ae4c5 not found: ID does not exist" Oct 31 00:14:53.636155 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:53.636090 4927 scope.go:122] "RemoveContainer" containerID="fb433bc4bcfdb57d32597b60f9be58977ea893e817dae7964a189cf70f59d542" Oct 31 00:14:53.654066 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:53.654039 4927 scope.go:122] "RemoveContainer" containerID="fb433bc4bcfdb57d32597b60f9be58977ea893e817dae7964a189cf70f59d542" Oct 31 00:14:53.654411 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:14:53.654383 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb433bc4bcfdb57d32597b60f9be58977ea893e817dae7964a189cf70f59d542\": container with ID starting with fb433bc4bcfdb57d32597b60f9be58977ea893e817dae7964a189cf70f59d542 not found: ID does not exist" containerID="fb433bc4bcfdb57d32597b60f9be58977ea893e817dae7964a189cf70f59d542" Oct 31 00:14:53.654493 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:53.654419 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb433bc4bcfdb57d32597b60f9be58977ea893e817dae7964a189cf70f59d542"} err="failed to get container status \"fb433bc4bcfdb57d32597b60f9be58977ea893e817dae7964a189cf70f59d542\": rpc error: code = NotFound desc = could not find container \"fb433bc4bcfdb57d32597b60f9be58977ea893e817dae7964a189cf70f59d542\": container with ID starting with fb433bc4bcfdb57d32597b60f9be58977ea893e817dae7964a189cf70f59d542 not found: ID does not exist" Oct 31 00:14:53.776897 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:53.776864 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-54ad8c-7c679ffcbb-kmx4b"] Oct 31 00:14:53.781488 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:53.781434 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-54ad8c-7c679ffcbb-kmx4b"] Oct 31 00:14:53.894509 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:53.894470 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-54ad8c-7b7f75c84c-nntg5"] Oct 31 00:14:53.901260 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:53.901233 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-54ad8c-7b7f75c84c-nntg5"] Oct 31 00:14:55.088799 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:55.088768 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="777b44d3-70ae-4988-b2bc-63c7a7830461" path="/var/lib/kubelet/pods/777b44d3-70ae-4988-b2bc-63c7a7830461/volumes" Oct 31 00:14:55.089223 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:55.089208 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9dccc739-8fcc-4f45-b236-58338e7fcd05" path="/var/lib/kubelet/pods/9dccc739-8fcc-4f45-b236-58338e7fcd05/volumes" Oct 31 00:14:57.806375 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:14:57.805551 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:15:00.224298 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:15:00.224258 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-986d37-29890095-h5msp"] Oct 31 00:15:00.225581 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:15:00.225559 4927 cpu_manager.go:414] "containerMap: removing container" podUID="1822bec6-cd20-4045-b362-d83766f3337b" containerName="istio-proxy" Oct 31 00:15:00.225651 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:15:00.225583 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1822bec6-cd20-4045-b362-d83766f3337b" containerName="istio-proxy" Oct 31 00:15:00.225651 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:15:00.225619 4927 cpu_manager.go:414] "containerMap: removing container" podUID="777b44d3-70ae-4988-b2bc-63c7a7830461" containerName="payload-pre-processing" Oct 31 00:15:00.225651 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:15:00.225628 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="777b44d3-70ae-4988-b2bc-63c7a7830461" containerName="payload-pre-processing" Oct 31 00:15:00.225752 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:15:00.225657 4927 cpu_manager.go:414] "containerMap: removing container" podUID="9dccc739-8fcc-4f45-b236-58338e7fcd05" containerName="payload-processing" Oct 31 00:15:00.225752 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:15:00.225665 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9dccc739-8fcc-4f45-b236-58338e7fcd05" containerName="payload-processing" Oct 31 00:15:00.225849 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:15:00.225837 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="777b44d3-70ae-4988-b2bc-63c7a7830461" containerName="payload-pre-processing" Oct 31 00:15:00.225892 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:15:00.225859 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="9dccc739-8fcc-4f45-b236-58338e7fcd05" containerName="payload-processing" Oct 31 00:15:00.225892 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:15:00.225881 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="1822bec6-cd20-4045-b362-d83766f3337b" containerName="istio-proxy" Oct 31 00:15:00.232379 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:15:00.232317 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-986d37-29890095-h5msp" Oct 31 00:15:00.250665 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:15:00.250637 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-27e82b-29890095-xq5q9"] Oct 31 00:15:00.255630 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:15:00.255610 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-27e82b-29890095-xq5q9" Oct 31 00:15:00.273300 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:15:00.273277 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-34f1ce-29890095-fkgnv"] Oct 31 00:15:00.277900 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:15:00.277883 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-34f1ce-29890095-fkgnv" Oct 31 00:15:00.310815 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:15:00.310782 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4a3dba-29890095-hn4b4"] Oct 31 00:15:00.317368 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:15:00.317325 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4a3dba-29890095-hn4b4" Oct 31 00:15:00.323181 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:15:00.323158 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4fdzs\"" Oct 31 00:15:00.326179 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:15:00.326147 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-986d37-29890095-h5msp"] Oct 31 00:15:00.360784 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:15:00.360750 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-27e82b-29890095-xq5q9"] Oct 31 00:15:00.372757 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:15:00.372726 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-34f1ce-29890095-fkgnv"] Oct 31 00:15:00.380141 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:15:00.380113 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/15c765a1-f9dd-4d99-a3d7-9447be49ecae-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-986d37-29890095-h5msp\" (UID: \"15c765a1-f9dd-4d99-a3d7-9447be49ecae\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-986d37-29890095-h5msp" Oct 31 00:15:00.380303 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:15:00.380233 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qbrmh\" (UniqueName: \"kubernetes.io/projected/5499c02e-e482-4329-b2b6-265ff423c83c-kube-api-access-qbrmh\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-4a3dba-29890095-hn4b4\" (UID: \"5499c02e-e482-4329-b2b6-265ff423c83c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4a3dba-29890095-hn4b4" Oct 31 00:15:00.380303 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:15:00.380290 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kpk99\" (UniqueName: \"kubernetes.io/projected/1a2039e2-43bf-4f84-9c06-f7ba01c95c5d-kube-api-access-kpk99\") pod \"maas-api-key-cleanup-e2e-worker-w2-27e82b-29890095-xq5q9\" (UID: \"1a2039e2-43bf-4f84-9c06-f7ba01c95c5d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-27e82b-29890095-xq5q9" Oct 31 00:15:00.380575 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:15:00.380361 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xs9g8\" (UniqueName: \"kubernetes.io/projected/528cb403-6318-45aa-afac-39095d3a8731-kube-api-access-xs9g8\") pod \"maas-api-key-cleanup-e2e-worker-w4-34f1ce-29890095-fkgnv\" (UID: \"528cb403-6318-45aa-afac-39095d3a8731\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-34f1ce-29890095-fkgnv" Oct 31 00:15:00.380575 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:15:00.380398 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5499c02e-e482-4329-b2b6-265ff423c83c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-4a3dba-29890095-hn4b4\" (UID: \"5499c02e-e482-4329-b2b6-265ff423c83c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4a3dba-29890095-hn4b4" Oct 31 00:15:00.380575 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:15:00.380470 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xlctd\" (UniqueName: \"kubernetes.io/projected/15c765a1-f9dd-4d99-a3d7-9447be49ecae-kube-api-access-xlctd\") pod \"maas-api-key-cleanup-e2e-worker-w3-986d37-29890095-h5msp\" (UID: \"15c765a1-f9dd-4d99-a3d7-9447be49ecae\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-986d37-29890095-h5msp" Oct 31 00:15:00.380575 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:15:00.380500 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a2039e2-43bf-4f84-9c06-f7ba01c95c5d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-27e82b-29890095-xq5q9\" (UID: \"1a2039e2-43bf-4f84-9c06-f7ba01c95c5d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-27e82b-29890095-xq5q9" Oct 31 00:15:00.380575 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:15:00.380533 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/528cb403-6318-45aa-afac-39095d3a8731-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-34f1ce-29890095-fkgnv\" (UID: \"528cb403-6318-45aa-afac-39095d3a8731\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-34f1ce-29890095-fkgnv" Oct 31 00:15:00.422318 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:15:00.422286 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4a3dba-29890095-hn4b4"] Oct 31 00:15:00.488264 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:15:00.488171 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/15c765a1-f9dd-4d99-a3d7-9447be49ecae-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-986d37-29890095-h5msp\" (UID: \"15c765a1-f9dd-4d99-a3d7-9447be49ecae\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-986d37-29890095-h5msp" Oct 31 00:15:00.488264 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:15:00.488256 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbrmh\" (UniqueName: \"kubernetes.io/projected/5499c02e-e482-4329-b2b6-265ff423c83c-kube-api-access-qbrmh\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-4a3dba-29890095-hn4b4\" (UID: \"5499c02e-e482-4329-b2b6-265ff423c83c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4a3dba-29890095-hn4b4" Oct 31 00:15:00.488530 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:15:00.488297 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kpk99\" (UniqueName: \"kubernetes.io/projected/1a2039e2-43bf-4f84-9c06-f7ba01c95c5d-kube-api-access-kpk99\") pod \"maas-api-key-cleanup-e2e-worker-w2-27e82b-29890095-xq5q9\" (UID: \"1a2039e2-43bf-4f84-9c06-f7ba01c95c5d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-27e82b-29890095-xq5q9" Oct 31 00:15:00.488530 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:15:00.488321 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xs9g8\" (UniqueName: \"kubernetes.io/projected/528cb403-6318-45aa-afac-39095d3a8731-kube-api-access-xs9g8\") pod \"maas-api-key-cleanup-e2e-worker-w4-34f1ce-29890095-fkgnv\" (UID: \"528cb403-6318-45aa-afac-39095d3a8731\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-34f1ce-29890095-fkgnv" Oct 31 00:15:00.488530 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:15:00.488391 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5499c02e-e482-4329-b2b6-265ff423c83c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-4a3dba-29890095-hn4b4\" (UID: \"5499c02e-e482-4329-b2b6-265ff423c83c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4a3dba-29890095-hn4b4" Oct 31 00:15:00.488530 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:15:00.488455 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xlctd\" (UniqueName: \"kubernetes.io/projected/15c765a1-f9dd-4d99-a3d7-9447be49ecae-kube-api-access-xlctd\") pod \"maas-api-key-cleanup-e2e-worker-w3-986d37-29890095-h5msp\" (UID: \"15c765a1-f9dd-4d99-a3d7-9447be49ecae\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-986d37-29890095-h5msp" Oct 31 00:15:00.488530 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:15:00.488482 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a2039e2-43bf-4f84-9c06-f7ba01c95c5d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-27e82b-29890095-xq5q9\" (UID: \"1a2039e2-43bf-4f84-9c06-f7ba01c95c5d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-27e82b-29890095-xq5q9" Oct 31 00:15:00.488530 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:15:00.488513 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/528cb403-6318-45aa-afac-39095d3a8731-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-34f1ce-29890095-fkgnv\" (UID: \"528cb403-6318-45aa-afac-39095d3a8731\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-34f1ce-29890095-fkgnv" Oct 31 00:15:00.489102 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:15:00.489075 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/15c765a1-f9dd-4d99-a3d7-9447be49ecae-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-986d37-29890095-h5msp\" (UID: \"15c765a1-f9dd-4d99-a3d7-9447be49ecae\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-986d37-29890095-h5msp" Oct 31 00:15:00.489216 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:15:00.489198 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/528cb403-6318-45aa-afac-39095d3a8731-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-34f1ce-29890095-fkgnv\" (UID: \"528cb403-6318-45aa-afac-39095d3a8731\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-34f1ce-29890095-fkgnv" Oct 31 00:15:00.489259 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:15:00.489217 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a2039e2-43bf-4f84-9c06-f7ba01c95c5d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-27e82b-29890095-xq5q9\" (UID: \"1a2039e2-43bf-4f84-9c06-f7ba01c95c5d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-27e82b-29890095-xq5q9" Oct 31 00:15:00.489314 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:15:00.489292 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5499c02e-e482-4329-b2b6-265ff423c83c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-4a3dba-29890095-hn4b4\" (UID: \"5499c02e-e482-4329-b2b6-265ff423c83c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4a3dba-29890095-hn4b4" Oct 31 00:15:00.553526 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:15:00.553491 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kpk99\" (UniqueName: \"kubernetes.io/projected/1a2039e2-43bf-4f84-9c06-f7ba01c95c5d-kube-api-access-kpk99\") pod \"maas-api-key-cleanup-e2e-worker-w2-27e82b-29890095-xq5q9\" (UID: \"1a2039e2-43bf-4f84-9c06-f7ba01c95c5d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-27e82b-29890095-xq5q9" Oct 31 00:15:00.553526 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:15:00.553514 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xs9g8\" (UniqueName: \"kubernetes.io/projected/528cb403-6318-45aa-afac-39095d3a8731-kube-api-access-xs9g8\") pod \"maas-api-key-cleanup-e2e-worker-w4-34f1ce-29890095-fkgnv\" (UID: \"528cb403-6318-45aa-afac-39095d3a8731\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-34f1ce-29890095-fkgnv" Oct 31 00:15:00.553759 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:15:00.553714 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbrmh\" (UniqueName: \"kubernetes.io/projected/5499c02e-e482-4329-b2b6-265ff423c83c-kube-api-access-qbrmh\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-4a3dba-29890095-hn4b4\" (UID: \"5499c02e-e482-4329-b2b6-265ff423c83c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4a3dba-29890095-hn4b4" Oct 31 00:15:00.553820 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:15:00.553793 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xlctd\" (UniqueName: \"kubernetes.io/projected/15c765a1-f9dd-4d99-a3d7-9447be49ecae-kube-api-access-xlctd\") pod \"maas-api-key-cleanup-e2e-worker-w3-986d37-29890095-h5msp\" (UID: \"15c765a1-f9dd-4d99-a3d7-9447be49ecae\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-986d37-29890095-h5msp" Oct 31 00:15:00.569519 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:15:00.569491 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-27e82b-29890095-xq5q9" Oct 31 00:15:00.591554 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:15:00.591523 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-34f1ce-29890095-fkgnv" Oct 31 00:15:00.649954 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:15:00.647540 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4a3dba-29890095-hn4b4" Oct 31 00:15:00.847093 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:15:00.847053 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-986d37-29890095-h5msp" Oct 31 00:15:00.976260 ip-10-0-141-180 kubenswrapper[4927]: W1031 00:15:00.976224 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a2039e2_43bf_4f84_9c06_f7ba01c95c5d.slice/crio-4358d71415d413f64619bdfbbeb87b3ad7ef944f8f15a65e2ee24a0a36121e2f WatchSource:0}: Error finding container 4358d71415d413f64619bdfbbeb87b3ad7ef944f8f15a65e2ee24a0a36121e2f: Status 404 returned error can't find the container with id 4358d71415d413f64619bdfbbeb87b3ad7ef944f8f15a65e2ee24a0a36121e2f Oct 31 00:15:00.979224 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:15:00.979187 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-27e82b-29890095-xq5q9"] Oct 31 00:15:01.020969 ip-10-0-141-180 kubenswrapper[4927]: W1031 00:15:01.020944 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod528cb403_6318_45aa_afac_39095d3a8731.slice/crio-a3f02bf77bf9f7b7489ee0d74939d083beb619e264bed7b53f9905deb92f2c2e WatchSource:0}: Error finding container a3f02bf77bf9f7b7489ee0d74939d083beb619e264bed7b53f9905deb92f2c2e: Status 404 returned error can't find the container with id a3f02bf77bf9f7b7489ee0d74939d083beb619e264bed7b53f9905deb92f2c2e Oct 31 00:15:01.021847 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:15:01.021820 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-34f1ce-29890095-fkgnv"] Oct 31 00:15:01.085757 ip-10-0-141-180 kubenswrapper[4927]: W1031 00:15:01.085720 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5499c02e_e482_4329_b2b6_265ff423c83c.slice/crio-819c9b10f09845a0f5255beb7d0deb1583704a569696b84dbce10c4e410f5e9a WatchSource:0}: Error finding container 819c9b10f09845a0f5255beb7d0deb1583704a569696b84dbce10c4e410f5e9a: Status 404 returned error can't find the container with id 819c9b10f09845a0f5255beb7d0deb1583704a569696b84dbce10c4e410f5e9a Oct 31 00:15:01.091574 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:15:01.091552 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4a3dba-29890095-hn4b4"] Oct 31 00:15:01.233494 ip-10-0-141-180 kubenswrapper[4927]: W1031 00:15:01.233461 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15c765a1_f9dd_4d99_a3d7_9447be49ecae.slice/crio-13607984e25732cd3c89edb2ac955969d4de7667648b39d9ed754a0359438d92 WatchSource:0}: Error finding container 13607984e25732cd3c89edb2ac955969d4de7667648b39d9ed754a0359438d92: Status 404 returned error can't find the container with id 13607984e25732cd3c89edb2ac955969d4de7667648b39d9ed754a0359438d92 Oct 31 00:15:01.234668 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:15:01.234632 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-986d37-29890095-h5msp"] Oct 31 00:15:01.775667 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:15:01.775566 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-986d37-29890095-h5msp" event={"ID":"15c765a1-f9dd-4d99-a3d7-9447be49ecae","Type":"ContainerStarted","Data":"11e159eb305a58cf768960f4fc567cad782a95346fbbcabdbe8f2aec4308c799"} Oct 31 00:15:01.775667 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:15:01.775619 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-986d37-29890095-h5msp" event={"ID":"15c765a1-f9dd-4d99-a3d7-9447be49ecae","Type":"ContainerStarted","Data":"13607984e25732cd3c89edb2ac955969d4de7667648b39d9ed754a0359438d92"} Oct 31 00:15:01.777437 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:15:01.777398 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-27e82b-29890095-xq5q9" event={"ID":"1a2039e2-43bf-4f84-9c06-f7ba01c95c5d","Type":"ContainerStarted","Data":"1655a006015189fc73ed674bf55d64d983e6eec5fd148753cd88f4f00e7bd439"} Oct 31 00:15:01.777582 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:15:01.777455 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-27e82b-29890095-xq5q9" event={"ID":"1a2039e2-43bf-4f84-9c06-f7ba01c95c5d","Type":"ContainerStarted","Data":"4358d71415d413f64619bdfbbeb87b3ad7ef944f8f15a65e2ee24a0a36121e2f"} Oct 31 00:15:01.779116 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:15:01.779082 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-34f1ce-29890095-fkgnv" event={"ID":"528cb403-6318-45aa-afac-39095d3a8731","Type":"ContainerStarted","Data":"dc494a272fa60e6d70f2e48e3e91c31751aa0fe1543c6f70ee48be0782166c09"} Oct 31 00:15:01.779236 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:15:01.779131 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-34f1ce-29890095-fkgnv" event={"ID":"528cb403-6318-45aa-afac-39095d3a8731","Type":"ContainerStarted","Data":"a3f02bf77bf9f7b7489ee0d74939d083beb619e264bed7b53f9905deb92f2c2e"} Oct 31 00:15:01.780666 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:15:01.780642 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4a3dba-29890095-hn4b4" event={"ID":"5499c02e-e482-4329-b2b6-265ff423c83c","Type":"ContainerStarted","Data":"f8faa8a512b610703ff93a91d03828a67e823db61939a4d9595fecd257787b7b"} Oct 31 00:15:01.780780 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:15:01.780688 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4a3dba-29890095-hn4b4" event={"ID":"5499c02e-e482-4329-b2b6-265ff423c83c","Type":"ContainerStarted","Data":"819c9b10f09845a0f5255beb7d0deb1583704a569696b84dbce10c4e410f5e9a"} Oct 31 00:15:01.897486 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:15:01.897432 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-986d37-29890095-h5msp" podStartSLOduration=1.8974172459999998 podStartE2EDuration="1.897417246s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:15:01.893486346 +0000 UTC m=+2014.929817167" watchObservedRunningTime="2026-10-31 00:15:01.897417246 +0000 UTC m=+2014.933748068" Oct 31 00:15:02.018911 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:15:02.018848 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4a3dba-29890095-hn4b4" podStartSLOduration=2.018827054 podStartE2EDuration="2.018827054s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:15:02.011131995 +0000 UTC m=+2015.047462831" watchObservedRunningTime="2026-10-31 00:15:02.018827054 +0000 UTC m=+2015.055157877" Oct 31 00:15:02.132975 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:15:02.132920 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-27e82b-29890095-xq5q9" podStartSLOduration=2.1329059519999998 podStartE2EDuration="2.132905952s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:15:02.13104081 +0000 UTC m=+2015.167371653" watchObservedRunningTime="2026-10-31 00:15:02.132905952 +0000 UTC m=+2015.169236775" Oct 31 00:15:02.250240 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:15:02.250176 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-34f1ce-29890095-fkgnv" podStartSLOduration=2.250156876 podStartE2EDuration="2.250156876s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:15:02.248523773 +0000 UTC m=+2015.284854614" watchObservedRunningTime="2026-10-31 00:15:02.250156876 +0000 UTC m=+2015.286487699" Oct 31 00:15:02.421608 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:15:02.421574 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:15:02.791163 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:15:02.791124 4927 generic.go:356] "Generic (PLEG): container finished" podID="528cb403-6318-45aa-afac-39095d3a8731" containerID="dc494a272fa60e6d70f2e48e3e91c31751aa0fe1543c6f70ee48be0782166c09" exitCode=60 Oct 31 00:15:02.791384 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:15:02.791208 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-34f1ce-29890095-fkgnv" event={"ID":"528cb403-6318-45aa-afac-39095d3a8731","Type":"ContainerDied","Data":"dc494a272fa60e6d70f2e48e3e91c31751aa0fe1543c6f70ee48be0782166c09"} Oct 31 00:15:02.791593 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:15:02.791576 4927 scope.go:122] "RemoveContainer" containerID="dc494a272fa60e6d70f2e48e3e91c31751aa0fe1543c6f70ee48be0782166c09" Oct 31 00:15:02.793313 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:15:02.793279 4927 generic.go:356] "Generic (PLEG): container finished" podID="5499c02e-e482-4329-b2b6-265ff423c83c" containerID="f8faa8a512b610703ff93a91d03828a67e823db61939a4d9595fecd257787b7b" exitCode=60 Oct 31 00:15:02.793486 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:15:02.793395 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4a3dba-29890095-hn4b4" event={"ID":"5499c02e-e482-4329-b2b6-265ff423c83c","Type":"ContainerDied","Data":"f8faa8a512b610703ff93a91d03828a67e823db61939a4d9595fecd257787b7b"} Oct 31 00:15:02.793695 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:15:02.793655 4927 scope.go:122] "RemoveContainer" containerID="f8faa8a512b610703ff93a91d03828a67e823db61939a4d9595fecd257787b7b" Oct 31 00:15:02.796095 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:15:02.796074 4927 generic.go:356] "Generic (PLEG): container finished" podID="15c765a1-f9dd-4d99-a3d7-9447be49ecae" containerID="11e159eb305a58cf768960f4fc567cad782a95346fbbcabdbe8f2aec4308c799" exitCode=60 Oct 31 00:15:02.796185 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:15:02.796161 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-986d37-29890095-h5msp" event={"ID":"15c765a1-f9dd-4d99-a3d7-9447be49ecae","Type":"ContainerDied","Data":"11e159eb305a58cf768960f4fc567cad782a95346fbbcabdbe8f2aec4308c799"} Oct 31 00:15:02.796444 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:15:02.796417 4927 scope.go:122] "RemoveContainer" containerID="11e159eb305a58cf768960f4fc567cad782a95346fbbcabdbe8f2aec4308c799" Oct 31 00:15:02.798039 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:15:02.798020 4927 generic.go:356] "Generic (PLEG): container finished" podID="1a2039e2-43bf-4f84-9c06-f7ba01c95c5d" containerID="1655a006015189fc73ed674bf55d64d983e6eec5fd148753cd88f4f00e7bd439" exitCode=60 Oct 31 00:15:02.798114 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:15:02.798066 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-27e82b-29890095-xq5q9" event={"ID":"1a2039e2-43bf-4f84-9c06-f7ba01c95c5d","Type":"ContainerDied","Data":"1655a006015189fc73ed674bf55d64d983e6eec5fd148753cd88f4f00e7bd439"} Oct 31 00:15:02.798298 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:15:02.798285 4927 scope.go:122] "RemoveContainer" containerID="1655a006015189fc73ed674bf55d64d983e6eec5fd148753cd88f4f00e7bd439" Oct 31 00:15:03.809931 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:15:03.809887 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-27e82b-29890095-xq5q9" event={"ID":"1a2039e2-43bf-4f84-9c06-f7ba01c95c5d","Type":"ContainerStarted","Data":"3eead72705c8f3d99885e749345b860458dda130d48ecacd49fdcfef88804064"} Oct 31 00:15:03.812395 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:15:03.812322 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-34f1ce-29890095-fkgnv" event={"ID":"528cb403-6318-45aa-afac-39095d3a8731","Type":"ContainerStarted","Data":"2ec8872956887ef9c5e95cdb40930ae6dd0f5ed9aca88b5f3f6b94e4bd038d68"} Oct 31 00:15:03.814605 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:15:03.814566 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4a3dba-29890095-hn4b4" event={"ID":"5499c02e-e482-4329-b2b6-265ff423c83c","Type":"ContainerStarted","Data":"c35bce14e4697d893c889f8df56b471994b955a83e11c8007a5c66a05093cb15"} Oct 31 00:15:03.816643 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:15:03.816615 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-986d37-29890095-h5msp" event={"ID":"15c765a1-f9dd-4d99-a3d7-9447be49ecae","Type":"ContainerStarted","Data":"b0824cb37b488f1e7918994446df06f624ff31ca09777d06e45baef6070d92bb"} Oct 31 00:15:04.828482 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:15:04.828440 4927 generic.go:356] "Generic (PLEG): container finished" podID="1a2039e2-43bf-4f84-9c06-f7ba01c95c5d" containerID="3eead72705c8f3d99885e749345b860458dda130d48ecacd49fdcfef88804064" exitCode=60 Oct 31 00:15:04.828978 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:15:04.828521 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-27e82b-29890095-xq5q9" event={"ID":"1a2039e2-43bf-4f84-9c06-f7ba01c95c5d","Type":"ContainerDied","Data":"3eead72705c8f3d99885e749345b860458dda130d48ecacd49fdcfef88804064"} Oct 31 00:15:04.828978 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:15:04.828591 4927 scope.go:122] "RemoveContainer" containerID="1655a006015189fc73ed674bf55d64d983e6eec5fd148753cd88f4f00e7bd439" Oct 31 00:15:04.828978 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:15:04.828973 4927 scope.go:122] "RemoveContainer" containerID="3eead72705c8f3d99885e749345b860458dda130d48ecacd49fdcfef88804064" Oct 31 00:15:04.829523 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:15:04.829492 4927 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-27e82b-29890095-xq5q9_odh-ai-gateway-infra(1a2039e2-43bf-4f84-9c06-f7ba01c95c5d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-27e82b-29890095-xq5q9" podUID="1a2039e2-43bf-4f84-9c06-f7ba01c95c5d" Oct 31 00:15:04.831199 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:15:04.831174 4927 generic.go:356] "Generic (PLEG): container finished" podID="528cb403-6318-45aa-afac-39095d3a8731" containerID="2ec8872956887ef9c5e95cdb40930ae6dd0f5ed9aca88b5f3f6b94e4bd038d68" exitCode=60 Oct 31 00:15:04.831306 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:15:04.831267 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-34f1ce-29890095-fkgnv" event={"ID":"528cb403-6318-45aa-afac-39095d3a8731","Type":"ContainerDied","Data":"2ec8872956887ef9c5e95cdb40930ae6dd0f5ed9aca88b5f3f6b94e4bd038d68"} Oct 31 00:15:04.831615 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:15:04.831598 4927 scope.go:122] "RemoveContainer" containerID="2ec8872956887ef9c5e95cdb40930ae6dd0f5ed9aca88b5f3f6b94e4bd038d68" Oct 31 00:15:04.832004 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:15:04.831980 4927 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-34f1ce-29890095-fkgnv_odh-ai-gateway-infra(528cb403-6318-45aa-afac-39095d3a8731)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-34f1ce-29890095-fkgnv" podUID="528cb403-6318-45aa-afac-39095d3a8731" Oct 31 00:15:04.833658 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:15:04.833610 4927 generic.go:356] "Generic (PLEG): container finished" podID="5499c02e-e482-4329-b2b6-265ff423c83c" containerID="c35bce14e4697d893c889f8df56b471994b955a83e11c8007a5c66a05093cb15" exitCode=60 Oct 31 00:15:04.833754 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:15:04.833654 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4a3dba-29890095-hn4b4" event={"ID":"5499c02e-e482-4329-b2b6-265ff423c83c","Type":"ContainerDied","Data":"c35bce14e4697d893c889f8df56b471994b955a83e11c8007a5c66a05093cb15"} Oct 31 00:15:04.833977 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:15:04.833961 4927 scope.go:122] "RemoveContainer" containerID="c35bce14e4697d893c889f8df56b471994b955a83e11c8007a5c66a05093cb15" Oct 31 00:15:04.834273 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:15:04.834254 4927 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-4a3dba-29890095-hn4b4_odh-ai-gateway-infra(5499c02e-e482-4329-b2b6-265ff423c83c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4a3dba-29890095-hn4b4" podUID="5499c02e-e482-4329-b2b6-265ff423c83c" Oct 31 00:15:04.836014 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:15:04.835993 4927 generic.go:356] "Generic (PLEG): container finished" podID="15c765a1-f9dd-4d99-a3d7-9447be49ecae" containerID="b0824cb37b488f1e7918994446df06f624ff31ca09777d06e45baef6070d92bb" exitCode=60 Oct 31 00:15:04.836124 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:15:04.836082 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-986d37-29890095-h5msp" event={"ID":"15c765a1-f9dd-4d99-a3d7-9447be49ecae","Type":"ContainerDied","Data":"b0824cb37b488f1e7918994446df06f624ff31ca09777d06e45baef6070d92bb"} Oct 31 00:15:04.836403 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:15:04.836383 4927 scope.go:122] "RemoveContainer" containerID="b0824cb37b488f1e7918994446df06f624ff31ca09777d06e45baef6070d92bb" Oct 31 00:15:04.836819 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:15:04.836743 4927 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-986d37-29890095-h5msp_odh-ai-gateway-infra(15c765a1-f9dd-4d99-a3d7-9447be49ecae)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-986d37-29890095-h5msp" podUID="15c765a1-f9dd-4d99-a3d7-9447be49ecae" Oct 31 00:15:04.845372 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:15:04.845350 4927 scope.go:122] "RemoveContainer" containerID="dc494a272fa60e6d70f2e48e3e91c31751aa0fe1543c6f70ee48be0782166c09" Oct 31 00:15:04.861350 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:15:04.861307 4927 scope.go:122] "RemoveContainer" containerID="f8faa8a512b610703ff93a91d03828a67e823db61939a4d9595fecd257787b7b" Oct 31 00:15:04.876522 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:15:04.876497 4927 scope.go:122] "RemoveContainer" containerID="11e159eb305a58cf768960f4fc567cad782a95346fbbcabdbe8f2aec4308c799" Oct 31 00:15:05.853069 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:15:05.853035 4927 scope.go:122] "RemoveContainer" containerID="3eead72705c8f3d99885e749345b860458dda130d48ecacd49fdcfef88804064" Oct 31 00:15:05.853559 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:15:05.853475 4927 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-27e82b-29890095-xq5q9_odh-ai-gateway-infra(1a2039e2-43bf-4f84-9c06-f7ba01c95c5d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-27e82b-29890095-xq5q9" podUID="1a2039e2-43bf-4f84-9c06-f7ba01c95c5d" Oct 31 00:15:05.854913 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:15:05.854884 4927 scope.go:122] "RemoveContainer" containerID="2ec8872956887ef9c5e95cdb40930ae6dd0f5ed9aca88b5f3f6b94e4bd038d68" Oct 31 00:15:05.855285 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:15:05.855257 4927 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-34f1ce-29890095-fkgnv_odh-ai-gateway-infra(528cb403-6318-45aa-afac-39095d3a8731)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-34f1ce-29890095-fkgnv" podUID="528cb403-6318-45aa-afac-39095d3a8731" Oct 31 00:15:05.856830 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:15:05.856811 4927 scope.go:122] "RemoveContainer" containerID="c35bce14e4697d893c889f8df56b471994b955a83e11c8007a5c66a05093cb15" Oct 31 00:15:05.857160 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:15:05.857128 4927 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-4a3dba-29890095-hn4b4_odh-ai-gateway-infra(5499c02e-e482-4329-b2b6-265ff423c83c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4a3dba-29890095-hn4b4" podUID="5499c02e-e482-4329-b2b6-265ff423c83c" Oct 31 00:15:05.858549 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:15:05.858530 4927 scope.go:122] "RemoveContainer" containerID="b0824cb37b488f1e7918994446df06f624ff31ca09777d06e45baef6070d92bb" Oct 31 00:15:05.858802 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:15:05.858781 4927 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-986d37-29890095-h5msp_odh-ai-gateway-infra(15c765a1-f9dd-4d99-a3d7-9447be49ecae)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-986d37-29890095-h5msp" podUID="15c765a1-f9dd-4d99-a3d7-9447be49ecae" Oct 31 00:15:06.078105 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:15:06.077258 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4e23415d-mtlsl"] Oct 31 00:15:06.090006 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:15:06.087736 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4e23415d-mtlsl"] Oct 31 00:15:07.104743 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:15:07.104664 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="358f8214-1852-4a02-880e-3659b48bb92a" path="/var/lib/kubelet/pods/358f8214-1852-4a02-880e-3659b48bb92a/volumes" Oct 31 00:15:07.562315 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:15:07.562277 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:15:12.069445 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:15:12.069402 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:15:16.079776 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:15:16.079740 4927 scope.go:122] "RemoveContainer" containerID="2ec8872956887ef9c5e95cdb40930ae6dd0f5ed9aca88b5f3f6b94e4bd038d68" Oct 31 00:15:16.080279 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:15:16.080026 4927 scope.go:122] "RemoveContainer" containerID="3eead72705c8f3d99885e749345b860458dda130d48ecacd49fdcfef88804064" Oct 31 00:15:16.609030 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:15:16.608993 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:15:16.972612 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:15:16.972570 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-27e82b-29890095-xq5q9" event={"ID":"1a2039e2-43bf-4f84-9c06-f7ba01c95c5d","Type":"ContainerStarted","Data":"55a1d76ec4f65de31298953940252b726d2ffdc16fdaf6d5f78d0e6036c29453"} Oct 31 00:15:16.974637 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:15:16.974612 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-34f1ce-29890095-fkgnv" event={"ID":"528cb403-6318-45aa-afac-39095d3a8731","Type":"ContainerStarted","Data":"c7d8122cd53345578b31cd06c599aaa25da2dbbd78a713be3a0d8a2aa6762b9c"} Oct 31 00:15:17.079463 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:15:17.079432 4927 scope.go:122] "RemoveContainer" containerID="c35bce14e4697d893c889f8df56b471994b955a83e11c8007a5c66a05093cb15" Oct 31 00:15:17.984697 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:15:17.984664 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4a3dba-29890095-hn4b4" event={"ID":"5499c02e-e482-4329-b2b6-265ff423c83c","Type":"ContainerStarted","Data":"c6db5ac3003eee1914e0286e871feb11e0491bae9e8f58345000b7d462992b2d"} Oct 31 00:15:17.987387 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:15:17.987360 4927 generic.go:356] "Generic (PLEG): container finished" podID="1a2039e2-43bf-4f84-9c06-f7ba01c95c5d" containerID="55a1d76ec4f65de31298953940252b726d2ffdc16fdaf6d5f78d0e6036c29453" exitCode=60 Oct 31 00:15:17.987527 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:15:17.987388 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-27e82b-29890095-xq5q9" event={"ID":"1a2039e2-43bf-4f84-9c06-f7ba01c95c5d","Type":"ContainerDied","Data":"55a1d76ec4f65de31298953940252b726d2ffdc16fdaf6d5f78d0e6036c29453"} Oct 31 00:15:17.987527 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:15:17.987441 4927 scope.go:122] "RemoveContainer" containerID="3eead72705c8f3d99885e749345b860458dda130d48ecacd49fdcfef88804064" Oct 31 00:15:17.987746 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:15:17.987728 4927 scope.go:122] "RemoveContainer" containerID="55a1d76ec4f65de31298953940252b726d2ffdc16fdaf6d5f78d0e6036c29453" Oct 31 00:15:17.988186 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:15:17.987995 4927 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-27e82b-29890095-xq5q9_odh-ai-gateway-infra(1a2039e2-43bf-4f84-9c06-f7ba01c95c5d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-27e82b-29890095-xq5q9" podUID="1a2039e2-43bf-4f84-9c06-f7ba01c95c5d" Oct 31 00:15:17.990163 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:15:17.990143 4927 generic.go:356] "Generic (PLEG): container finished" podID="528cb403-6318-45aa-afac-39095d3a8731" containerID="c7d8122cd53345578b31cd06c599aaa25da2dbbd78a713be3a0d8a2aa6762b9c" exitCode=60 Oct 31 00:15:17.990317 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:15:17.990174 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-34f1ce-29890095-fkgnv" event={"ID":"528cb403-6318-45aa-afac-39095d3a8731","Type":"ContainerDied","Data":"c7d8122cd53345578b31cd06c599aaa25da2dbbd78a713be3a0d8a2aa6762b9c"} Oct 31 00:15:17.990541 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:15:17.990516 4927 scope.go:122] "RemoveContainer" containerID="c7d8122cd53345578b31cd06c599aaa25da2dbbd78a713be3a0d8a2aa6762b9c" Oct 31 00:15:17.990952 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:15:17.990787 4927 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-34f1ce-29890095-fkgnv_odh-ai-gateway-infra(528cb403-6318-45aa-afac-39095d3a8731)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-34f1ce-29890095-fkgnv" podUID="528cb403-6318-45aa-afac-39095d3a8731" Oct 31 00:15:18.002409 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:15:18.002382 4927 scope.go:122] "RemoveContainer" containerID="2ec8872956887ef9c5e95cdb40930ae6dd0f5ed9aca88b5f3f6b94e4bd038d68" Oct 31 00:15:18.078826 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:15:18.078804 4927 scope.go:122] "RemoveContainer" containerID="b0824cb37b488f1e7918994446df06f624ff31ca09777d06e45baef6070d92bb" Oct 31 00:15:18.106100 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:15:18.106074 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-27e82b-29890095-xq5q9"] Oct 31 00:15:18.294641 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:15:18.294566 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-34f1ce-29890095-fkgnv"] Oct 31 00:15:19.002850 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:15:19.002810 4927 generic.go:356] "Generic (PLEG): container finished" podID="5499c02e-e482-4329-b2b6-265ff423c83c" containerID="c6db5ac3003eee1914e0286e871feb11e0491bae9e8f58345000b7d462992b2d" exitCode=60 Oct 31 00:15:19.003269 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:15:19.002882 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4a3dba-29890095-hn4b4" event={"ID":"5499c02e-e482-4329-b2b6-265ff423c83c","Type":"ContainerDied","Data":"c6db5ac3003eee1914e0286e871feb11e0491bae9e8f58345000b7d462992b2d"} Oct 31 00:15:19.003269 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:15:19.003012 4927 scope.go:122] "RemoveContainer" containerID="c35bce14e4697d893c889f8df56b471994b955a83e11c8007a5c66a05093cb15" Oct 31 00:15:19.003442 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:15:19.003391 4927 scope.go:122] "RemoveContainer" containerID="c6db5ac3003eee1914e0286e871feb11e0491bae9e8f58345000b7d462992b2d" Oct 31 00:15:19.003758 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:15:19.003731 4927 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-4a3dba-29890095-hn4b4_odh-ai-gateway-infra(5499c02e-e482-4329-b2b6-265ff423c83c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4a3dba-29890095-hn4b4" podUID="5499c02e-e482-4329-b2b6-265ff423c83c" Oct 31 00:15:19.005482 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:15:19.005458 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-986d37-29890095-h5msp" event={"ID":"15c765a1-f9dd-4d99-a3d7-9447be49ecae","Type":"ContainerStarted","Data":"96a916d035526178d6affa031388431572a656def208f80a46c90229c16f1d90"} Oct 31 00:15:19.137752 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:15:19.137418 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4a3dba-29890095-hn4b4"] Oct 31 00:15:19.265011 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:15:19.264953 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-34f1ce-29890095-fkgnv" Oct 31 00:15:19.267948 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:15:19.267912 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-27e82b-29890095-xq5q9" Oct 31 00:15:19.304074 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:15:19.304047 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a2039e2-43bf-4f84-9c06-f7ba01c95c5d-kube-api-access-kpk99\" (UniqueName: \"kubernetes.io/projected/1a2039e2-43bf-4f84-9c06-f7ba01c95c5d-kube-api-access-kpk99\") pod \"1a2039e2-43bf-4f84-9c06-f7ba01c95c5d\" (UID: \"1a2039e2-43bf-4f84-9c06-f7ba01c95c5d\") " Oct 31 00:15:19.304194 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:15:19.304179 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/528cb403-6318-45aa-afac-39095d3a8731-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/528cb403-6318-45aa-afac-39095d3a8731-maas-api-service-ca\") pod \"528cb403-6318-45aa-afac-39095d3a8731\" (UID: \"528cb403-6318-45aa-afac-39095d3a8731\") " Oct 31 00:15:19.304283 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:15:19.304268 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/528cb403-6318-45aa-afac-39095d3a8731-kube-api-access-xs9g8\" (UniqueName: \"kubernetes.io/projected/528cb403-6318-45aa-afac-39095d3a8731-kube-api-access-xs9g8\") pod \"528cb403-6318-45aa-afac-39095d3a8731\" (UID: \"528cb403-6318-45aa-afac-39095d3a8731\") " Oct 31 00:15:19.304327 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:15:19.304302 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a2039e2-43bf-4f84-9c06-f7ba01c95c5d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a2039e2-43bf-4f84-9c06-f7ba01c95c5d-maas-api-service-ca\") pod \"1a2039e2-43bf-4f84-9c06-f7ba01c95c5d\" (UID: \"1a2039e2-43bf-4f84-9c06-f7ba01c95c5d\") " Oct 31 00:15:19.304704 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:15:19.304612 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/528cb403-6318-45aa-afac-39095d3a8731-maas-api-service-ca" pod "528cb403-6318-45aa-afac-39095d3a8731" (UID: "528cb403-6318-45aa-afac-39095d3a8731"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:15:19.304799 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:15:19.304766 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a2039e2-43bf-4f84-9c06-f7ba01c95c5d-maas-api-service-ca" pod "1a2039e2-43bf-4f84-9c06-f7ba01c95c5d" (UID: "1a2039e2-43bf-4f84-9c06-f7ba01c95c5d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:15:19.305074 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:15:19.305057 4927 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/528cb403-6318-45aa-afac-39095d3a8731-maas-api-service-ca\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:15:19.305135 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:15:19.305077 4927 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a2039e2-43bf-4f84-9c06-f7ba01c95c5d-maas-api-service-ca\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:15:19.306491 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:15:19.306470 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/528cb403-6318-45aa-afac-39095d3a8731-kube-api-access-xs9g8" pod "528cb403-6318-45aa-afac-39095d3a8731" (UID: "528cb403-6318-45aa-afac-39095d3a8731"). InnerVolumeSpecName "kube-api-access-xs9g8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:15:19.306604 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:15:19.306585 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a2039e2-43bf-4f84-9c06-f7ba01c95c5d-kube-api-access-kpk99" pod "1a2039e2-43bf-4f84-9c06-f7ba01c95c5d" (UID: "1a2039e2-43bf-4f84-9c06-f7ba01c95c5d"). InnerVolumeSpecName "kube-api-access-kpk99". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:15:19.406383 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:15:19.406344 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xs9g8\" (UniqueName: \"kubernetes.io/projected/528cb403-6318-45aa-afac-39095d3a8731-kube-api-access-xs9g8\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:15:19.406383 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:15:19.406382 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kpk99\" (UniqueName: \"kubernetes.io/projected/1a2039e2-43bf-4f84-9c06-f7ba01c95c5d-kube-api-access-kpk99\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:15:19.682545 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:15:19.682509 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sqv69" containerName="extractor" Oct 31 00:15:20.022000 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:15:20.021918 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-27e82b-29890095-xq5q9" Oct 31 00:15:20.022000 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:15:20.021946 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-27e82b-29890095-xq5q9" event={"ID":"1a2039e2-43bf-4f84-9c06-f7ba01c95c5d","Type":"ContainerDied","Data":"4358d71415d413f64619bdfbbeb87b3ad7ef944f8f15a65e2ee24a0a36121e2f"} Oct 31 00:15:20.022565 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:15:20.022011 4927 scope.go:122] "RemoveContainer" containerID="55a1d76ec4f65de31298953940252b726d2ffdc16fdaf6d5f78d0e6036c29453" Oct 31 00:15:20.023968 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:15:20.023940 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-34f1ce-29890095-fkgnv" Oct 31 00:15:20.024103 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:15:20.024009 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-34f1ce-29890095-fkgnv" event={"ID":"528cb403-6318-45aa-afac-39095d3a8731","Type":"ContainerDied","Data":"a3f02bf77bf9f7b7489ee0d74939d083beb619e264bed7b53f9905deb92f2c2e"} Oct 31 00:15:20.037501 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:15:20.037476 4927 generic.go:356] "Generic (PLEG): container finished" podID="15c765a1-f9dd-4d99-a3d7-9447be49ecae" containerID="96a916d035526178d6affa031388431572a656def208f80a46c90229c16f1d90" exitCode=60 Oct 31 00:15:20.037595 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:15:20.037557 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-986d37-29890095-h5msp" event={"ID":"15c765a1-f9dd-4d99-a3d7-9447be49ecae","Type":"ContainerDied","Data":"96a916d035526178d6affa031388431572a656def208f80a46c90229c16f1d90"} Oct 31 00:15:20.038021 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:15:20.037980 4927 scope.go:122] "RemoveContainer" containerID="96a916d035526178d6affa031388431572a656def208f80a46c90229c16f1d90" Oct 31 00:15:20.045648 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:15:20.045537 4927 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-986d37-29890095-h5msp_odh-ai-gateway-infra(15c765a1-f9dd-4d99-a3d7-9447be49ecae)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-986d37-29890095-h5msp" podUID="15c765a1-f9dd-4d99-a3d7-9447be49ecae" Oct 31 00:15:20.046010 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:15:20.045986 4927 scope.go:122] "RemoveContainer" containerID="c7d8122cd53345578b31cd06c599aaa25da2dbbd78a713be3a0d8a2aa6762b9c" Oct 31 00:15:20.059255 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:15:20.059236 4927 scope.go:122] "RemoveContainer" containerID="b0824cb37b488f1e7918994446df06f624ff31ca09777d06e45baef6070d92bb" Oct 31 00:15:20.203636 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:15:20.203609 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-34f1ce-29890095-fkgnv"] Oct 31 00:15:20.209456 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:15:20.209431 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-34f1ce-29890095-fkgnv"] Oct 31 00:15:20.229718 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:15:20.229691 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4a3dba-29890095-hn4b4" Oct 31 00:15:20.263383 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:15:20.263356 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-986d37-29890095-h5msp"] Oct 31 00:15:20.347262 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:15:20.347182 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5499c02e-e482-4329-b2b6-265ff423c83c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5499c02e-e482-4329-b2b6-265ff423c83c-maas-api-service-ca\") pod \"5499c02e-e482-4329-b2b6-265ff423c83c\" (UID: \"5499c02e-e482-4329-b2b6-265ff423c83c\") " Oct 31 00:15:20.347465 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:15:20.347448 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5499c02e-e482-4329-b2b6-265ff423c83c-kube-api-access-qbrmh\" (UniqueName: \"kubernetes.io/projected/5499c02e-e482-4329-b2b6-265ff423c83c-kube-api-access-qbrmh\") pod \"5499c02e-e482-4329-b2b6-265ff423c83c\" (UID: \"5499c02e-e482-4329-b2b6-265ff423c83c\") " Oct 31 00:15:20.347620 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:15:20.347596 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5499c02e-e482-4329-b2b6-265ff423c83c-maas-api-service-ca" pod "5499c02e-e482-4329-b2b6-265ff423c83c" (UID: "5499c02e-e482-4329-b2b6-265ff423c83c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:15:20.347983 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:15:20.347964 4927 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5499c02e-e482-4329-b2b6-265ff423c83c-maas-api-service-ca\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:15:20.349645 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:15:20.349615 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5499c02e-e482-4329-b2b6-265ff423c83c-kube-api-access-qbrmh" pod "5499c02e-e482-4329-b2b6-265ff423c83c" (UID: "5499c02e-e482-4329-b2b6-265ff423c83c"). InnerVolumeSpecName "kube-api-access-qbrmh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:15:20.449514 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:15:20.449473 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qbrmh\" (UniqueName: \"kubernetes.io/projected/5499c02e-e482-4329-b2b6-265ff423c83c-kube-api-access-qbrmh\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:15:20.646622 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:15:20.646541 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-27e82b-29890095-xq5q9"] Oct 31 00:15:20.651623 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:15:20.651596 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-27e82b-29890095-xq5q9"] Oct 31 00:15:21.052609 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:15:21.052579 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4a3dba-29890095-hn4b4" Oct 31 00:15:21.053081 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:15:21.052578 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4a3dba-29890095-hn4b4" event={"ID":"5499c02e-e482-4329-b2b6-265ff423c83c","Type":"ContainerDied","Data":"819c9b10f09845a0f5255beb7d0deb1583704a569696b84dbce10c4e410f5e9a"} Oct 31 00:15:21.053081 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:15:21.052705 4927 scope.go:122] "RemoveContainer" containerID="c6db5ac3003eee1914e0286e871feb11e0491bae9e8f58345000b7d462992b2d" Oct 31 00:15:21.094630 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:15:21.094598 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a2039e2-43bf-4f84-9c06-f7ba01c95c5d" path="/var/lib/kubelet/pods/1a2039e2-43bf-4f84-9c06-f7ba01c95c5d/volumes" Oct 31 00:15:21.095108 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:15:21.095092 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="528cb403-6318-45aa-afac-39095d3a8731" path="/var/lib/kubelet/pods/528cb403-6318-45aa-afac-39095d3a8731/volumes" Oct 31 00:15:21.237214 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:15:21.237182 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:15:21.273394 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:15:21.273363 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-986d37-29890095-h5msp" Oct 31 00:15:21.429433 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:15:21.395922 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15c765a1-f9dd-4d99-a3d7-9447be49ecae-kube-api-access-xlctd\" (UniqueName: \"kubernetes.io/projected/15c765a1-f9dd-4d99-a3d7-9447be49ecae-kube-api-access-xlctd\") pod \"15c765a1-f9dd-4d99-a3d7-9447be49ecae\" (UID: \"15c765a1-f9dd-4d99-a3d7-9447be49ecae\") " Oct 31 00:15:21.429433 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:15:21.396391 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/15c765a1-f9dd-4d99-a3d7-9447be49ecae-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/15c765a1-f9dd-4d99-a3d7-9447be49ecae-maas-api-service-ca\") pod \"15c765a1-f9dd-4d99-a3d7-9447be49ecae\" (UID: \"15c765a1-f9dd-4d99-a3d7-9447be49ecae\") " Oct 31 00:15:21.429433 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:15:21.399956 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15c765a1-f9dd-4d99-a3d7-9447be49ecae-maas-api-service-ca" pod "15c765a1-f9dd-4d99-a3d7-9447be49ecae" (UID: "15c765a1-f9dd-4d99-a3d7-9447be49ecae"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:15:21.429433 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:15:21.424047 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:15:21.432202 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:15:21.432167 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15c765a1-f9dd-4d99-a3d7-9447be49ecae-kube-api-access-xlctd" pod "15c765a1-f9dd-4d99-a3d7-9447be49ecae" (UID: "15c765a1-f9dd-4d99-a3d7-9447be49ecae"). InnerVolumeSpecName "kube-api-access-xlctd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:15:21.460736 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:15:21.458554 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4a3dba-29890095-hn4b4"] Oct 31 00:15:21.491392 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:15:21.487762 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4a3dba-29890095-hn4b4"] Oct 31 00:15:21.519353 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:15:21.503056 4927 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/15c765a1-f9dd-4d99-a3d7-9447be49ecae-maas-api-service-ca\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:15:21.519662 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:15:21.519630 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xlctd\" (UniqueName: \"kubernetes.io/projected/15c765a1-f9dd-4d99-a3d7-9447be49ecae-kube-api-access-xlctd\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:15:21.672658 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:15:21.672621 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:15:22.073321 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:15:22.073287 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-986d37-29890095-h5msp" Oct 31 00:15:22.073321 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:15:22.073304 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-986d37-29890095-h5msp" event={"ID":"15c765a1-f9dd-4d99-a3d7-9447be49ecae","Type":"ContainerDied","Data":"13607984e25732cd3c89edb2ac955969d4de7667648b39d9ed754a0359438d92"} Oct 31 00:15:22.073913 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:15:22.073433 4927 scope.go:122] "RemoveContainer" containerID="96a916d035526178d6affa031388431572a656def208f80a46c90229c16f1d90" Oct 31 00:15:22.087500 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:15:22.087472 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:15:22.256999 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:15:22.256959 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-986d37-29890095-h5msp"] Oct 31 00:15:22.261445 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:15:22.261417 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-986d37-29890095-h5msp"] Oct 31 00:15:22.374449 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:15:22.374320 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:15:23.088669 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:15:23.088638 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="15c765a1-f9dd-4d99-a3d7-9447be49ecae" path="/var/lib/kubelet/pods/15c765a1-f9dd-4d99-a3d7-9447be49ecae/volumes" Oct 31 00:15:23.089120 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:15:23.089104 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5499c02e-e482-4329-b2b6-265ff423c83c" path="/var/lib/kubelet/pods/5499c02e-e482-4329-b2b6-265ff423c83c/volumes" Oct 31 00:15:26.051347 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:15:26.051304 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:15:26.252221 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:15:26.252176 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:15:26.465954 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:15:26.465921 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:15:29.491406 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:15:29.491366 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:15:31.020181 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:15:31.020128 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:15:31.209685 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:15:31.209648 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:15:35.735318 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:15:35.735278 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:15:40.250747 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:15:40.250705 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:15:40.475742 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:15:40.475688 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:15:40.633093 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:15:40.633005 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:15:45.194447 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:15:45.194411 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:15:49.546251 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:15:49.546216 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:15:49.679493 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:15:49.679454 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sqv69" containerName="extractor" Oct 31 00:15:50.111374 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:15:50.107924 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:15:54.856851 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:15:54.856804 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:15:55.472638 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:15:55.472597 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-ab0ac9-847986794b-bbf7n"] Oct 31 00:15:55.591820 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:15:55.591763 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-ab0ac9-7978cbf74c-qxdvb"] Oct 31 00:15:55.842679 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:15:55.842589 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:15:55.999980 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:15:55.999945 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:15:57.821226 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:15:57.821189 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:15:57.985909 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:15:57.985878 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:15:58.205421 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:15:58.205390 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:15:58.366596 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:15:58.366554 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:15:58.821238 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:15:58.821179 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-54ad8c-5c7f67b585-gn54m"] Oct 31 00:15:58.822091 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:15:58.822055 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-54ad8c-5c7f67b585-gn54m" podUID="6ecf922c-de0b-4e64-a2a2-07067e279990" containerName="maas-api" containerID="cri-o://f95528cfc6a6f0a4ca51e4c57ab61b569a21076b6a102f43c39cd0f8de0c23ef" gracePeriod=30 Oct 31 00:15:58.967552 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:15:58.967511 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:15:59.514091 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:15:59.514059 4927 generic.go:356] "Generic (PLEG): container finished" podID="6ecf922c-de0b-4e64-a2a2-07067e279990" containerID="f95528cfc6a6f0a4ca51e4c57ab61b569a21076b6a102f43c39cd0f8de0c23ef" exitCode=0 Oct 31 00:15:59.514285 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:15:59.514137 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-54ad8c-5c7f67b585-gn54m" event={"ID":"6ecf922c-de0b-4e64-a2a2-07067e279990","Type":"ContainerDied","Data":"f95528cfc6a6f0a4ca51e4c57ab61b569a21076b6a102f43c39cd0f8de0c23ef"} Oct 31 00:15:59.573768 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:15:59.573741 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-54ad8c-5c7f67b585-gn54m" Oct 31 00:15:59.686651 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:15:59.686619 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6ecf922c-de0b-4e64-a2a2-07067e279990-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6ecf922c-de0b-4e64-a2a2-07067e279990-maas-api-tls\") pod \"6ecf922c-de0b-4e64-a2a2-07067e279990\" (UID: \"6ecf922c-de0b-4e64-a2a2-07067e279990\") " Oct 31 00:15:59.686869 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:15:59.686855 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ecf922c-de0b-4e64-a2a2-07067e279990-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ecf922c-de0b-4e64-a2a2-07067e279990-openshift-service-ca\") pod \"6ecf922c-de0b-4e64-a2a2-07067e279990\" (UID: \"6ecf922c-de0b-4e64-a2a2-07067e279990\") " Oct 31 00:15:59.686976 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:15:59.686951 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6ecf922c-de0b-4e64-a2a2-07067e279990-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6ecf922c-de0b-4e64-a2a2-07067e279990-metrics-certs\") pod \"6ecf922c-de0b-4e64-a2a2-07067e279990\" (UID: \"6ecf922c-de0b-4e64-a2a2-07067e279990\") " Oct 31 00:15:59.687094 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:15:59.687010 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ecf922c-de0b-4e64-a2a2-07067e279990-kube-api-access-wbjzr\" (UniqueName: \"kubernetes.io/projected/6ecf922c-de0b-4e64-a2a2-07067e279990-kube-api-access-wbjzr\") pod \"6ecf922c-de0b-4e64-a2a2-07067e279990\" (UID: \"6ecf922c-de0b-4e64-a2a2-07067e279990\") " Oct 31 00:15:59.687221 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:15:59.687196 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ecf922c-de0b-4e64-a2a2-07067e279990-openshift-service-ca" pod "6ecf922c-de0b-4e64-a2a2-07067e279990" (UID: "6ecf922c-de0b-4e64-a2a2-07067e279990"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:15:59.687577 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:15:59.687549 4927 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ecf922c-de0b-4e64-a2a2-07067e279990-openshift-service-ca\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:15:59.688832 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:15:59.688804 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ecf922c-de0b-4e64-a2a2-07067e279990-maas-api-tls" pod "6ecf922c-de0b-4e64-a2a2-07067e279990" (UID: "6ecf922c-de0b-4e64-a2a2-07067e279990"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:15:59.688973 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:15:59.688951 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ecf922c-de0b-4e64-a2a2-07067e279990-metrics-certs" pod "6ecf922c-de0b-4e64-a2a2-07067e279990" (UID: "6ecf922c-de0b-4e64-a2a2-07067e279990"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:15:59.689109 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:15:59.689096 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ecf922c-de0b-4e64-a2a2-07067e279990-kube-api-access-wbjzr" pod "6ecf922c-de0b-4e64-a2a2-07067e279990" (UID: "6ecf922c-de0b-4e64-a2a2-07067e279990"). InnerVolumeSpecName "kube-api-access-wbjzr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:15:59.974806 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:15:59.974715 4927 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6ecf922c-de0b-4e64-a2a2-07067e279990-metrics-certs\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:15:59.974806 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:15:59.974741 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wbjzr\" (UniqueName: \"kubernetes.io/projected/6ecf922c-de0b-4e64-a2a2-07067e279990-kube-api-access-wbjzr\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:15:59.974806 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:15:59.974752 4927 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6ecf922c-de0b-4e64-a2a2-07067e279990-maas-api-tls\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:16:00.473082 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:00.473039 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-ab0ac9-847986794b-bbf7n" podUID="5a2e97c5-6cd5-498d-b83f-13e94fc34b28" containerName="payload-processing" containerID="cri-o://c6b04a8bcb3c58da0ce7fafb8e2b818a26b6eecea7ee531ea7a28a64310ac02b" gracePeriod=25 Oct 31 00:16:00.474281 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:00.474251 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:16:00.524510 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:00.524481 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-54ad8c-5c7f67b585-gn54m" Oct 31 00:16:00.524672 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:00.524517 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-54ad8c-5c7f67b585-gn54m" event={"ID":"6ecf922c-de0b-4e64-a2a2-07067e279990","Type":"ContainerDied","Data":"527ff3aafa8bec41550e55d55c9c1e85df9cb10d3096f337ca27f62b55202c00"} Oct 31 00:16:00.524672 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:00.524576 4927 scope.go:122] "RemoveContainer" containerID="f95528cfc6a6f0a4ca51e4c57ab61b569a21076b6a102f43c39cd0f8de0c23ef" Oct 31 00:16:00.592600 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:00.592565 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-ab0ac9-7978cbf74c-qxdvb" podUID="6874430f-3348-469c-adc3-a605fa1ff16a" containerName="payload-pre-processing" containerID="cri-o://6c47ebef07bf10252586dc9a818cf86f00d375138f965e29cbf91a4a60748630" gracePeriod=25 Oct 31 00:16:00.705381 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:00.700388 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-54ad8c-5c7f67b585-gn54m"] Oct 31 00:16:00.722826 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:00.722793 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-54ad8c-5c7f67b585-gn54m"] Oct 31 00:16:00.738197 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:00.738119 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:16:00.794729 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:00.794704 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-ab0ac9-847986794b-bbf7n" Oct 31 00:16:00.889546 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:00.889142 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a2e97c5-6cd5-498d-b83f-13e94fc34b28-kube-api-access-849gc\" (UniqueName: \"kubernetes.io/projected/5a2e97c5-6cd5-498d-b83f-13e94fc34b28-kube-api-access-849gc\") pod \"5a2e97c5-6cd5-498d-b83f-13e94fc34b28\" (UID: \"5a2e97c5-6cd5-498d-b83f-13e94fc34b28\") " Oct 31 00:16:00.889546 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:00.889386 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a2e97c5-6cd5-498d-b83f-13e94fc34b28-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a2e97c5-6cd5-498d-b83f-13e94fc34b28-plugins-config-volume\") pod \"5a2e97c5-6cd5-498d-b83f-13e94fc34b28\" (UID: \"5a2e97c5-6cd5-498d-b83f-13e94fc34b28\") " Oct 31 00:16:00.896384 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:00.893529 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a2e97c5-6cd5-498d-b83f-13e94fc34b28-plugins-config-volume" pod "5a2e97c5-6cd5-498d-b83f-13e94fc34b28" (UID: "5a2e97c5-6cd5-498d-b83f-13e94fc34b28"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:16:00.920219 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:00.917844 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:16:00.924417 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:00.924374 4927 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a2e97c5-6cd5-498d-b83f-13e94fc34b28-plugins-config-volume\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:16:00.942699 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:00.942661 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a2e97c5-6cd5-498d-b83f-13e94fc34b28-kube-api-access-849gc" pod "5a2e97c5-6cd5-498d-b83f-13e94fc34b28" (UID: "5a2e97c5-6cd5-498d-b83f-13e94fc34b28"). InnerVolumeSpecName "kube-api-access-849gc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:16:00.961449 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:00.961417 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-ab0ac9-7978cbf74c-qxdvb" Oct 31 00:16:01.025403 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:01.025304 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6874430f-3348-469c-adc3-a605fa1ff16a-kube-api-access-m6j6g\" (UniqueName: \"kubernetes.io/projected/6874430f-3348-469c-adc3-a605fa1ff16a-kube-api-access-m6j6g\") pod \"6874430f-3348-469c-adc3-a605fa1ff16a\" (UID: \"6874430f-3348-469c-adc3-a605fa1ff16a\") " Oct 31 00:16:01.025886 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:01.025460 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6874430f-3348-469c-adc3-a605fa1ff16a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6874430f-3348-469c-adc3-a605fa1ff16a-plugins-config-volume\") pod \"6874430f-3348-469c-adc3-a605fa1ff16a\" (UID: \"6874430f-3348-469c-adc3-a605fa1ff16a\") " Oct 31 00:16:01.025886 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:01.025811 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6874430f-3348-469c-adc3-a605fa1ff16a-plugins-config-volume" pod "6874430f-3348-469c-adc3-a605fa1ff16a" (UID: "6874430f-3348-469c-adc3-a605fa1ff16a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:16:01.026119 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:01.026101 4927 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6874430f-3348-469c-adc3-a605fa1ff16a-plugins-config-volume\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:16:01.026187 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:01.026123 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-849gc\" (UniqueName: \"kubernetes.io/projected/5a2e97c5-6cd5-498d-b83f-13e94fc34b28-kube-api-access-849gc\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:16:01.027537 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:01.027512 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6874430f-3348-469c-adc3-a605fa1ff16a-kube-api-access-m6j6g" pod "6874430f-3348-469c-adc3-a605fa1ff16a" (UID: "6874430f-3348-469c-adc3-a605fa1ff16a"). InnerVolumeSpecName "kube-api-access-m6j6g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:16:01.089087 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:01.089055 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ecf922c-de0b-4e64-a2a2-07067e279990" path="/var/lib/kubelet/pods/6ecf922c-de0b-4e64-a2a2-07067e279990/volumes" Oct 31 00:16:01.127436 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:01.127397 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m6j6g\" (UniqueName: \"kubernetes.io/projected/6874430f-3348-469c-adc3-a605fa1ff16a-kube-api-access-m6j6g\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:16:01.533969 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:01.533929 4927 generic.go:356] "Generic (PLEG): container finished" podID="5a2e97c5-6cd5-498d-b83f-13e94fc34b28" containerID="c6b04a8bcb3c58da0ce7fafb8e2b818a26b6eecea7ee531ea7a28a64310ac02b" exitCode=0 Oct 31 00:16:01.534166 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:01.533980 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-ab0ac9-847986794b-bbf7n" Oct 31 00:16:01.534166 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:01.534020 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-ab0ac9-847986794b-bbf7n" event={"ID":"5a2e97c5-6cd5-498d-b83f-13e94fc34b28","Type":"ContainerDied","Data":"c6b04a8bcb3c58da0ce7fafb8e2b818a26b6eecea7ee531ea7a28a64310ac02b"} Oct 31 00:16:01.534166 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:01.534090 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-ab0ac9-847986794b-bbf7n" event={"ID":"5a2e97c5-6cd5-498d-b83f-13e94fc34b28","Type":"ContainerDied","Data":"c420c20352966c1d9cdb747c723c23924fd480fd79a097ab09b49fc211f5bfed"} Oct 31 00:16:01.534166 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:01.534126 4927 scope.go:122] "RemoveContainer" containerID="c6b04a8bcb3c58da0ce7fafb8e2b818a26b6eecea7ee531ea7a28a64310ac02b" Oct 31 00:16:01.537242 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:01.537219 4927 generic.go:356] "Generic (PLEG): container finished" podID="6874430f-3348-469c-adc3-a605fa1ff16a" containerID="6c47ebef07bf10252586dc9a818cf86f00d375138f965e29cbf91a4a60748630" exitCode=0 Oct 31 00:16:01.537421 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:01.537280 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-ab0ac9-7978cbf74c-qxdvb" Oct 31 00:16:01.537421 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:01.537292 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-ab0ac9-7978cbf74c-qxdvb" event={"ID":"6874430f-3348-469c-adc3-a605fa1ff16a","Type":"ContainerDied","Data":"6c47ebef07bf10252586dc9a818cf86f00d375138f965e29cbf91a4a60748630"} Oct 31 00:16:01.537421 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:01.537365 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-ab0ac9-7978cbf74c-qxdvb" event={"ID":"6874430f-3348-469c-adc3-a605fa1ff16a","Type":"ContainerDied","Data":"e820c91c27c945ee5a30e9759980e4561a049d39b78dd0f5266ab93b86e91d66"} Oct 31 00:16:01.547282 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:01.547260 4927 scope.go:122] "RemoveContainer" containerID="c6b04a8bcb3c58da0ce7fafb8e2b818a26b6eecea7ee531ea7a28a64310ac02b" Oct 31 00:16:01.547659 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:16:01.547637 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c6b04a8bcb3c58da0ce7fafb8e2b818a26b6eecea7ee531ea7a28a64310ac02b\": container with ID starting with c6b04a8bcb3c58da0ce7fafb8e2b818a26b6eecea7ee531ea7a28a64310ac02b not found: ID does not exist" containerID="c6b04a8bcb3c58da0ce7fafb8e2b818a26b6eecea7ee531ea7a28a64310ac02b" Oct 31 00:16:01.547728 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:01.547668 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6b04a8bcb3c58da0ce7fafb8e2b818a26b6eecea7ee531ea7a28a64310ac02b"} err="failed to get container status \"c6b04a8bcb3c58da0ce7fafb8e2b818a26b6eecea7ee531ea7a28a64310ac02b\": rpc error: code = NotFound desc = could not find container \"c6b04a8bcb3c58da0ce7fafb8e2b818a26b6eecea7ee531ea7a28a64310ac02b\": container with ID starting with c6b04a8bcb3c58da0ce7fafb8e2b818a26b6eecea7ee531ea7a28a64310ac02b not found: ID does not exist" Oct 31 00:16:01.547728 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:01.547687 4927 scope.go:122] "RemoveContainer" containerID="6c47ebef07bf10252586dc9a818cf86f00d375138f965e29cbf91a4a60748630" Oct 31 00:16:01.558173 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:01.558147 4927 scope.go:122] "RemoveContainer" containerID="6c47ebef07bf10252586dc9a818cf86f00d375138f965e29cbf91a4a60748630" Oct 31 00:16:01.558488 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:16:01.558467 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c47ebef07bf10252586dc9a818cf86f00d375138f965e29cbf91a4a60748630\": container with ID starting with 6c47ebef07bf10252586dc9a818cf86f00d375138f965e29cbf91a4a60748630 not found: ID does not exist" containerID="6c47ebef07bf10252586dc9a818cf86f00d375138f965e29cbf91a4a60748630" Oct 31 00:16:01.558555 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:01.558499 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c47ebef07bf10252586dc9a818cf86f00d375138f965e29cbf91a4a60748630"} err="failed to get container status \"6c47ebef07bf10252586dc9a818cf86f00d375138f965e29cbf91a4a60748630\": rpc error: code = NotFound desc = could not find container \"6c47ebef07bf10252586dc9a818cf86f00d375138f965e29cbf91a4a60748630\": container with ID starting with 6c47ebef07bf10252586dc9a818cf86f00d375138f965e29cbf91a4a60748630 not found: ID does not exist" Oct 31 00:16:01.707298 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:01.707261 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:16:01.718595 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:01.718562 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-ab0ac9-847986794b-bbf7n"] Oct 31 00:16:01.727449 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:01.727422 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-ab0ac9-847986794b-bbf7n"] Oct 31 00:16:01.847087 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:01.847010 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-ab0ac9-7978cbf74c-qxdvb"] Oct 31 00:16:01.853836 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:01.853807 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-ab0ac9-7978cbf74c-qxdvb"] Oct 31 00:16:03.090803 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:03.090763 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a2e97c5-6cd5-498d-b83f-13e94fc34b28" path="/var/lib/kubelet/pods/5a2e97c5-6cd5-498d-b83f-13e94fc34b28/volumes" Oct 31 00:16:03.091355 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:03.091322 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6874430f-3348-469c-adc3-a605fa1ff16a" path="/var/lib/kubelet/pods/6874430f-3348-469c-adc3-a605fa1ff16a/volumes" Oct 31 00:16:05.509774 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:05.509739 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-c9c21e-68d69b8994-hnwg4"] Oct 31 00:16:05.510557 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:05.510541 4927 cpu_manager.go:414] "containerMap: removing container" podUID="15c765a1-f9dd-4d99-a3d7-9447be49ecae" containerName="cleanup" Oct 31 00:16:05.510615 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:05.510558 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15c765a1-f9dd-4d99-a3d7-9447be49ecae" containerName="cleanup" Oct 31 00:16:05.510615 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:05.510572 4927 cpu_manager.go:414] "containerMap: removing container" podUID="528cb403-6318-45aa-afac-39095d3a8731" containerName="cleanup" Oct 31 00:16:05.510615 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:05.510578 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="528cb403-6318-45aa-afac-39095d3a8731" containerName="cleanup" Oct 31 00:16:05.510615 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:05.510585 4927 cpu_manager.go:414] "containerMap: removing container" podUID="1a2039e2-43bf-4f84-9c06-f7ba01c95c5d" containerName="cleanup" Oct 31 00:16:05.510615 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:05.510590 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a2039e2-43bf-4f84-9c06-f7ba01c95c5d" containerName="cleanup" Oct 31 00:16:05.510615 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:05.510601 4927 cpu_manager.go:414] "containerMap: removing container" podUID="6874430f-3348-469c-adc3-a605fa1ff16a" containerName="payload-pre-processing" Oct 31 00:16:05.510615 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:05.510606 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6874430f-3348-469c-adc3-a605fa1ff16a" containerName="payload-pre-processing" Oct 31 00:16:05.510840 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:05.510621 4927 cpu_manager.go:414] "containerMap: removing container" podUID="1a2039e2-43bf-4f84-9c06-f7ba01c95c5d" containerName="cleanup" Oct 31 00:16:05.510840 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:05.510627 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a2039e2-43bf-4f84-9c06-f7ba01c95c5d" containerName="cleanup" Oct 31 00:16:05.510840 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:05.510639 4927 cpu_manager.go:414] "containerMap: removing container" podUID="1a2039e2-43bf-4f84-9c06-f7ba01c95c5d" containerName="cleanup" Oct 31 00:16:05.510840 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:05.510643 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a2039e2-43bf-4f84-9c06-f7ba01c95c5d" containerName="cleanup" Oct 31 00:16:05.510840 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:05.510659 4927 cpu_manager.go:414] "containerMap: removing container" podUID="5a2e97c5-6cd5-498d-b83f-13e94fc34b28" containerName="payload-processing" Oct 31 00:16:05.510840 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:05.510663 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a2e97c5-6cd5-498d-b83f-13e94fc34b28" containerName="payload-processing" Oct 31 00:16:05.510840 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:05.510680 4927 cpu_manager.go:414] "containerMap: removing container" podUID="5499c02e-e482-4329-b2b6-265ff423c83c" containerName="cleanup" Oct 31 00:16:05.510840 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:05.510685 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5499c02e-e482-4329-b2b6-265ff423c83c" containerName="cleanup" Oct 31 00:16:05.510840 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:05.510694 4927 cpu_manager.go:414] "containerMap: removing container" podUID="15c765a1-f9dd-4d99-a3d7-9447be49ecae" containerName="cleanup" Oct 31 00:16:05.510840 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:05.510698 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15c765a1-f9dd-4d99-a3d7-9447be49ecae" containerName="cleanup" Oct 31 00:16:05.510840 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:05.510708 4927 cpu_manager.go:414] "containerMap: removing container" podUID="6ecf922c-de0b-4e64-a2a2-07067e279990" containerName="maas-api" Oct 31 00:16:05.510840 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:05.510712 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ecf922c-de0b-4e64-a2a2-07067e279990" containerName="maas-api" Oct 31 00:16:05.510840 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:05.510721 4927 cpu_manager.go:414] "containerMap: removing container" podUID="5499c02e-e482-4329-b2b6-265ff423c83c" containerName="cleanup" Oct 31 00:16:05.510840 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:05.510726 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5499c02e-e482-4329-b2b6-265ff423c83c" containerName="cleanup" Oct 31 00:16:05.510840 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:05.510736 4927 cpu_manager.go:414] "containerMap: removing container" podUID="15c765a1-f9dd-4d99-a3d7-9447be49ecae" containerName="cleanup" Oct 31 00:16:05.510840 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:05.510741 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15c765a1-f9dd-4d99-a3d7-9447be49ecae" containerName="cleanup" Oct 31 00:16:05.510840 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:05.510748 4927 cpu_manager.go:414] "containerMap: removing container" podUID="5499c02e-e482-4329-b2b6-265ff423c83c" containerName="cleanup" Oct 31 00:16:05.510840 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:05.510752 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5499c02e-e482-4329-b2b6-265ff423c83c" containerName="cleanup" Oct 31 00:16:05.510840 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:05.510759 4927 cpu_manager.go:414] "containerMap: removing container" podUID="528cb403-6318-45aa-afac-39095d3a8731" containerName="cleanup" Oct 31 00:16:05.510840 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:05.510763 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="528cb403-6318-45aa-afac-39095d3a8731" containerName="cleanup" Oct 31 00:16:05.511456 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:05.510886 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="15c765a1-f9dd-4d99-a3d7-9447be49ecae" containerName="cleanup" Oct 31 00:16:05.511456 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:05.510894 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ecf922c-de0b-4e64-a2a2-07067e279990" containerName="maas-api" Oct 31 00:16:05.511456 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:05.510900 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="5499c02e-e482-4329-b2b6-265ff423c83c" containerName="cleanup" Oct 31 00:16:05.511456 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:05.510906 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="15c765a1-f9dd-4d99-a3d7-9447be49ecae" containerName="cleanup" Oct 31 00:16:05.511456 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:05.510913 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="528cb403-6318-45aa-afac-39095d3a8731" containerName="cleanup" Oct 31 00:16:05.511456 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:05.510920 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="528cb403-6318-45aa-afac-39095d3a8731" containerName="cleanup" Oct 31 00:16:05.511456 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:05.510927 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="15c765a1-f9dd-4d99-a3d7-9447be49ecae" containerName="cleanup" Oct 31 00:16:05.511456 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:05.510934 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a2039e2-43bf-4f84-9c06-f7ba01c95c5d" containerName="cleanup" Oct 31 00:16:05.511456 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:05.510940 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="6874430f-3348-469c-adc3-a605fa1ff16a" containerName="payload-pre-processing" Oct 31 00:16:05.511456 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:05.510948 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a2039e2-43bf-4f84-9c06-f7ba01c95c5d" containerName="cleanup" Oct 31 00:16:05.511456 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:05.510955 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a2e97c5-6cd5-498d-b83f-13e94fc34b28" containerName="payload-processing" Oct 31 00:16:05.511456 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:05.510962 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="5499c02e-e482-4329-b2b6-265ff423c83c" containerName="cleanup" Oct 31 00:16:05.516446 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:05.516429 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c9c21e-68d69b8994-hnwg4" Oct 31 00:16:05.570283 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:05.570242 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-c9c21e-68d69b8994-hnwg4"] Oct 31 00:16:05.606041 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:05.606005 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7k79d\" (UniqueName: \"kubernetes.io/projected/dbec9857-9469-47ff-8977-0f912e5d0a9c-kube-api-access-7k79d\") pod \"maas-api-e2e-api-a-c9c21e-68d69b8994-hnwg4\" (UID: \"dbec9857-9469-47ff-8977-0f912e5d0a9c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c9c21e-68d69b8994-hnwg4" Oct 31 00:16:05.606312 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:05.606283 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dbec9857-9469-47ff-8977-0f912e5d0a9c-metrics-certs\") pod \"maas-api-e2e-api-a-c9c21e-68d69b8994-hnwg4\" (UID: \"dbec9857-9469-47ff-8977-0f912e5d0a9c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c9c21e-68d69b8994-hnwg4" Oct 31 00:16:05.606465 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:05.606415 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dbec9857-9469-47ff-8977-0f912e5d0a9c-maas-api-tls\") pod \"maas-api-e2e-api-a-c9c21e-68d69b8994-hnwg4\" (UID: \"dbec9857-9469-47ff-8977-0f912e5d0a9c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c9c21e-68d69b8994-hnwg4" Oct 31 00:16:05.606465 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:05.606447 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dbec9857-9469-47ff-8977-0f912e5d0a9c-openshift-service-ca\") pod \"maas-api-e2e-api-a-c9c21e-68d69b8994-hnwg4\" (UID: \"dbec9857-9469-47ff-8977-0f912e5d0a9c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c9c21e-68d69b8994-hnwg4" Oct 31 00:16:05.615116 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:05.615083 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-c9c21e-serving-cert\"" Oct 31 00:16:05.673308 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:05.673275 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:16:05.707389 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:05.707357 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dbec9857-9469-47ff-8977-0f912e5d0a9c-metrics-certs\") pod \"maas-api-e2e-api-a-c9c21e-68d69b8994-hnwg4\" (UID: \"dbec9857-9469-47ff-8977-0f912e5d0a9c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c9c21e-68d69b8994-hnwg4" Oct 31 00:16:05.707563 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:05.707411 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dbec9857-9469-47ff-8977-0f912e5d0a9c-maas-api-tls\") pod \"maas-api-e2e-api-a-c9c21e-68d69b8994-hnwg4\" (UID: \"dbec9857-9469-47ff-8977-0f912e5d0a9c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c9c21e-68d69b8994-hnwg4" Oct 31 00:16:05.707563 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:05.707440 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dbec9857-9469-47ff-8977-0f912e5d0a9c-openshift-service-ca\") pod \"maas-api-e2e-api-a-c9c21e-68d69b8994-hnwg4\" (UID: \"dbec9857-9469-47ff-8977-0f912e5d0a9c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c9c21e-68d69b8994-hnwg4" Oct 31 00:16:05.707563 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:05.707503 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7k79d\" (UniqueName: \"kubernetes.io/projected/dbec9857-9469-47ff-8977-0f912e5d0a9c-kube-api-access-7k79d\") pod \"maas-api-e2e-api-a-c9c21e-68d69b8994-hnwg4\" (UID: \"dbec9857-9469-47ff-8977-0f912e5d0a9c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c9c21e-68d69b8994-hnwg4" Oct 31 00:16:05.707740 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:16:05.707568 4927 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-c9c21e-serving-cert: secret "maas-api-e2e-api-a-c9c21e-serving-cert" not found Oct 31 00:16:05.707740 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:16:05.707636 4927 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dbec9857-9469-47ff-8977-0f912e5d0a9c-maas-api-tls podName:dbec9857-9469-47ff-8977-0f912e5d0a9c nodeName:}" failed. No retries permitted until 2026-10-31 00:16:06.207615312 +0000 UTC m=+2079.243946133 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/dbec9857-9469-47ff-8977-0f912e5d0a9c-maas-api-tls") pod "maas-api-e2e-api-a-c9c21e-68d69b8994-hnwg4" (UID: "dbec9857-9469-47ff-8977-0f912e5d0a9c") : secret "maas-api-e2e-api-a-c9c21e-serving-cert" not found Oct 31 00:16:05.708125 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:05.708104 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dbec9857-9469-47ff-8977-0f912e5d0a9c-openshift-service-ca\") pod \"maas-api-e2e-api-a-c9c21e-68d69b8994-hnwg4\" (UID: \"dbec9857-9469-47ff-8977-0f912e5d0a9c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c9c21e-68d69b8994-hnwg4" Oct 31 00:16:05.709815 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:05.709799 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dbec9857-9469-47ff-8977-0f912e5d0a9c-metrics-certs\") pod \"maas-api-e2e-api-a-c9c21e-68d69b8994-hnwg4\" (UID: \"dbec9857-9469-47ff-8977-0f912e5d0a9c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c9c21e-68d69b8994-hnwg4" Oct 31 00:16:05.779988 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:05.779911 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7k79d\" (UniqueName: \"kubernetes.io/projected/dbec9857-9469-47ff-8977-0f912e5d0a9c-kube-api-access-7k79d\") pod \"maas-api-e2e-api-a-c9c21e-68d69b8994-hnwg4\" (UID: \"dbec9857-9469-47ff-8977-0f912e5d0a9c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c9c21e-68d69b8994-hnwg4" Oct 31 00:16:05.837956 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:05.837916 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:16:06.083112 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:06.083024 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:16:06.218020 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:06.217985 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dbec9857-9469-47ff-8977-0f912e5d0a9c-maas-api-tls\") pod \"maas-api-e2e-api-a-c9c21e-68d69b8994-hnwg4\" (UID: \"dbec9857-9469-47ff-8977-0f912e5d0a9c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c9c21e-68d69b8994-hnwg4" Oct 31 00:16:06.220572 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:06.220539 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dbec9857-9469-47ff-8977-0f912e5d0a9c-maas-api-tls\") pod \"maas-api-e2e-api-a-c9c21e-68d69b8994-hnwg4\" (UID: \"dbec9857-9469-47ff-8977-0f912e5d0a9c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c9c21e-68d69b8994-hnwg4" Oct 31 00:16:06.431036 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:06.430999 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c9c21e-68d69b8994-hnwg4" Oct 31 00:16:06.827266 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:06.827236 4927 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 00:16:06.828787 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:06.828748 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-c9c21e-68d69b8994-hnwg4"] Oct 31 00:16:07.188093 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:07.188053 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-zvnnz"] Oct 31 00:16:07.188433 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:07.188380 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-zvnnz" podUID="6ba0721f-a4d2-49f6-b161-6d6f0f8ae4c2" containerName="istio-proxy" containerID="cri-o://8257ddc463e6fa3295e8c23e45ff1f3698e14ecabea72d76fe68ae6b3477a1d6" gracePeriod=30 Oct 31 00:16:07.236238 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:07.236199 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-8fws9"] Oct 31 00:16:07.236486 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:07.236457 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-8fws9" podUID="daac49a1-fcf0-4772-b555-5f43b01f8c3f" containerName="istio-proxy" containerID="cri-o://cbb778d1425557c3b18f70209e5b26a2f2daaf067005fd468bb053b35672a38b" gracePeriod=30 Oct 31 00:16:07.241908 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:07.241875 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-dpcqs"] Oct 31 00:16:07.242158 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:07.242126 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-dpcqs" podUID="8328651d-33b5-4160-8f99-9d6de87ba69f" containerName="istio-proxy" containerID="cri-o://08fc703beb7ed599fb280382f6d36d5ac0040a3441246ecb40098cb8c7e9cbe2" gracePeriod=30 Oct 31 00:16:07.616453 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:07.616365 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c9c21e-68d69b8994-hnwg4" event={"ID":"dbec9857-9469-47ff-8977-0f912e5d0a9c","Type":"ContainerStarted","Data":"69ff9641ddad1615b0bc1b55d6d9b1f82fdd0179e092830ec55f4b7cb3adc74e"} Oct 31 00:16:08.393767 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:08.393729 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:16:08.604713 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:08.604678 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:16:08.630438 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:08.630384 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c9c21e-68d69b8994-hnwg4" event={"ID":"dbec9857-9469-47ff-8977-0f912e5d0a9c","Type":"ContainerStarted","Data":"940cdd8be4499fe8a28e46a34be6582ec41c7eaa033c5b80b90ac40c8e3ad083"} Oct 31 00:16:08.630648 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:08.630523 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c9c21e-68d69b8994-hnwg4" Oct 31 00:16:08.744682 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:08.744619 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-54ad8c-openshift-default-575c545fdf-gvfbr"] Oct 31 00:16:08.745004 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:08.744936 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-54ad8c-openshift-default-575c545fdf-gvfbr" podUID="85098bb1-0c3e-4ecb-a771-3f57503a7cd8" containerName="istio-proxy" containerID="cri-o://efb5ceaf3b35dc1ac4f015f54c759b4866359273451e2abac9bdbfd626bbca2a" gracePeriod=30 Oct 31 00:16:08.766839 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:08.766550 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:16:08.789412 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:08.789361 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c9c21e-68d69b8994-hnwg4" podStartSLOduration=2.852941878 podStartE2EDuration="3.789347678s" totalImagesPullingTime="936.4058ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:16:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:16:08.785813862 +0000 UTC m=+2081.822144683" watchObservedRunningTime="2026-10-31 00:16:08.789347678 +0000 UTC m=+2081.825678499" Oct 31 00:16:10.780066 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:10.780024 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:16:11.593007 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:11.592971 4927 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-zvnnz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 00:16:11.593193 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:11.593041 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-zvnnz" podUID="6ba0721f-a4d2-49f6-b161-6d6f0f8ae4c2" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:16:12.462548 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:12.462051 4927 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-dpcqs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" start-of-body= Oct 31 00:16:12.462548 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:12.462119 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-dpcqs" podUID="8328651d-33b5-4160-8f99-9d6de87ba69f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" Oct 31 00:16:12.592359 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:12.590436 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-zvnnz" Oct 31 00:16:12.628290 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:12.628260 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-8fws9" Oct 31 00:16:12.651921 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:12.651895 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-dpcqs" Oct 31 00:16:12.695977 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:12.695946 4927 generic.go:356] "Generic (PLEG): container finished" podID="6ba0721f-a4d2-49f6-b161-6d6f0f8ae4c2" containerID="8257ddc463e6fa3295e8c23e45ff1f3698e14ecabea72d76fe68ae6b3477a1d6" exitCode=0 Oct 31 00:16:12.696160 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:12.696007 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-zvnnz" Oct 31 00:16:12.696160 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:12.696027 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-zvnnz" event={"ID":"6ba0721f-a4d2-49f6-b161-6d6f0f8ae4c2","Type":"ContainerDied","Data":"8257ddc463e6fa3295e8c23e45ff1f3698e14ecabea72d76fe68ae6b3477a1d6"} Oct 31 00:16:12.696160 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:12.696076 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-zvnnz" event={"ID":"6ba0721f-a4d2-49f6-b161-6d6f0f8ae4c2","Type":"ContainerDied","Data":"dad7abb14198704cf22325290071d73dc8b4e57f05687463f2dc38060448473a"} Oct 31 00:16:12.696160 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:12.696104 4927 scope.go:122] "RemoveContainer" containerID="8257ddc463e6fa3295e8c23e45ff1f3698e14ecabea72d76fe68ae6b3477a1d6" Oct 31 00:16:12.697563 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:12.697543 4927 generic.go:356] "Generic (PLEG): container finished" podID="8328651d-33b5-4160-8f99-9d6de87ba69f" containerID="08fc703beb7ed599fb280382f6d36d5ac0040a3441246ecb40098cb8c7e9cbe2" exitCode=0 Oct 31 00:16:12.697665 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:12.697584 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-dpcqs" Oct 31 00:16:12.697665 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:12.697635 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-dpcqs" event={"ID":"8328651d-33b5-4160-8f99-9d6de87ba69f","Type":"ContainerDied","Data":"08fc703beb7ed599fb280382f6d36d5ac0040a3441246ecb40098cb8c7e9cbe2"} Oct 31 00:16:12.697779 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:12.697685 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-dpcqs" event={"ID":"8328651d-33b5-4160-8f99-9d6de87ba69f","Type":"ContainerDied","Data":"ebffea387d264ae2137ca37fe1e4702608376ff1b3a84e25f9ae6059cc853d1c"} Oct 31 00:16:12.699288 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:12.699265 4927 generic.go:356] "Generic (PLEG): container finished" podID="daac49a1-fcf0-4772-b555-5f43b01f8c3f" containerID="cbb778d1425557c3b18f70209e5b26a2f2daaf067005fd468bb053b35672a38b" exitCode=0 Oct 31 00:16:12.699398 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:12.699290 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-8fws9" event={"ID":"daac49a1-fcf0-4772-b555-5f43b01f8c3f","Type":"ContainerDied","Data":"cbb778d1425557c3b18f70209e5b26a2f2daaf067005fd468bb053b35672a38b"} Oct 31 00:16:12.699398 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:12.699321 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-8fws9" Oct 31 00:16:12.699398 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:12.699343 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-8fws9" event={"ID":"daac49a1-fcf0-4772-b555-5f43b01f8c3f","Type":"ContainerDied","Data":"7094a9598ef2afef35c2335160d4eee0bd375e4ac40f7322204bce0ec654f0df"} Oct 31 00:16:12.707861 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:12.707838 4927 scope.go:122] "RemoveContainer" containerID="8257ddc463e6fa3295e8c23e45ff1f3698e14ecabea72d76fe68ae6b3477a1d6" Oct 31 00:16:12.708157 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:16:12.708130 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8257ddc463e6fa3295e8c23e45ff1f3698e14ecabea72d76fe68ae6b3477a1d6\": container with ID starting with 8257ddc463e6fa3295e8c23e45ff1f3698e14ecabea72d76fe68ae6b3477a1d6 not found: ID does not exist" containerID="8257ddc463e6fa3295e8c23e45ff1f3698e14ecabea72d76fe68ae6b3477a1d6" Oct 31 00:16:12.708239 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:12.708162 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8257ddc463e6fa3295e8c23e45ff1f3698e14ecabea72d76fe68ae6b3477a1d6"} err="failed to get container status \"8257ddc463e6fa3295e8c23e45ff1f3698e14ecabea72d76fe68ae6b3477a1d6\": rpc error: code = NotFound desc = could not find container \"8257ddc463e6fa3295e8c23e45ff1f3698e14ecabea72d76fe68ae6b3477a1d6\": container with ID starting with 8257ddc463e6fa3295e8c23e45ff1f3698e14ecabea72d76fe68ae6b3477a1d6 not found: ID does not exist" Oct 31 00:16:12.708239 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:12.708181 4927 scope.go:122] "RemoveContainer" containerID="08fc703beb7ed599fb280382f6d36d5ac0040a3441246ecb40098cb8c7e9cbe2" Oct 31 00:16:12.719520 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:12.719499 4927 scope.go:122] "RemoveContainer" containerID="08fc703beb7ed599fb280382f6d36d5ac0040a3441246ecb40098cb8c7e9cbe2" Oct 31 00:16:12.719873 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:16:12.719850 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08fc703beb7ed599fb280382f6d36d5ac0040a3441246ecb40098cb8c7e9cbe2\": container with ID starting with 08fc703beb7ed599fb280382f6d36d5ac0040a3441246ecb40098cb8c7e9cbe2 not found: ID does not exist" containerID="08fc703beb7ed599fb280382f6d36d5ac0040a3441246ecb40098cb8c7e9cbe2" Oct 31 00:16:12.719984 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:12.719882 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08fc703beb7ed599fb280382f6d36d5ac0040a3441246ecb40098cb8c7e9cbe2"} err="failed to get container status \"08fc703beb7ed599fb280382f6d36d5ac0040a3441246ecb40098cb8c7e9cbe2\": rpc error: code = NotFound desc = could not find container \"08fc703beb7ed599fb280382f6d36d5ac0040a3441246ecb40098cb8c7e9cbe2\": container with ID starting with 08fc703beb7ed599fb280382f6d36d5ac0040a3441246ecb40098cb8c7e9cbe2 not found: ID does not exist" Oct 31 00:16:12.719984 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:12.719906 4927 scope.go:122] "RemoveContainer" containerID="cbb778d1425557c3b18f70209e5b26a2f2daaf067005fd468bb053b35672a38b" Oct 31 00:16:12.730433 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:12.730412 4927 scope.go:122] "RemoveContainer" containerID="cbb778d1425557c3b18f70209e5b26a2f2daaf067005fd468bb053b35672a38b" Oct 31 00:16:12.730742 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:16:12.730720 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cbb778d1425557c3b18f70209e5b26a2f2daaf067005fd468bb053b35672a38b\": container with ID starting with cbb778d1425557c3b18f70209e5b26a2f2daaf067005fd468bb053b35672a38b not found: ID does not exist" containerID="cbb778d1425557c3b18f70209e5b26a2f2daaf067005fd468bb053b35672a38b" Oct 31 00:16:12.730800 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:12.730752 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cbb778d1425557c3b18f70209e5b26a2f2daaf067005fd468bb053b35672a38b"} err="failed to get container status \"cbb778d1425557c3b18f70209e5b26a2f2daaf067005fd468bb053b35672a38b\": rpc error: code = NotFound desc = could not find container \"cbb778d1425557c3b18f70209e5b26a2f2daaf067005fd468bb053b35672a38b\": container with ID starting with cbb778d1425557c3b18f70209e5b26a2f2daaf067005fd468bb053b35672a38b not found: ID does not exist" Oct 31 00:16:12.764622 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:12.764584 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8328651d-33b5-4160-8f99-9d6de87ba69f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8328651d-33b5-4160-8f99-9d6de87ba69f-istio-envoy\") pod \"8328651d-33b5-4160-8f99-9d6de87ba69f\" (UID: \"8328651d-33b5-4160-8f99-9d6de87ba69f\") " Oct 31 00:16:12.764830 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:12.764635 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/daac49a1-fcf0-4772-b555-5f43b01f8c3f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/daac49a1-fcf0-4772-b555-5f43b01f8c3f-credential-socket\") pod \"daac49a1-fcf0-4772-b555-5f43b01f8c3f\" (UID: \"daac49a1-fcf0-4772-b555-5f43b01f8c3f\") " Oct 31 00:16:12.764830 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:12.764669 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/daac49a1-fcf0-4772-b555-5f43b01f8c3f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/daac49a1-fcf0-4772-b555-5f43b01f8c3f-istio-envoy\") pod \"daac49a1-fcf0-4772-b555-5f43b01f8c3f\" (UID: \"daac49a1-fcf0-4772-b555-5f43b01f8c3f\") " Oct 31 00:16:12.764830 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:12.764745 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8328651d-33b5-4160-8f99-9d6de87ba69f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8328651d-33b5-4160-8f99-9d6de87ba69f-credential-socket\") pod \"8328651d-33b5-4160-8f99-9d6de87ba69f\" (UID: \"8328651d-33b5-4160-8f99-9d6de87ba69f\") " Oct 31 00:16:12.764830 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:12.764811 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8328651d-33b5-4160-8f99-9d6de87ba69f-istio-token\" (UniqueName: \"kubernetes.io/projected/8328651d-33b5-4160-8f99-9d6de87ba69f-istio-token\") pod \"8328651d-33b5-4160-8f99-9d6de87ba69f\" (UID: \"8328651d-33b5-4160-8f99-9d6de87ba69f\") " Oct 31 00:16:12.766567 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:12.764856 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/daac49a1-fcf0-4772-b555-5f43b01f8c3f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/daac49a1-fcf0-4772-b555-5f43b01f8c3f-istio-data\") pod \"daac49a1-fcf0-4772-b555-5f43b01f8c3f\" (UID: \"daac49a1-fcf0-4772-b555-5f43b01f8c3f\") " Oct 31 00:16:12.766567 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:12.764894 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/daac49a1-fcf0-4772-b555-5f43b01f8c3f-istio-token\" (UniqueName: \"kubernetes.io/projected/daac49a1-fcf0-4772-b555-5f43b01f8c3f-istio-token\") pod \"daac49a1-fcf0-4772-b555-5f43b01f8c3f\" (UID: \"daac49a1-fcf0-4772-b555-5f43b01f8c3f\") " Oct 31 00:16:12.766567 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:12.764921 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8328651d-33b5-4160-8f99-9d6de87ba69f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8328651d-33b5-4160-8f99-9d6de87ba69f-istio-podinfo\") pod \"8328651d-33b5-4160-8f99-9d6de87ba69f\" (UID: \"8328651d-33b5-4160-8f99-9d6de87ba69f\") " Oct 31 00:16:12.766567 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:12.764974 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ba0721f-a4d2-49f6-b161-6d6f0f8ae4c2-istio-token\" (UniqueName: \"kubernetes.io/projected/6ba0721f-a4d2-49f6-b161-6d6f0f8ae4c2-istio-token\") pod \"6ba0721f-a4d2-49f6-b161-6d6f0f8ae4c2\" (UID: \"6ba0721f-a4d2-49f6-b161-6d6f0f8ae4c2\") " Oct 31 00:16:12.766567 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:12.765001 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6ba0721f-a4d2-49f6-b161-6d6f0f8ae4c2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ba0721f-a4d2-49f6-b161-6d6f0f8ae4c2-istio-podinfo\") pod \"6ba0721f-a4d2-49f6-b161-6d6f0f8ae4c2\" (UID: \"6ba0721f-a4d2-49f6-b161-6d6f0f8ae4c2\") " Oct 31 00:16:12.766567 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:12.765037 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ba0721f-a4d2-49f6-b161-6d6f0f8ae4c2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ba0721f-a4d2-49f6-b161-6d6f0f8ae4c2-istio-data\") pod \"6ba0721f-a4d2-49f6-b161-6d6f0f8ae4c2\" (UID: \"6ba0721f-a4d2-49f6-b161-6d6f0f8ae4c2\") " Oct 31 00:16:12.766567 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:12.765060 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8328651d-33b5-4160-8f99-9d6de87ba69f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8328651d-33b5-4160-8f99-9d6de87ba69f-istiod-ca-cert\") pod \"8328651d-33b5-4160-8f99-9d6de87ba69f\" (UID: \"8328651d-33b5-4160-8f99-9d6de87ba69f\") " Oct 31 00:16:12.766567 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:12.765109 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8328651d-33b5-4160-8f99-9d6de87ba69f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8328651d-33b5-4160-8f99-9d6de87ba69f-istio-data\") pod \"8328651d-33b5-4160-8f99-9d6de87ba69f\" (UID: \"8328651d-33b5-4160-8f99-9d6de87ba69f\") " Oct 31 00:16:12.766567 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:12.765131 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ba0721f-a4d2-49f6-b161-6d6f0f8ae4c2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ba0721f-a4d2-49f6-b161-6d6f0f8ae4c2-istiod-ca-cert\") pod \"6ba0721f-a4d2-49f6-b161-6d6f0f8ae4c2\" (UID: \"6ba0721f-a4d2-49f6-b161-6d6f0f8ae4c2\") " Oct 31 00:16:12.766567 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:12.765185 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ba0721f-a4d2-49f6-b161-6d6f0f8ae4c2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ba0721f-a4d2-49f6-b161-6d6f0f8ae4c2-workload-socket\") pod \"6ba0721f-a4d2-49f6-b161-6d6f0f8ae4c2\" (UID: \"6ba0721f-a4d2-49f6-b161-6d6f0f8ae4c2\") " Oct 31 00:16:12.766567 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:12.765259 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8328651d-33b5-4160-8f99-9d6de87ba69f-credential-socket" pod "8328651d-33b5-4160-8f99-9d6de87ba69f" (UID: "8328651d-33b5-4160-8f99-9d6de87ba69f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:16:12.766567 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:12.765274 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/daac49a1-fcf0-4772-b555-5f43b01f8c3f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/daac49a1-fcf0-4772-b555-5f43b01f8c3f-workload-socket\") pod \"daac49a1-fcf0-4772-b555-5f43b01f8c3f\" (UID: \"daac49a1-fcf0-4772-b555-5f43b01f8c3f\") " Oct 31 00:16:12.766567 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:12.765542 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/daac49a1-fcf0-4772-b555-5f43b01f8c3f-kube-api-access-wbxjq\" (UniqueName: \"kubernetes.io/projected/daac49a1-fcf0-4772-b555-5f43b01f8c3f-kube-api-access-wbxjq\") pod \"daac49a1-fcf0-4772-b555-5f43b01f8c3f\" (UID: \"daac49a1-fcf0-4772-b555-5f43b01f8c3f\") " Oct 31 00:16:12.766567 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:12.765621 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/daac49a1-fcf0-4772-b555-5f43b01f8c3f-workload-socket" pod "daac49a1-fcf0-4772-b555-5f43b01f8c3f" (UID: "daac49a1-fcf0-4772-b555-5f43b01f8c3f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:16:12.766567 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:12.765645 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/daac49a1-fcf0-4772-b555-5f43b01f8c3f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/daac49a1-fcf0-4772-b555-5f43b01f8c3f-istio-podinfo\") pod \"daac49a1-fcf0-4772-b555-5f43b01f8c3f\" (UID: \"daac49a1-fcf0-4772-b555-5f43b01f8c3f\") " Oct 31 00:16:12.766567 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:12.766283 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/daac49a1-fcf0-4772-b555-5f43b01f8c3f-credential-socket" pod "daac49a1-fcf0-4772-b555-5f43b01f8c3f" (UID: "daac49a1-fcf0-4772-b555-5f43b01f8c3f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:16:12.766567 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:12.766431 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/daac49a1-fcf0-4772-b555-5f43b01f8c3f-istio-data" pod "daac49a1-fcf0-4772-b555-5f43b01f8c3f" (UID: "daac49a1-fcf0-4772-b555-5f43b01f8c3f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:16:12.766567 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:12.766478 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/daac49a1-fcf0-4772-b555-5f43b01f8c3f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/daac49a1-fcf0-4772-b555-5f43b01f8c3f-workload-certs\") pod \"daac49a1-fcf0-4772-b555-5f43b01f8c3f\" (UID: \"daac49a1-fcf0-4772-b555-5f43b01f8c3f\") " Oct 31 00:16:12.766567 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:12.766521 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8328651d-33b5-4160-8f99-9d6de87ba69f-kube-api-access-4zhlt\" (UniqueName: \"kubernetes.io/projected/8328651d-33b5-4160-8f99-9d6de87ba69f-kube-api-access-4zhlt\") pod \"8328651d-33b5-4160-8f99-9d6de87ba69f\" (UID: \"8328651d-33b5-4160-8f99-9d6de87ba69f\") " Oct 31 00:16:12.767617 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:12.766604 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ba0721f-a4d2-49f6-b161-6d6f0f8ae4c2-workload-socket" pod "6ba0721f-a4d2-49f6-b161-6d6f0f8ae4c2" (UID: "6ba0721f-a4d2-49f6-b161-6d6f0f8ae4c2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:16:12.767617 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:12.766616 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ba0721f-a4d2-49f6-b161-6d6f0f8ae4c2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ba0721f-a4d2-49f6-b161-6d6f0f8ae4c2-credential-socket\") pod \"6ba0721f-a4d2-49f6-b161-6d6f0f8ae4c2\" (UID: \"6ba0721f-a4d2-49f6-b161-6d6f0f8ae4c2\") " Oct 31 00:16:12.767617 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:12.766633 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8328651d-33b5-4160-8f99-9d6de87ba69f-istio-data" pod "8328651d-33b5-4160-8f99-9d6de87ba69f" (UID: "8328651d-33b5-4160-8f99-9d6de87ba69f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:16:12.767617 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:12.766665 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ba0721f-a4d2-49f6-b161-6d6f0f8ae4c2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ba0721f-a4d2-49f6-b161-6d6f0f8ae4c2-workload-certs\") pod \"6ba0721f-a4d2-49f6-b161-6d6f0f8ae4c2\" (UID: \"6ba0721f-a4d2-49f6-b161-6d6f0f8ae4c2\") " Oct 31 00:16:12.767617 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:12.766689 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8328651d-33b5-4160-8f99-9d6de87ba69f-istiod-ca-cert" pod "8328651d-33b5-4160-8f99-9d6de87ba69f" (UID: "8328651d-33b5-4160-8f99-9d6de87ba69f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:16:12.767617 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:12.766730 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ba0721f-a4d2-49f6-b161-6d6f0f8ae4c2-kube-api-access-55vnq\" (UniqueName: \"kubernetes.io/projected/6ba0721f-a4d2-49f6-b161-6d6f0f8ae4c2-kube-api-access-55vnq\") pod \"6ba0721f-a4d2-49f6-b161-6d6f0f8ae4c2\" (UID: \"6ba0721f-a4d2-49f6-b161-6d6f0f8ae4c2\") " Oct 31 00:16:12.767617 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:12.766786 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8328651d-33b5-4160-8f99-9d6de87ba69f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8328651d-33b5-4160-8f99-9d6de87ba69f-workload-certs\") pod \"8328651d-33b5-4160-8f99-9d6de87ba69f\" (UID: \"8328651d-33b5-4160-8f99-9d6de87ba69f\") " Oct 31 00:16:12.767617 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:12.766832 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ba0721f-a4d2-49f6-b161-6d6f0f8ae4c2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ba0721f-a4d2-49f6-b161-6d6f0f8ae4c2-istio-envoy\") pod \"6ba0721f-a4d2-49f6-b161-6d6f0f8ae4c2\" (UID: \"6ba0721f-a4d2-49f6-b161-6d6f0f8ae4c2\") " Oct 31 00:16:12.767617 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:12.766839 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ba0721f-a4d2-49f6-b161-6d6f0f8ae4c2-istio-data" pod "6ba0721f-a4d2-49f6-b161-6d6f0f8ae4c2" (UID: "6ba0721f-a4d2-49f6-b161-6d6f0f8ae4c2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:16:12.767617 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:12.766861 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/daac49a1-fcf0-4772-b555-5f43b01f8c3f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/daac49a1-fcf0-4772-b555-5f43b01f8c3f-istiod-ca-cert\") pod \"daac49a1-fcf0-4772-b555-5f43b01f8c3f\" (UID: \"daac49a1-fcf0-4772-b555-5f43b01f8c3f\") " Oct 31 00:16:12.767617 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:12.766893 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8328651d-33b5-4160-8f99-9d6de87ba69f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8328651d-33b5-4160-8f99-9d6de87ba69f-workload-socket\") pod \"8328651d-33b5-4160-8f99-9d6de87ba69f\" (UID: \"8328651d-33b5-4160-8f99-9d6de87ba69f\") " Oct 31 00:16:12.767617 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:12.766966 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ba0721f-a4d2-49f6-b161-6d6f0f8ae4c2-istiod-ca-cert" pod "6ba0721f-a4d2-49f6-b161-6d6f0f8ae4c2" (UID: "6ba0721f-a4d2-49f6-b161-6d6f0f8ae4c2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:16:12.767617 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:12.767237 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ba0721f-a4d2-49f6-b161-6d6f0f8ae4c2-workload-certs" pod "6ba0721f-a4d2-49f6-b161-6d6f0f8ae4c2" (UID: "6ba0721f-a4d2-49f6-b161-6d6f0f8ae4c2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:16:12.768278 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:12.767837 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8328651d-33b5-4160-8f99-9d6de87ba69f-workload-socket" pod "8328651d-33b5-4160-8f99-9d6de87ba69f" (UID: "8328651d-33b5-4160-8f99-9d6de87ba69f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:16:12.768278 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:12.767894 4927 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/daac49a1-fcf0-4772-b555-5f43b01f8c3f-credential-socket\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:16:12.768278 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:12.767921 4927 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8328651d-33b5-4160-8f99-9d6de87ba69f-credential-socket\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:16:12.768278 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:12.767938 4927 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/daac49a1-fcf0-4772-b555-5f43b01f8c3f-istio-data\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:16:12.768278 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:12.767954 4927 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ba0721f-a4d2-49f6-b161-6d6f0f8ae4c2-istio-data\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:16:12.768278 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:12.767968 4927 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8328651d-33b5-4160-8f99-9d6de87ba69f-istiod-ca-cert\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:16:12.768278 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:12.767982 4927 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8328651d-33b5-4160-8f99-9d6de87ba69f-istio-data\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:16:12.768278 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:12.767996 4927 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ba0721f-a4d2-49f6-b161-6d6f0f8ae4c2-istiod-ca-cert\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:16:12.768278 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:12.768010 4927 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ba0721f-a4d2-49f6-b161-6d6f0f8ae4c2-workload-socket\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:16:12.768278 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:12.768025 4927 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/daac49a1-fcf0-4772-b555-5f43b01f8c3f-workload-socket\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:16:12.768278 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:12.768038 4927 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ba0721f-a4d2-49f6-b161-6d6f0f8ae4c2-workload-certs\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:16:12.768278 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:12.768264 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/daac49a1-fcf0-4772-b555-5f43b01f8c3f-workload-certs" pod "daac49a1-fcf0-4772-b555-5f43b01f8c3f" (UID: "daac49a1-fcf0-4772-b555-5f43b01f8c3f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:16:12.768963 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:12.768492 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/daac49a1-fcf0-4772-b555-5f43b01f8c3f-istio-token" pod "daac49a1-fcf0-4772-b555-5f43b01f8c3f" (UID: "daac49a1-fcf0-4772-b555-5f43b01f8c3f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:16:12.769781 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:12.769751 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/daac49a1-fcf0-4772-b555-5f43b01f8c3f-istiod-ca-cert" pod "daac49a1-fcf0-4772-b555-5f43b01f8c3f" (UID: "daac49a1-fcf0-4772-b555-5f43b01f8c3f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:16:12.769914 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:12.769828 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8328651d-33b5-4160-8f99-9d6de87ba69f-istio-envoy" pod "8328651d-33b5-4160-8f99-9d6de87ba69f" (UID: "8328651d-33b5-4160-8f99-9d6de87ba69f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:16:12.770395 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:12.770320 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8328651d-33b5-4160-8f99-9d6de87ba69f-workload-certs" pod "8328651d-33b5-4160-8f99-9d6de87ba69f" (UID: "8328651d-33b5-4160-8f99-9d6de87ba69f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:16:12.770576 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:12.770550 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ba0721f-a4d2-49f6-b161-6d6f0f8ae4c2-credential-socket" pod "6ba0721f-a4d2-49f6-b161-6d6f0f8ae4c2" (UID: "6ba0721f-a4d2-49f6-b161-6d6f0f8ae4c2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:16:12.771566 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:12.771546 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ba0721f-a4d2-49f6-b161-6d6f0f8ae4c2-istio-envoy" pod "6ba0721f-a4d2-49f6-b161-6d6f0f8ae4c2" (UID: "6ba0721f-a4d2-49f6-b161-6d6f0f8ae4c2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:16:12.771950 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:12.771930 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8328651d-33b5-4160-8f99-9d6de87ba69f-istio-podinfo" pod "8328651d-33b5-4160-8f99-9d6de87ba69f" (UID: "8328651d-33b5-4160-8f99-9d6de87ba69f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:16:12.772195 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:12.772176 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8328651d-33b5-4160-8f99-9d6de87ba69f-kube-api-access-4zhlt" pod "8328651d-33b5-4160-8f99-9d6de87ba69f" (UID: "8328651d-33b5-4160-8f99-9d6de87ba69f"). InnerVolumeSpecName "kube-api-access-4zhlt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:16:12.772275 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:12.772219 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ba0721f-a4d2-49f6-b161-6d6f0f8ae4c2-kube-api-access-55vnq" pod "6ba0721f-a4d2-49f6-b161-6d6f0f8ae4c2" (UID: "6ba0721f-a4d2-49f6-b161-6d6f0f8ae4c2"). InnerVolumeSpecName "kube-api-access-55vnq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:16:12.774278 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:12.774170 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6ba0721f-a4d2-49f6-b161-6d6f0f8ae4c2-istio-podinfo" pod "6ba0721f-a4d2-49f6-b161-6d6f0f8ae4c2" (UID: "6ba0721f-a4d2-49f6-b161-6d6f0f8ae4c2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:16:12.774278 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:12.774181 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/daac49a1-fcf0-4772-b555-5f43b01f8c3f-istio-envoy" pod "daac49a1-fcf0-4772-b555-5f43b01f8c3f" (UID: "daac49a1-fcf0-4772-b555-5f43b01f8c3f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:16:12.774278 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:12.774198 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/daac49a1-fcf0-4772-b555-5f43b01f8c3f-istio-podinfo" pod "daac49a1-fcf0-4772-b555-5f43b01f8c3f" (UID: "daac49a1-fcf0-4772-b555-5f43b01f8c3f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:16:12.774278 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:12.774221 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/daac49a1-fcf0-4772-b555-5f43b01f8c3f-kube-api-access-wbxjq" pod "daac49a1-fcf0-4772-b555-5f43b01f8c3f" (UID: "daac49a1-fcf0-4772-b555-5f43b01f8c3f"). InnerVolumeSpecName "kube-api-access-wbxjq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:16:12.774278 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:12.774253 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8328651d-33b5-4160-8f99-9d6de87ba69f-istio-token" pod "8328651d-33b5-4160-8f99-9d6de87ba69f" (UID: "8328651d-33b5-4160-8f99-9d6de87ba69f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:16:12.775570 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:12.775550 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ba0721f-a4d2-49f6-b161-6d6f0f8ae4c2-istio-token" pod "6ba0721f-a4d2-49f6-b161-6d6f0f8ae4c2" (UID: "6ba0721f-a4d2-49f6-b161-6d6f0f8ae4c2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:16:12.869013 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:12.868974 4927 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8328651d-33b5-4160-8f99-9d6de87ba69f-istio-podinfo\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:16:12.869013 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:12.869008 4927 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ba0721f-a4d2-49f6-b161-6d6f0f8ae4c2-istio-token\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:16:12.869013 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:12.869018 4927 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ba0721f-a4d2-49f6-b161-6d6f0f8ae4c2-istio-podinfo\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:16:12.869265 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:12.869027 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wbxjq\" (UniqueName: \"kubernetes.io/projected/daac49a1-fcf0-4772-b555-5f43b01f8c3f-kube-api-access-wbxjq\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:16:12.869265 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:12.869038 4927 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/daac49a1-fcf0-4772-b555-5f43b01f8c3f-istio-podinfo\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:16:12.869265 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:12.869046 4927 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/daac49a1-fcf0-4772-b555-5f43b01f8c3f-workload-certs\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:16:12.869265 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:12.869055 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4zhlt\" (UniqueName: \"kubernetes.io/projected/8328651d-33b5-4160-8f99-9d6de87ba69f-kube-api-access-4zhlt\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:16:12.869265 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:12.869064 4927 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ba0721f-a4d2-49f6-b161-6d6f0f8ae4c2-credential-socket\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:16:12.869265 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:12.869073 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-55vnq\" (UniqueName: \"kubernetes.io/projected/6ba0721f-a4d2-49f6-b161-6d6f0f8ae4c2-kube-api-access-55vnq\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:16:12.869265 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:12.869083 4927 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8328651d-33b5-4160-8f99-9d6de87ba69f-workload-certs\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:16:12.869265 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:12.869091 4927 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ba0721f-a4d2-49f6-b161-6d6f0f8ae4c2-istio-envoy\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:16:12.869265 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:12.869100 4927 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/daac49a1-fcf0-4772-b555-5f43b01f8c3f-istiod-ca-cert\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:16:12.869265 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:12.869108 4927 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8328651d-33b5-4160-8f99-9d6de87ba69f-workload-socket\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:16:12.869265 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:12.869116 4927 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8328651d-33b5-4160-8f99-9d6de87ba69f-istio-envoy\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:16:12.869265 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:12.869129 4927 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/daac49a1-fcf0-4772-b555-5f43b01f8c3f-istio-envoy\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:16:12.869265 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:12.869137 4927 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8328651d-33b5-4160-8f99-9d6de87ba69f-istio-token\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:16:12.869265 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:12.869146 4927 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/daac49a1-fcf0-4772-b555-5f43b01f8c3f-istio-token\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:16:13.189792 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:13.189753 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-8fws9"] Oct 31 00:16:13.194255 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:13.194233 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-8fws9"] Oct 31 00:16:13.306795 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:13.306712 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-dpcqs"] Oct 31 00:16:13.311948 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:13.311921 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-dpcqs"] Oct 31 00:16:13.429785 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:13.429747 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-zvnnz"] Oct 31 00:16:13.436846 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:13.436814 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-zvnnz"] Oct 31 00:16:13.584524 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:13.584436 4927 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-8fws9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:16:13.584524 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:13.584500 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-8fws9" podUID="daac49a1-fcf0-4772-b555-5f43b01f8c3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:16:14.063108 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:14.063079 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-54ad8c-openshift-default-575c545fdf-gvfbr" Oct 31 00:16:14.192922 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:14.192882 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85098bb1-0c3e-4ecb-a771-3f57503a7cd8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85098bb1-0c3e-4ecb-a771-3f57503a7cd8-istio-envoy\") pod \"85098bb1-0c3e-4ecb-a771-3f57503a7cd8\" (UID: \"85098bb1-0c3e-4ecb-a771-3f57503a7cd8\") " Oct 31 00:16:14.193113 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:14.192966 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85098bb1-0c3e-4ecb-a771-3f57503a7cd8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85098bb1-0c3e-4ecb-a771-3f57503a7cd8-credential-socket\") pod \"85098bb1-0c3e-4ecb-a771-3f57503a7cd8\" (UID: \"85098bb1-0c3e-4ecb-a771-3f57503a7cd8\") " Oct 31 00:16:14.193113 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:14.193033 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/85098bb1-0c3e-4ecb-a771-3f57503a7cd8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85098bb1-0c3e-4ecb-a771-3f57503a7cd8-istiod-ca-cert\") pod \"85098bb1-0c3e-4ecb-a771-3f57503a7cd8\" (UID: \"85098bb1-0c3e-4ecb-a771-3f57503a7cd8\") " Oct 31 00:16:14.193113 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:14.193060 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85098bb1-0c3e-4ecb-a771-3f57503a7cd8-istio-token\" (UniqueName: \"kubernetes.io/projected/85098bb1-0c3e-4ecb-a771-3f57503a7cd8-istio-token\") pod \"85098bb1-0c3e-4ecb-a771-3f57503a7cd8\" (UID: \"85098bb1-0c3e-4ecb-a771-3f57503a7cd8\") " Oct 31 00:16:14.193317 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:14.193133 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85098bb1-0c3e-4ecb-a771-3f57503a7cd8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85098bb1-0c3e-4ecb-a771-3f57503a7cd8-workload-socket\") pod \"85098bb1-0c3e-4ecb-a771-3f57503a7cd8\" (UID: \"85098bb1-0c3e-4ecb-a771-3f57503a7cd8\") " Oct 31 00:16:14.193317 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:14.193289 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/85098bb1-0c3e-4ecb-a771-3f57503a7cd8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85098bb1-0c3e-4ecb-a771-3f57503a7cd8-istio-podinfo\") pod \"85098bb1-0c3e-4ecb-a771-3f57503a7cd8\" (UID: \"85098bb1-0c3e-4ecb-a771-3f57503a7cd8\") " Oct 31 00:16:14.193317 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:14.193299 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85098bb1-0c3e-4ecb-a771-3f57503a7cd8-credential-socket" pod "85098bb1-0c3e-4ecb-a771-3f57503a7cd8" (UID: "85098bb1-0c3e-4ecb-a771-3f57503a7cd8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:16:14.193563 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:14.193366 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85098bb1-0c3e-4ecb-a771-3f57503a7cd8-kube-api-access-drxp5\" (UniqueName: \"kubernetes.io/projected/85098bb1-0c3e-4ecb-a771-3f57503a7cd8-kube-api-access-drxp5\") pod \"85098bb1-0c3e-4ecb-a771-3f57503a7cd8\" (UID: \"85098bb1-0c3e-4ecb-a771-3f57503a7cd8\") " Oct 31 00:16:14.193563 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:14.193398 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85098bb1-0c3e-4ecb-a771-3f57503a7cd8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85098bb1-0c3e-4ecb-a771-3f57503a7cd8-istio-data\") pod \"85098bb1-0c3e-4ecb-a771-3f57503a7cd8\" (UID: \"85098bb1-0c3e-4ecb-a771-3f57503a7cd8\") " Oct 31 00:16:14.193563 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:14.193440 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85098bb1-0c3e-4ecb-a771-3f57503a7cd8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85098bb1-0c3e-4ecb-a771-3f57503a7cd8-workload-certs\") pod \"85098bb1-0c3e-4ecb-a771-3f57503a7cd8\" (UID: \"85098bb1-0c3e-4ecb-a771-3f57503a7cd8\") " Oct 31 00:16:14.193743 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:14.193565 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85098bb1-0c3e-4ecb-a771-3f57503a7cd8-workload-socket" pod "85098bb1-0c3e-4ecb-a771-3f57503a7cd8" (UID: "85098bb1-0c3e-4ecb-a771-3f57503a7cd8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:16:14.193743 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:14.193482 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85098bb1-0c3e-4ecb-a771-3f57503a7cd8-istiod-ca-cert" pod "85098bb1-0c3e-4ecb-a771-3f57503a7cd8" (UID: "85098bb1-0c3e-4ecb-a771-3f57503a7cd8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:16:14.194122 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:14.194092 4927 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85098bb1-0c3e-4ecb-a771-3f57503a7cd8-credential-socket\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:16:14.194231 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:14.194125 4927 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85098bb1-0c3e-4ecb-a771-3f57503a7cd8-istiod-ca-cert\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:16:14.194231 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:14.194139 4927 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85098bb1-0c3e-4ecb-a771-3f57503a7cd8-workload-socket\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:16:14.194447 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:14.194424 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85098bb1-0c3e-4ecb-a771-3f57503a7cd8-workload-certs" pod "85098bb1-0c3e-4ecb-a771-3f57503a7cd8" (UID: "85098bb1-0c3e-4ecb-a771-3f57503a7cd8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:16:14.194518 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:14.194444 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85098bb1-0c3e-4ecb-a771-3f57503a7cd8-istio-data" pod "85098bb1-0c3e-4ecb-a771-3f57503a7cd8" (UID: "85098bb1-0c3e-4ecb-a771-3f57503a7cd8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:16:14.196224 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:14.196170 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85098bb1-0c3e-4ecb-a771-3f57503a7cd8-istio-token" pod "85098bb1-0c3e-4ecb-a771-3f57503a7cd8" (UID: "85098bb1-0c3e-4ecb-a771-3f57503a7cd8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:16:14.196224 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:14.196213 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85098bb1-0c3e-4ecb-a771-3f57503a7cd8-istio-envoy" pod "85098bb1-0c3e-4ecb-a771-3f57503a7cd8" (UID: "85098bb1-0c3e-4ecb-a771-3f57503a7cd8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:16:14.196441 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:14.196252 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/85098bb1-0c3e-4ecb-a771-3f57503a7cd8-istio-podinfo" pod "85098bb1-0c3e-4ecb-a771-3f57503a7cd8" (UID: "85098bb1-0c3e-4ecb-a771-3f57503a7cd8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:16:14.196441 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:14.196405 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85098bb1-0c3e-4ecb-a771-3f57503a7cd8-kube-api-access-drxp5" pod "85098bb1-0c3e-4ecb-a771-3f57503a7cd8" (UID: "85098bb1-0c3e-4ecb-a771-3f57503a7cd8"). InnerVolumeSpecName "kube-api-access-drxp5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:16:14.295971 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:14.295931 4927 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85098bb1-0c3e-4ecb-a771-3f57503a7cd8-istio-podinfo\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:16:14.295971 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:14.295963 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-drxp5\" (UniqueName: \"kubernetes.io/projected/85098bb1-0c3e-4ecb-a771-3f57503a7cd8-kube-api-access-drxp5\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:16:14.296250 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:14.295979 4927 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85098bb1-0c3e-4ecb-a771-3f57503a7cd8-istio-data\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:16:14.296250 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:14.295993 4927 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85098bb1-0c3e-4ecb-a771-3f57503a7cd8-workload-certs\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:16:14.296250 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:14.296005 4927 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85098bb1-0c3e-4ecb-a771-3f57503a7cd8-istio-envoy\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:16:14.296250 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:14.296017 4927 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85098bb1-0c3e-4ecb-a771-3f57503a7cd8-istio-token\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:16:14.675972 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:14.675944 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c9c21e-68d69b8994-hnwg4" Oct 31 00:16:14.723108 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:14.723073 4927 generic.go:356] "Generic (PLEG): container finished" podID="85098bb1-0c3e-4ecb-a771-3f57503a7cd8" containerID="efb5ceaf3b35dc1ac4f015f54c759b4866359273451e2abac9bdbfd626bbca2a" exitCode=0 Oct 31 00:16:14.723301 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:14.723115 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-54ad8c-openshift-default-575c545fdf-gvfbr" event={"ID":"85098bb1-0c3e-4ecb-a771-3f57503a7cd8","Type":"ContainerDied","Data":"efb5ceaf3b35dc1ac4f015f54c759b4866359273451e2abac9bdbfd626bbca2a"} Oct 31 00:16:14.723301 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:14.723130 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-54ad8c-openshift-default-575c545fdf-gvfbr" Oct 31 00:16:14.723301 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:14.723157 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-54ad8c-openshift-default-575c545fdf-gvfbr" event={"ID":"85098bb1-0c3e-4ecb-a771-3f57503a7cd8","Type":"ContainerDied","Data":"0a4eba3644d482523efed24cb40a74b3ac226d24591a0d958a79e2ce7513b4ab"} Oct 31 00:16:14.723301 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:14.723179 4927 scope.go:122] "RemoveContainer" containerID="efb5ceaf3b35dc1ac4f015f54c759b4866359273451e2abac9bdbfd626bbca2a" Oct 31 00:16:14.737726 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:14.737701 4927 scope.go:122] "RemoveContainer" containerID="7b5c45f7b944ca39203a0d1ceca2181c2fe8985765f8f818ff71500e3dbcfc04" Oct 31 00:16:14.751012 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:14.750982 4927 scope.go:122] "RemoveContainer" containerID="efb5ceaf3b35dc1ac4f015f54c759b4866359273451e2abac9bdbfd626bbca2a" Oct 31 00:16:14.751319 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:16:14.751297 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"efb5ceaf3b35dc1ac4f015f54c759b4866359273451e2abac9bdbfd626bbca2a\": container with ID starting with efb5ceaf3b35dc1ac4f015f54c759b4866359273451e2abac9bdbfd626bbca2a not found: ID does not exist" containerID="efb5ceaf3b35dc1ac4f015f54c759b4866359273451e2abac9bdbfd626bbca2a" Oct 31 00:16:14.751435 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:14.751326 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"efb5ceaf3b35dc1ac4f015f54c759b4866359273451e2abac9bdbfd626bbca2a"} err="failed to get container status \"efb5ceaf3b35dc1ac4f015f54c759b4866359273451e2abac9bdbfd626bbca2a\": rpc error: code = NotFound desc = could not find container \"efb5ceaf3b35dc1ac4f015f54c759b4866359273451e2abac9bdbfd626bbca2a\": container with ID starting with efb5ceaf3b35dc1ac4f015f54c759b4866359273451e2abac9bdbfd626bbca2a not found: ID does not exist" Oct 31 00:16:14.751435 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:14.751360 4927 scope.go:122] "RemoveContainer" containerID="7b5c45f7b944ca39203a0d1ceca2181c2fe8985765f8f818ff71500e3dbcfc04" Oct 31 00:16:14.751687 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:16:14.751662 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b5c45f7b944ca39203a0d1ceca2181c2fe8985765f8f818ff71500e3dbcfc04\": container with ID starting with 7b5c45f7b944ca39203a0d1ceca2181c2fe8985765f8f818ff71500e3dbcfc04 not found: ID does not exist" containerID="7b5c45f7b944ca39203a0d1ceca2181c2fe8985765f8f818ff71500e3dbcfc04" Oct 31 00:16:14.751740 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:14.751697 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b5c45f7b944ca39203a0d1ceca2181c2fe8985765f8f818ff71500e3dbcfc04"} err="failed to get container status \"7b5c45f7b944ca39203a0d1ceca2181c2fe8985765f8f818ff71500e3dbcfc04\": rpc error: code = NotFound desc = could not find container \"7b5c45f7b944ca39203a0d1ceca2181c2fe8985765f8f818ff71500e3dbcfc04\": container with ID starting with 7b5c45f7b944ca39203a0d1ceca2181c2fe8985765f8f818ff71500e3dbcfc04 not found: ID does not exist" Oct 31 00:16:14.927672 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:14.927575 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-54ad8c-openshift-default-575c545fdf-gvfbr"] Oct 31 00:16:14.934704 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:14.934670 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-54ad8c-openshift-default-575c545fdf-gvfbr"] Oct 31 00:16:15.088631 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:15.088600 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ba0721f-a4d2-49f6-b161-6d6f0f8ae4c2" path="/var/lib/kubelet/pods/6ba0721f-a4d2-49f6-b161-6d6f0f8ae4c2/volumes" Oct 31 00:16:15.089125 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:15.089112 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8328651d-33b5-4160-8f99-9d6de87ba69f" path="/var/lib/kubelet/pods/8328651d-33b5-4160-8f99-9d6de87ba69f/volumes" Oct 31 00:16:15.089611 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:15.089596 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="85098bb1-0c3e-4ecb-a771-3f57503a7cd8" path="/var/lib/kubelet/pods/85098bb1-0c3e-4ecb-a771-3f57503a7cd8/volumes" Oct 31 00:16:15.090395 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:15.090379 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="daac49a1-fcf0-4772-b555-5f43b01f8c3f" path="/var/lib/kubelet/pods/daac49a1-fcf0-4772-b555-5f43b01f8c3f/volumes" Oct 31 00:16:15.820546 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:15.820511 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:16:19.339754 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:19.339717 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-q6whh"] Oct 31 00:16:19.340527 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:19.340510 4927 cpu_manager.go:414] "containerMap: removing container" podUID="8328651d-33b5-4160-8f99-9d6de87ba69f" containerName="istio-proxy" Oct 31 00:16:19.340597 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:19.340529 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8328651d-33b5-4160-8f99-9d6de87ba69f" containerName="istio-proxy" Oct 31 00:16:19.340597 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:19.340558 4927 cpu_manager.go:414] "containerMap: removing container" podUID="528cb403-6318-45aa-afac-39095d3a8731" containerName="cleanup" Oct 31 00:16:19.340597 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:19.340563 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="528cb403-6318-45aa-afac-39095d3a8731" containerName="cleanup" Oct 31 00:16:19.340597 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:19.340578 4927 cpu_manager.go:414] "containerMap: removing container" podUID="85098bb1-0c3e-4ecb-a771-3f57503a7cd8" containerName="istio-proxy" Oct 31 00:16:19.340597 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:19.340583 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85098bb1-0c3e-4ecb-a771-3f57503a7cd8" containerName="istio-proxy" Oct 31 00:16:19.340597 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:19.340591 4927 cpu_manager.go:414] "containerMap: removing container" podUID="6ba0721f-a4d2-49f6-b161-6d6f0f8ae4c2" containerName="istio-proxy" Oct 31 00:16:19.340597 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:19.340596 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ba0721f-a4d2-49f6-b161-6d6f0f8ae4c2" containerName="istio-proxy" Oct 31 00:16:19.340807 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:19.340610 4927 cpu_manager.go:414] "containerMap: removing container" podUID="daac49a1-fcf0-4772-b555-5f43b01f8c3f" containerName="istio-proxy" Oct 31 00:16:19.340807 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:19.340614 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="daac49a1-fcf0-4772-b555-5f43b01f8c3f" containerName="istio-proxy" Oct 31 00:16:19.340807 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:19.340621 4927 cpu_manager.go:414] "containerMap: removing container" podUID="85098bb1-0c3e-4ecb-a771-3f57503a7cd8" containerName="istio-proxy" Oct 31 00:16:19.340807 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:19.340625 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85098bb1-0c3e-4ecb-a771-3f57503a7cd8" containerName="istio-proxy" Oct 31 00:16:19.340807 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:19.340726 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="daac49a1-fcf0-4772-b555-5f43b01f8c3f" containerName="istio-proxy" Oct 31 00:16:19.340807 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:19.340733 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="85098bb1-0c3e-4ecb-a771-3f57503a7cd8" containerName="istio-proxy" Oct 31 00:16:19.340807 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:19.340744 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="8328651d-33b5-4160-8f99-9d6de87ba69f" containerName="istio-proxy" Oct 31 00:16:19.340807 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:19.340750 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="5499c02e-e482-4329-b2b6-265ff423c83c" containerName="cleanup" Oct 31 00:16:19.340807 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:19.340761 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="528cb403-6318-45aa-afac-39095d3a8731" containerName="cleanup" Oct 31 00:16:19.340807 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:19.340771 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ba0721f-a4d2-49f6-b161-6d6f0f8ae4c2" containerName="istio-proxy" Oct 31 00:16:19.340807 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:19.340777 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a2039e2-43bf-4f84-9c06-f7ba01c95c5d" containerName="cleanup" Oct 31 00:16:19.346886 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:19.346861 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-q6whh" Oct 31 00:16:19.376161 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:19.376127 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1a9527e-1f11-49b9-a3bf-dae62908d3a9-istio-podinfo\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-q6whh\" (UID: \"f1a9527e-1f11-49b9-a3bf-dae62908d3a9\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-q6whh" Oct 31 00:16:19.376353 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:19.376186 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1a9527e-1f11-49b9-a3bf-dae62908d3a9-workload-socket\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-q6whh\" (UID: \"f1a9527e-1f11-49b9-a3bf-dae62908d3a9\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-q6whh" Oct 31 00:16:19.376353 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:19.376289 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1a9527e-1f11-49b9-a3bf-dae62908d3a9-istio-data\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-q6whh\" (UID: \"f1a9527e-1f11-49b9-a3bf-dae62908d3a9\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-q6whh" Oct 31 00:16:19.376353 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:19.376321 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97dl5\" (UniqueName: \"kubernetes.io/projected/f1a9527e-1f11-49b9-a3bf-dae62908d3a9-kube-api-access-97dl5\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-q6whh\" (UID: \"f1a9527e-1f11-49b9-a3bf-dae62908d3a9\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-q6whh" Oct 31 00:16:19.376530 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:19.376511 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1a9527e-1f11-49b9-a3bf-dae62908d3a9-istiod-ca-cert\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-q6whh\" (UID: \"f1a9527e-1f11-49b9-a3bf-dae62908d3a9\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-q6whh" Oct 31 00:16:19.376578 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:19.376548 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1a9527e-1f11-49b9-a3bf-dae62908d3a9-istio-token\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-q6whh\" (UID: \"f1a9527e-1f11-49b9-a3bf-dae62908d3a9\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-q6whh" Oct 31 00:16:19.376724 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:19.376705 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1a9527e-1f11-49b9-a3bf-dae62908d3a9-credential-socket\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-q6whh\" (UID: \"f1a9527e-1f11-49b9-a3bf-dae62908d3a9\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-q6whh" Oct 31 00:16:19.376874 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:19.376855 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1a9527e-1f11-49b9-a3bf-dae62908d3a9-workload-certs\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-q6whh\" (UID: \"f1a9527e-1f11-49b9-a3bf-dae62908d3a9\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-q6whh" Oct 31 00:16:19.376959 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:19.376911 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1a9527e-1f11-49b9-a3bf-dae62908d3a9-istio-envoy\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-q6whh\" (UID: \"f1a9527e-1f11-49b9-a3bf-dae62908d3a9\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-q6whh" Oct 31 00:16:19.390536 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:19.390501 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-c55xx"] Oct 31 00:16:19.391448 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:19.391428 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="85098bb1-0c3e-4ecb-a771-3f57503a7cd8" containerName="istio-proxy" Oct 31 00:16:19.395491 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:19.395459 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-c55xx" Oct 31 00:16:19.444182 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:19.444142 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-q6whh"] Oct 31 00:16:19.478236 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:19.478204 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40dd0190-5043-4730-8830-2e8aad25cade-workload-certs\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-c55xx\" (UID: \"40dd0190-5043-4730-8830-2e8aad25cade\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-c55xx" Oct 31 00:16:19.478424 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:19.478245 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1a9527e-1f11-49b9-a3bf-dae62908d3a9-workload-certs\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-q6whh\" (UID: \"f1a9527e-1f11-49b9-a3bf-dae62908d3a9\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-q6whh" Oct 31 00:16:19.478424 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:19.478266 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40dd0190-5043-4730-8830-2e8aad25cade-istio-data\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-c55xx\" (UID: \"40dd0190-5043-4730-8830-2e8aad25cade\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-c55xx" Oct 31 00:16:19.478424 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:19.478361 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1a9527e-1f11-49b9-a3bf-dae62908d3a9-istio-envoy\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-q6whh\" (UID: \"f1a9527e-1f11-49b9-a3bf-dae62908d3a9\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-q6whh" Oct 31 00:16:19.478424 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:19.478417 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40dd0190-5043-4730-8830-2e8aad25cade-istio-podinfo\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-c55xx\" (UID: \"40dd0190-5043-4730-8830-2e8aad25cade\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-c55xx" Oct 31 00:16:19.478683 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:19.478452 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40dd0190-5043-4730-8830-2e8aad25cade-istio-envoy\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-c55xx\" (UID: \"40dd0190-5043-4730-8830-2e8aad25cade\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-c55xx" Oct 31 00:16:19.478683 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:19.478497 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40dd0190-5043-4730-8830-2e8aad25cade-istio-token\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-c55xx\" (UID: \"40dd0190-5043-4730-8830-2e8aad25cade\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-c55xx" Oct 31 00:16:19.478683 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:19.478540 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1a9527e-1f11-49b9-a3bf-dae62908d3a9-istio-podinfo\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-q6whh\" (UID: \"f1a9527e-1f11-49b9-a3bf-dae62908d3a9\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-q6whh" Oct 31 00:16:19.478683 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:19.478581 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1a9527e-1f11-49b9-a3bf-dae62908d3a9-workload-socket\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-q6whh\" (UID: \"f1a9527e-1f11-49b9-a3bf-dae62908d3a9\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-q6whh" Oct 31 00:16:19.478683 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:19.478623 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ttk8x\" (UniqueName: \"kubernetes.io/projected/40dd0190-5043-4730-8830-2e8aad25cade-kube-api-access-ttk8x\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-c55xx\" (UID: \"40dd0190-5043-4730-8830-2e8aad25cade\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-c55xx" Oct 31 00:16:19.478683 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:19.478664 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1a9527e-1f11-49b9-a3bf-dae62908d3a9-istio-data\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-q6whh\" (UID: \"f1a9527e-1f11-49b9-a3bf-dae62908d3a9\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-q6whh" Oct 31 00:16:19.479013 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:19.478693 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97dl5\" (UniqueName: \"kubernetes.io/projected/f1a9527e-1f11-49b9-a3bf-dae62908d3a9-kube-api-access-97dl5\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-q6whh\" (UID: \"f1a9527e-1f11-49b9-a3bf-dae62908d3a9\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-q6whh" Oct 31 00:16:19.479013 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:19.478774 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40dd0190-5043-4730-8830-2e8aad25cade-workload-socket\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-c55xx\" (UID: \"40dd0190-5043-4730-8830-2e8aad25cade\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-c55xx" Oct 31 00:16:19.479013 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:19.478820 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1a9527e-1f11-49b9-a3bf-dae62908d3a9-istiod-ca-cert\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-q6whh\" (UID: \"f1a9527e-1f11-49b9-a3bf-dae62908d3a9\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-q6whh" Oct 31 00:16:19.479013 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:19.478860 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1a9527e-1f11-49b9-a3bf-dae62908d3a9-istio-token\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-q6whh\" (UID: \"f1a9527e-1f11-49b9-a3bf-dae62908d3a9\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-q6whh" Oct 31 00:16:19.479013 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:19.478970 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40dd0190-5043-4730-8830-2e8aad25cade-istiod-ca-cert\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-c55xx\" (UID: \"40dd0190-5043-4730-8830-2e8aad25cade\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-c55xx" Oct 31 00:16:19.479279 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:19.479028 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1a9527e-1f11-49b9-a3bf-dae62908d3a9-credential-socket\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-q6whh\" (UID: \"f1a9527e-1f11-49b9-a3bf-dae62908d3a9\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-q6whh" Oct 31 00:16:19.479279 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:19.479086 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40dd0190-5043-4730-8830-2e8aad25cade-credential-socket\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-c55xx\" (UID: \"40dd0190-5043-4730-8830-2e8aad25cade\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-c55xx" Oct 31 00:16:19.481810 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:19.478822 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1a9527e-1f11-49b9-a3bf-dae62908d3a9-workload-certs\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-q6whh\" (UID: \"f1a9527e-1f11-49b9-a3bf-dae62908d3a9\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-q6whh" Oct 31 00:16:19.481810 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:19.480624 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1a9527e-1f11-49b9-a3bf-dae62908d3a9-istiod-ca-cert\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-q6whh\" (UID: \"f1a9527e-1f11-49b9-a3bf-dae62908d3a9\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-q6whh" Oct 31 00:16:19.481810 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:19.480788 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1a9527e-1f11-49b9-a3bf-dae62908d3a9-credential-socket\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-q6whh\" (UID: \"f1a9527e-1f11-49b9-a3bf-dae62908d3a9\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-q6whh" Oct 31 00:16:19.482081 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:19.481914 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1a9527e-1f11-49b9-a3bf-dae62908d3a9-workload-socket\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-q6whh\" (UID: \"f1a9527e-1f11-49b9-a3bf-dae62908d3a9\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-q6whh" Oct 31 00:16:19.482245 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:19.482183 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1a9527e-1f11-49b9-a3bf-dae62908d3a9-istio-data\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-q6whh\" (UID: \"f1a9527e-1f11-49b9-a3bf-dae62908d3a9\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-q6whh" Oct 31 00:16:19.482764 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:19.482740 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1a9527e-1f11-49b9-a3bf-dae62908d3a9-istio-envoy\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-q6whh\" (UID: \"f1a9527e-1f11-49b9-a3bf-dae62908d3a9\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-q6whh" Oct 31 00:16:19.482991 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:19.482969 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1a9527e-1f11-49b9-a3bf-dae62908d3a9-istio-podinfo\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-q6whh\" (UID: \"f1a9527e-1f11-49b9-a3bf-dae62908d3a9\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-q6whh" Oct 31 00:16:19.487374 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:19.487267 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-a251fdfb-openshift-default-dockercfg-k8cql\"" Oct 31 00:16:19.495446 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:19.495417 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-c55xx"] Oct 31 00:16:19.551872 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:19.551750 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1a9527e-1f11-49b9-a3bf-dae62908d3a9-istio-token\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-q6whh\" (UID: \"f1a9527e-1f11-49b9-a3bf-dae62908d3a9\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-q6whh" Oct 31 00:16:19.552783 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:19.552745 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-97dl5\" (UniqueName: \"kubernetes.io/projected/f1a9527e-1f11-49b9-a3bf-dae62908d3a9-kube-api-access-97dl5\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-q6whh\" (UID: \"f1a9527e-1f11-49b9-a3bf-dae62908d3a9\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-q6whh" Oct 31 00:16:19.581185 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:19.581123 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40dd0190-5043-4730-8830-2e8aad25cade-istiod-ca-cert\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-c55xx\" (UID: \"40dd0190-5043-4730-8830-2e8aad25cade\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-c55xx" Oct 31 00:16:19.581455 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:19.581225 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40dd0190-5043-4730-8830-2e8aad25cade-credential-socket\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-c55xx\" (UID: \"40dd0190-5043-4730-8830-2e8aad25cade\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-c55xx" Oct 31 00:16:19.581455 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:19.581286 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40dd0190-5043-4730-8830-2e8aad25cade-workload-certs\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-c55xx\" (UID: \"40dd0190-5043-4730-8830-2e8aad25cade\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-c55xx" Oct 31 00:16:19.581455 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:19.581326 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40dd0190-5043-4730-8830-2e8aad25cade-istio-data\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-c55xx\" (UID: \"40dd0190-5043-4730-8830-2e8aad25cade\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-c55xx" Oct 31 00:16:19.581455 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:19.581399 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40dd0190-5043-4730-8830-2e8aad25cade-istio-podinfo\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-c55xx\" (UID: \"40dd0190-5043-4730-8830-2e8aad25cade\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-c55xx" Oct 31 00:16:19.581455 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:19.581428 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40dd0190-5043-4730-8830-2e8aad25cade-istio-envoy\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-c55xx\" (UID: \"40dd0190-5043-4730-8830-2e8aad25cade\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-c55xx" Oct 31 00:16:19.581784 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:19.581472 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40dd0190-5043-4730-8830-2e8aad25cade-istio-token\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-c55xx\" (UID: \"40dd0190-5043-4730-8830-2e8aad25cade\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-c55xx" Oct 31 00:16:19.581784 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:19.581536 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttk8x\" (UniqueName: \"kubernetes.io/projected/40dd0190-5043-4730-8830-2e8aad25cade-kube-api-access-ttk8x\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-c55xx\" (UID: \"40dd0190-5043-4730-8830-2e8aad25cade\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-c55xx" Oct 31 00:16:19.581784 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:19.581632 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40dd0190-5043-4730-8830-2e8aad25cade-workload-socket\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-c55xx\" (UID: \"40dd0190-5043-4730-8830-2e8aad25cade\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-c55xx" Oct 31 00:16:19.582486 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:19.582140 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40dd0190-5043-4730-8830-2e8aad25cade-workload-socket\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-c55xx\" (UID: \"40dd0190-5043-4730-8830-2e8aad25cade\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-c55xx" Oct 31 00:16:19.583191 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:19.582851 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40dd0190-5043-4730-8830-2e8aad25cade-istiod-ca-cert\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-c55xx\" (UID: \"40dd0190-5043-4730-8830-2e8aad25cade\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-c55xx" Oct 31 00:16:19.583191 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:19.582913 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40dd0190-5043-4730-8830-2e8aad25cade-istio-data\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-c55xx\" (UID: \"40dd0190-5043-4730-8830-2e8aad25cade\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-c55xx" Oct 31 00:16:19.583191 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:19.582969 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40dd0190-5043-4730-8830-2e8aad25cade-workload-certs\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-c55xx\" (UID: \"40dd0190-5043-4730-8830-2e8aad25cade\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-c55xx" Oct 31 00:16:19.583191 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:19.583156 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40dd0190-5043-4730-8830-2e8aad25cade-credential-socket\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-c55xx\" (UID: \"40dd0190-5043-4730-8830-2e8aad25cade\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-c55xx" Oct 31 00:16:19.585793 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:19.585765 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40dd0190-5043-4730-8830-2e8aad25cade-istio-envoy\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-c55xx\" (UID: \"40dd0190-5043-4730-8830-2e8aad25cade\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-c55xx" Oct 31 00:16:19.586132 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:19.586110 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40dd0190-5043-4730-8830-2e8aad25cade-istio-podinfo\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-c55xx\" (UID: \"40dd0190-5043-4730-8830-2e8aad25cade\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-c55xx" Oct 31 00:16:19.647383 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:19.647295 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40dd0190-5043-4730-8830-2e8aad25cade-istio-token\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-c55xx\" (UID: \"40dd0190-5043-4730-8830-2e8aad25cade\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-c55xx" Oct 31 00:16:19.647631 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:19.647602 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttk8x\" (UniqueName: \"kubernetes.io/projected/40dd0190-5043-4730-8830-2e8aad25cade-kube-api-access-ttk8x\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-c55xx\" (UID: \"40dd0190-5043-4730-8830-2e8aad25cade\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-c55xx" Oct 31 00:16:19.718933 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:19.718879 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-q6whh" Oct 31 00:16:19.746602 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:19.726509 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-c55xx" Oct 31 00:16:19.875965 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:16:19.875897 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sqv69" containerName="extractor" Oct 31 00:16:20.314518 ip-10-0-141-180 kubenswrapper[4927]: W1031 00:16:20.314480 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40dd0190_5043_4730_8830_2e8aad25cade.slice/crio-6735e23360b5646866d6fce27b567c6108dbe9eff234fffa3d72f50a50bf8a58 WatchSource:0}: Error finding container 6735e23360b5646866d6fce27b567c6108dbe9eff234fffa3d72f50a50bf8a58: Status 404 returned error can't find the container with id 6735e23360b5646866d6fce27b567c6108dbe9eff234fffa3d72f50a50bf8a58 Oct 31 00:16:20.314986 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:20.314965 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-c55xx"] Oct 31 00:16:20.317385 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:20.317343 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:16:20.317486 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:20.317421 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:16:20.317486 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:20.317459 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:16:20.334830 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:20.334807 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-q6whh"] Oct 31 00:16:20.354831 ip-10-0-141-180 kubenswrapper[4927]: W1031 00:16:20.354802 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1a9527e_1f11_49b9_a3bf_dae62908d3a9.slice/crio-e066abb39050adca65f03b6be6f5d86baf3dd2998461838bc3cbd2c46c16df5c WatchSource:0}: Error finding container e066abb39050adca65f03b6be6f5d86baf3dd2998461838bc3cbd2c46c16df5c: Status 404 returned error can't find the container with id e066abb39050adca65f03b6be6f5d86baf3dd2998461838bc3cbd2c46c16df5c Oct 31 00:16:20.358382 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:20.358322 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:16:20.358468 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:20.358439 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:16:20.358518 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:20.358489 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:16:20.592786 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:20.592699 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:16:20.846415 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:20.839490 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-c55xx" event={"ID":"40dd0190-5043-4730-8830-2e8aad25cade","Type":"ContainerStarted","Data":"6946f32d9ffb28294eff3233bc6fbd3298655fff11993cb9a3bfcea3307ea78e"} Oct 31 00:16:20.846415 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:20.839603 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-c55xx" event={"ID":"40dd0190-5043-4730-8830-2e8aad25cade","Type":"ContainerStarted","Data":"6735e23360b5646866d6fce27b567c6108dbe9eff234fffa3d72f50a50bf8a58"} Oct 31 00:16:20.846415 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:20.845210 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-q6whh" event={"ID":"f1a9527e-1f11-49b9-a3bf-dae62908d3a9","Type":"ContainerStarted","Data":"fb92d5c2411c710b2e44b4bb5e1228f647f9ded4358b64e748c2cb740913cca1"} Oct 31 00:16:20.846415 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:20.845267 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-q6whh" event={"ID":"f1a9527e-1f11-49b9-a3bf-dae62908d3a9","Type":"ContainerStarted","Data":"e066abb39050adca65f03b6be6f5d86baf3dd2998461838bc3cbd2c46c16df5c"} Oct 31 00:16:20.971074 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:20.971006 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-c55xx" podStartSLOduration=1.970986318 podStartE2EDuration="1.970986318s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:16:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:16:20.963236761 +0000 UTC m=+2093.999567601" watchObservedRunningTime="2026-10-31 00:16:20.970986318 +0000 UTC m=+2094.007317142" Oct 31 00:16:21.094351 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:21.093769 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-q6whh" podStartSLOduration=2.093755343 podStartE2EDuration="2.093755343s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:16:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:16:21.090956407 +0000 UTC m=+2094.127287250" watchObservedRunningTime="2026-10-31 00:16:21.093755343 +0000 UTC m=+2094.130086165" Oct 31 00:16:21.494358 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:21.470114 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:16:21.705466 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:21.704690 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:16:21.833450 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:21.832873 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-q6whh" Oct 31 00:16:21.833450 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:21.833021 4927 patch_prober.go:29] interesting pod/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-q6whh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused" start-of-body= Oct 31 00:16:21.833450 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:21.833079 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-q6whh" podUID="f1a9527e-1f11-49b9-a3bf-dae62908d3a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused" Oct 31 00:16:21.869127 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:21.869087 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-c55xx" Oct 31 00:16:21.869825 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:21.869799 4927 patch_prober.go:29] interesting pod/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-c55xx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused" start-of-body= Oct 31 00:16:21.870027 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:21.870002 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-c55xx" podUID="40dd0190-5043-4730-8830-2e8aad25cade" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused" Oct 31 00:16:22.731743 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:22.731703 4927 patch_prober.go:29] interesting pod/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-c55xx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused" start-of-body= Oct 31 00:16:22.732374 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:22.732313 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-c55xx" podUID="40dd0190-5043-4730-8830-2e8aad25cade" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused" Oct 31 00:16:22.733139 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:22.733115 4927 patch_prober.go:29] interesting pod/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-q6whh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused" start-of-body= Oct 31 00:16:22.733310 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:22.733288 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-q6whh" podUID="f1a9527e-1f11-49b9-a3bf-dae62908d3a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused" Oct 31 00:16:23.729949 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:23.729898 4927 patch_prober.go:29] interesting pod/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-c55xx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused" start-of-body= Oct 31 00:16:23.730253 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:23.730203 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-c55xx" podUID="40dd0190-5043-4730-8830-2e8aad25cade" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused" Oct 31 00:16:23.730910 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:23.730872 4927 patch_prober.go:29] interesting pod/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-q6whh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused" start-of-body= Oct 31 00:16:23.731083 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:23.731062 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-q6whh" podUID="f1a9527e-1f11-49b9-a3bf-dae62908d3a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused" Oct 31 00:16:24.729356 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:24.728934 4927 patch_prober.go:29] interesting pod/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-q6whh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused" start-of-body= Oct 31 00:16:24.729356 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:24.728999 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-q6whh" podUID="f1a9527e-1f11-49b9-a3bf-dae62908d3a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused" Oct 31 00:16:24.729970 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:24.729429 4927 patch_prober.go:29] interesting pod/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-c55xx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused" start-of-body= Oct 31 00:16:24.729970 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:24.729486 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-c55xx" podUID="40dd0190-5043-4730-8830-2e8aad25cade" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused" Oct 31 00:16:25.401356 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:25.400490 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:16:25.709374 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:25.707245 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:16:25.753357 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:25.752873 4927 patch_prober.go:29] interesting pod/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-q6whh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused" start-of-body= Oct 31 00:16:25.753357 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:25.752978 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-q6whh" podUID="f1a9527e-1f11-49b9-a3bf-dae62908d3a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused" Oct 31 00:16:25.753948 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:25.753519 4927 patch_prober.go:29] interesting pod/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-c55xx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused" start-of-body= Oct 31 00:16:25.753948 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:25.753582 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-c55xx" podUID="40dd0190-5043-4730-8830-2e8aad25cade" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused" Oct 31 00:16:25.899343 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:25.899290 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:16:26.731357 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:26.731164 4927 patch_prober.go:29] interesting pod/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-q6whh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused" start-of-body= Oct 31 00:16:26.731357 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:26.731235 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-q6whh" podUID="f1a9527e-1f11-49b9-a3bf-dae62908d3a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused" Oct 31 00:16:26.750357 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:26.731674 4927 patch_prober.go:29] interesting pod/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-c55xx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused" start-of-body= Oct 31 00:16:26.750357 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:26.731722 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-c55xx" podUID="40dd0190-5043-4730-8830-2e8aad25cade" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused" Oct 31 00:16:27.778856 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:27.778734 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-c55xx" Oct 31 00:16:27.780911 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:27.780687 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-c55xx" Oct 31 00:16:27.782313 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:27.782290 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-c55xx" Oct 31 00:16:27.841890 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:27.841858 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-q6whh" Oct 31 00:16:27.843130 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:27.843104 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-q6whh" Oct 31 00:16:27.844274 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:27.844238 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-q6whh" Oct 31 00:16:28.595559 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:28.595528 4927 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-pqt2c_6382dfb5-a363-47a2-bc79-0b96f7bec0c3/console-operator/3.log" Oct 31 00:16:29.596877 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:29.596842 4927 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-pqt2c_6382dfb5-a363-47a2-bc79-0b96f7bec0c3/console-operator/3.log" Oct 31 00:16:29.830034 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:29.829991 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:16:29.993435 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:29.993404 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:16:31.373491 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:31.373408 4927 generic.go:356] "Generic (PLEG): container finished" podID="d058674e-2706-41c4-8495-d7005ac1f562" containerID="7ff637e1e8bfb168ede519666b1271d01cd05dc174570cadd1fb02b573ac1d6d" exitCode=0 Oct 31 00:16:31.373901 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:31.373486 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-460d4e-openshift-default-575d5fbfb7-ftjgz" event={"ID":"d058674e-2706-41c4-8495-d7005ac1f562","Type":"ContainerDied","Data":"7ff637e1e8bfb168ede519666b1271d01cd05dc174570cadd1fb02b573ac1d6d"} Oct 31 00:16:31.374022 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:31.374005 4927 scope.go:122] "RemoveContainer" containerID="7ff637e1e8bfb168ede519666b1271d01cd05dc174570cadd1fb02b573ac1d6d" Oct 31 00:16:31.375932 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:31.375899 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:16:31.376034 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:31.375972 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:16:31.376034 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:31.376004 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:16:32.395921 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:32.395872 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-460d4e-openshift-default-575d5fbfb7-ftjgz" event={"ID":"d058674e-2706-41c4-8495-d7005ac1f562","Type":"ContainerStarted","Data":"b467bcb4497c1dc2f83567a7020a33b6ac5a1e9219577fb1c368f8866aa3ff14"} Oct 31 00:16:32.632355 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:32.631737 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-460d4e-openshift-default-575d5fbfb7-ftjgz" Oct 31 00:16:32.636353 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:32.635444 4927 patch_prober.go:29] interesting pod/e2e-inf-b-460d4e-openshift-default-575d5fbfb7-ftjgz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused" start-of-body= Oct 31 00:16:32.636353 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:32.635518 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-460d4e-openshift-default-575d5fbfb7-ftjgz" podUID="d058674e-2706-41c4-8495-d7005ac1f562" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused" Oct 31 00:16:32.696588 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:32.696556 4927 scope.go:122] "RemoveContainer" containerID="d082b3e8a811385fe51b2a4d6dffdb4b0020fa0afc5caa5bf30562be21ab7651" Oct 31 00:16:33.631752 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:33.631709 4927 patch_prober.go:29] interesting pod/e2e-inf-b-460d4e-openshift-default-575d5fbfb7-ftjgz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused" start-of-body= Oct 31 00:16:33.632903 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:33.632868 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-460d4e-openshift-default-575d5fbfb7-ftjgz" podUID="d058674e-2706-41c4-8495-d7005ac1f562" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused" Oct 31 00:16:34.503616 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:34.503581 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-460d4e-openshift-default-575d5fbfb7-ftjgz" Oct 31 00:16:34.549904 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:34.549859 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-fxvfs"] Oct 31 00:16:34.558849 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:34.558182 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-fxvfs" Oct 31 00:16:34.652725 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:34.652563 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:16:34.655290 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:34.655252 4927 patch_prober.go:29] interesting pod/e2e-inf-b-460d4e-openshift-default-575d5fbfb7-ftjgz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused" start-of-body= Oct 31 00:16:34.655524 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:34.655498 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-460d4e-openshift-default-575d5fbfb7-ftjgz" podUID="d058674e-2706-41c4-8495-d7005ac1f562" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused" Oct 31 00:16:34.660871 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:34.660846 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-fxvfs"] Oct 31 00:16:34.719397 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:34.694355 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/671d81ce-d6d7-4973-a112-ad795eaeac84-workload-socket\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-fxvfs\" (UID: \"671d81ce-d6d7-4973-a112-ad795eaeac84\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-fxvfs" Oct 31 00:16:34.719806 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:34.719772 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/671d81ce-d6d7-4973-a112-ad795eaeac84-istio-token\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-fxvfs\" (UID: \"671d81ce-d6d7-4973-a112-ad795eaeac84\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-fxvfs" Oct 31 00:16:34.720005 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:34.719984 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/671d81ce-d6d7-4973-a112-ad795eaeac84-workload-certs\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-fxvfs\" (UID: \"671d81ce-d6d7-4973-a112-ad795eaeac84\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-fxvfs" Oct 31 00:16:34.761588 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:34.720170 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/671d81ce-d6d7-4973-a112-ad795eaeac84-istio-podinfo\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-fxvfs\" (UID: \"671d81ce-d6d7-4973-a112-ad795eaeac84\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-fxvfs" Oct 31 00:16:34.781037 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:34.781001 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-krnpc\" (UniqueName: \"kubernetes.io/projected/671d81ce-d6d7-4973-a112-ad795eaeac84-kube-api-access-krnpc\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-fxvfs\" (UID: \"671d81ce-d6d7-4973-a112-ad795eaeac84\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-fxvfs" Oct 31 00:16:34.781466 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:34.781440 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/671d81ce-d6d7-4973-a112-ad795eaeac84-istio-data\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-fxvfs\" (UID: \"671d81ce-d6d7-4973-a112-ad795eaeac84\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-fxvfs" Oct 31 00:16:34.781638 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:34.781615 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/671d81ce-d6d7-4973-a112-ad795eaeac84-istiod-ca-cert\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-fxvfs\" (UID: \"671d81ce-d6d7-4973-a112-ad795eaeac84\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-fxvfs" Oct 31 00:16:34.781716 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:34.781667 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/671d81ce-d6d7-4973-a112-ad795eaeac84-istio-envoy\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-fxvfs\" (UID: \"671d81ce-d6d7-4973-a112-ad795eaeac84\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-fxvfs" Oct 31 00:16:34.781785 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:34.781768 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/671d81ce-d6d7-4973-a112-ad795eaeac84-credential-socket\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-fxvfs\" (UID: \"671d81ce-d6d7-4973-a112-ad795eaeac84\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-fxvfs" Oct 31 00:16:34.883269 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:34.883224 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/671d81ce-d6d7-4973-a112-ad795eaeac84-istio-data\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-fxvfs\" (UID: \"671d81ce-d6d7-4973-a112-ad795eaeac84\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-fxvfs" Oct 31 00:16:34.883501 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:34.883355 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/671d81ce-d6d7-4973-a112-ad795eaeac84-istiod-ca-cert\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-fxvfs\" (UID: \"671d81ce-d6d7-4973-a112-ad795eaeac84\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-fxvfs" Oct 31 00:16:34.883501 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:34.883394 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/671d81ce-d6d7-4973-a112-ad795eaeac84-istio-envoy\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-fxvfs\" (UID: \"671d81ce-d6d7-4973-a112-ad795eaeac84\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-fxvfs" Oct 31 00:16:34.883501 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:34.883460 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/671d81ce-d6d7-4973-a112-ad795eaeac84-credential-socket\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-fxvfs\" (UID: \"671d81ce-d6d7-4973-a112-ad795eaeac84\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-fxvfs" Oct 31 00:16:34.883685 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:34.883552 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/671d81ce-d6d7-4973-a112-ad795eaeac84-workload-socket\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-fxvfs\" (UID: \"671d81ce-d6d7-4973-a112-ad795eaeac84\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-fxvfs" Oct 31 00:16:34.883685 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:34.883579 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/671d81ce-d6d7-4973-a112-ad795eaeac84-istio-token\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-fxvfs\" (UID: \"671d81ce-d6d7-4973-a112-ad795eaeac84\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-fxvfs" Oct 31 00:16:34.883685 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:34.883585 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/671d81ce-d6d7-4973-a112-ad795eaeac84-istio-data\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-fxvfs\" (UID: \"671d81ce-d6d7-4973-a112-ad795eaeac84\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-fxvfs" Oct 31 00:16:34.883685 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:34.883612 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/671d81ce-d6d7-4973-a112-ad795eaeac84-workload-certs\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-fxvfs\" (UID: \"671d81ce-d6d7-4973-a112-ad795eaeac84\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-fxvfs" Oct 31 00:16:34.883907 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:34.883781 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/671d81ce-d6d7-4973-a112-ad795eaeac84-credential-socket\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-fxvfs\" (UID: \"671d81ce-d6d7-4973-a112-ad795eaeac84\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-fxvfs" Oct 31 00:16:34.883997 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:34.883973 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/671d81ce-d6d7-4973-a112-ad795eaeac84-workload-certs\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-fxvfs\" (UID: \"671d81ce-d6d7-4973-a112-ad795eaeac84\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-fxvfs" Oct 31 00:16:34.884071 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:34.884040 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/671d81ce-d6d7-4973-a112-ad795eaeac84-istio-podinfo\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-fxvfs\" (UID: \"671d81ce-d6d7-4973-a112-ad795eaeac84\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-fxvfs" Oct 31 00:16:34.884128 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:34.884067 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/671d81ce-d6d7-4973-a112-ad795eaeac84-workload-socket\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-fxvfs\" (UID: \"671d81ce-d6d7-4973-a112-ad795eaeac84\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-fxvfs" Oct 31 00:16:34.884181 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:34.884161 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krnpc\" (UniqueName: \"kubernetes.io/projected/671d81ce-d6d7-4973-a112-ad795eaeac84-kube-api-access-krnpc\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-fxvfs\" (UID: \"671d81ce-d6d7-4973-a112-ad795eaeac84\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-fxvfs" Oct 31 00:16:34.884290 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:34.884164 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/671d81ce-d6d7-4973-a112-ad795eaeac84-istiod-ca-cert\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-fxvfs\" (UID: \"671d81ce-d6d7-4973-a112-ad795eaeac84\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-fxvfs" Oct 31 00:16:34.885927 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:34.885903 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/671d81ce-d6d7-4973-a112-ad795eaeac84-istio-envoy\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-fxvfs\" (UID: \"671d81ce-d6d7-4973-a112-ad795eaeac84\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-fxvfs" Oct 31 00:16:34.887238 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:34.887219 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/671d81ce-d6d7-4973-a112-ad795eaeac84-istio-podinfo\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-fxvfs\" (UID: \"671d81ce-d6d7-4973-a112-ad795eaeac84\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-fxvfs" Oct 31 00:16:34.946679 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:34.946645 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/671d81ce-d6d7-4973-a112-ad795eaeac84-istio-token\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-fxvfs\" (UID: \"671d81ce-d6d7-4973-a112-ad795eaeac84\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-fxvfs" Oct 31 00:16:34.947087 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:34.947061 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-krnpc\" (UniqueName: \"kubernetes.io/projected/671d81ce-d6d7-4973-a112-ad795eaeac84-kube-api-access-krnpc\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-fxvfs\" (UID: \"671d81ce-d6d7-4973-a112-ad795eaeac84\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-fxvfs" Oct 31 00:16:35.178460 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:35.178434 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-fxvfs" Oct 31 00:16:35.635934 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:35.635429 4927 patch_prober.go:29] interesting pod/e2e-inf-b-460d4e-openshift-default-575d5fbfb7-ftjgz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused" start-of-body= Oct 31 00:16:35.635934 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:35.635490 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-460d4e-openshift-default-575d5fbfb7-ftjgz" podUID="d058674e-2706-41c4-8495-d7005ac1f562" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused" Oct 31 00:16:35.664471 ip-10-0-141-180 kubenswrapper[4927]: W1031 00:16:35.664421 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod671d81ce_d6d7_4973_a112_ad795eaeac84.slice/crio-77ffd3b888e61b41da625582632dc12049d0d4beff6eede577ba8d0d82f246ef WatchSource:0}: Error finding container 77ffd3b888e61b41da625582632dc12049d0d4beff6eede577ba8d0d82f246ef: Status 404 returned error can't find the container with id 77ffd3b888e61b41da625582632dc12049d0d4beff6eede577ba8d0d82f246ef Oct 31 00:16:35.664912 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:35.664523 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-fxvfs"] Oct 31 00:16:35.667728 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:35.667578 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:16:35.667728 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:35.667655 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:16:35.667728 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:35.667705 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:16:36.479948 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:36.479915 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-fxvfs" event={"ID":"671d81ce-d6d7-4973-a112-ad795eaeac84","Type":"ContainerStarted","Data":"dfb7b227af3f459480d783b2124743080f0c67f8b6813ababc0060db977bd188"} Oct 31 00:16:36.480211 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:36.479967 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-fxvfs" event={"ID":"671d81ce-d6d7-4973-a112-ad795eaeac84","Type":"ContainerStarted","Data":"77ffd3b888e61b41da625582632dc12049d0d4beff6eede577ba8d0d82f246ef"} Oct 31 00:16:36.606357 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:36.605661 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-fxvfs" podStartSLOduration=2.605637951 podStartE2EDuration="2.605637951s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:16:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:16:36.604177841 +0000 UTC m=+2109.640508663" watchObservedRunningTime="2026-10-31 00:16:36.605637951 +0000 UTC m=+2109.641968800" Oct 31 00:16:36.639354 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:36.636730 4927 patch_prober.go:29] interesting pod/e2e-inf-b-460d4e-openshift-default-575d5fbfb7-ftjgz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused" start-of-body= Oct 31 00:16:36.639354 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:36.636809 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-460d4e-openshift-default-575d5fbfb7-ftjgz" podUID="d058674e-2706-41c4-8495-d7005ac1f562" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused" Oct 31 00:16:37.179560 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:37.179522 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-fxvfs" Oct 31 00:16:37.180281 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:37.180252 4927 patch_prober.go:29] interesting pod/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-fxvfs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.228:15021/healthz/ready\": dial tcp 10.132.0.228:15021: connect: connection refused" start-of-body= Oct 31 00:16:37.180364 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:37.180307 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-fxvfs" podUID="671d81ce-d6d7-4973-a112-ad795eaeac84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.228:15021/healthz/ready\": dial tcp 10.132.0.228:15021: connect: connection refused" Oct 31 00:16:37.628868 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:37.628771 4927 patch_prober.go:29] interesting pod/e2e-inf-b-460d4e-openshift-default-575d5fbfb7-ftjgz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused" start-of-body= Oct 31 00:16:37.629132 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:37.629100 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-460d4e-openshift-default-575d5fbfb7-ftjgz" podUID="d058674e-2706-41c4-8495-d7005ac1f562" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused" Oct 31 00:16:38.179076 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:38.179037 4927 patch_prober.go:29] interesting pod/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-fxvfs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.228:15021/healthz/ready\": dial tcp 10.132.0.228:15021: connect: connection refused" start-of-body= Oct 31 00:16:38.179300 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:38.179100 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-fxvfs" podUID="671d81ce-d6d7-4973-a112-ad795eaeac84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.228:15021/healthz/ready\": dial tcp 10.132.0.228:15021: connect: connection refused" Oct 31 00:16:38.579759 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:38.579651 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-71b76a-7c9bf5b98c-sr6mg"] Oct 31 00:16:38.580326 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:38.579928 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-71b76a-7c9bf5b98c-sr6mg" podUID="b306ffa9-b393-4c4f-a00f-9f9722b5a97a" containerName="maas-api" containerID="cri-o://b4e32f3cba7d477bb4dac18bfa8606f1ca290d495a9612a7461f0f75837097d2" gracePeriod=30 Oct 31 00:16:38.667735 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:38.666983 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-460d4e-openshift-default-575d5fbfb7-ftjgz" Oct 31 00:16:38.671454 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:38.671290 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-460d4e-openshift-default-575d5fbfb7-ftjgz" Oct 31 00:16:38.691407 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:16:38.691307 4927 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb306ffa9_b393_4c4f_a00f_9f9722b5a97a.slice/crio-conmon-b4e32f3cba7d477bb4dac18bfa8606f1ca290d495a9612a7461f0f75837097d2.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb306ffa9_b393_4c4f_a00f_9f9722b5a97a.slice/crio-b4e32f3cba7d477bb4dac18bfa8606f1ca290d495a9612a7461f0f75837097d2.scope\": RecentStats: unable to find data in memory cache]" Oct 31 00:16:38.743923 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:38.743865 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:16:39.224358 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:39.197753 4927 patch_prober.go:29] interesting pod/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-fxvfs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.228:15021/healthz/ready\": dial tcp 10.132.0.228:15021: connect: connection refused" start-of-body= Oct 31 00:16:39.224358 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:39.197812 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-fxvfs" podUID="671d81ce-d6d7-4973-a112-ad795eaeac84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.228:15021/healthz/ready\": dial tcp 10.132.0.228:15021: connect: connection refused" Oct 31 00:16:39.349363 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:39.347950 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-71b76a-7c9bf5b98c-sr6mg" Oct 31 00:16:39.397512 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:39.397481 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b306ffa9-b393-4c4f-a00f-9f9722b5a97a-kube-api-access-pdslh\" (UniqueName: \"kubernetes.io/projected/b306ffa9-b393-4c4f-a00f-9f9722b5a97a-kube-api-access-pdslh\") pod \"b306ffa9-b393-4c4f-a00f-9f9722b5a97a\" (UID: \"b306ffa9-b393-4c4f-a00f-9f9722b5a97a\") " Oct 31 00:16:39.397713 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:39.397547 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b306ffa9-b393-4c4f-a00f-9f9722b5a97a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b306ffa9-b393-4c4f-a00f-9f9722b5a97a-maas-api-tls\") pod \"b306ffa9-b393-4c4f-a00f-9f9722b5a97a\" (UID: \"b306ffa9-b393-4c4f-a00f-9f9722b5a97a\") " Oct 31 00:16:39.397713 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:39.397592 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b306ffa9-b393-4c4f-a00f-9f9722b5a97a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b306ffa9-b393-4c4f-a00f-9f9722b5a97a-metrics-certs\") pod \"b306ffa9-b393-4c4f-a00f-9f9722b5a97a\" (UID: \"b306ffa9-b393-4c4f-a00f-9f9722b5a97a\") " Oct 31 00:16:39.397713 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:39.397649 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b306ffa9-b393-4c4f-a00f-9f9722b5a97a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b306ffa9-b393-4c4f-a00f-9f9722b5a97a-openshift-service-ca\") pod \"b306ffa9-b393-4c4f-a00f-9f9722b5a97a\" (UID: \"b306ffa9-b393-4c4f-a00f-9f9722b5a97a\") " Oct 31 00:16:39.398111 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:39.398079 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b306ffa9-b393-4c4f-a00f-9f9722b5a97a-openshift-service-ca" pod "b306ffa9-b393-4c4f-a00f-9f9722b5a97a" (UID: "b306ffa9-b393-4c4f-a00f-9f9722b5a97a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:16:39.399892 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:39.399859 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b306ffa9-b393-4c4f-a00f-9f9722b5a97a-metrics-certs" pod "b306ffa9-b393-4c4f-a00f-9f9722b5a97a" (UID: "b306ffa9-b393-4c4f-a00f-9f9722b5a97a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:16:39.399892 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:39.399883 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b306ffa9-b393-4c4f-a00f-9f9722b5a97a-maas-api-tls" pod "b306ffa9-b393-4c4f-a00f-9f9722b5a97a" (UID: "b306ffa9-b393-4c4f-a00f-9f9722b5a97a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:16:39.400075 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:39.399967 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b306ffa9-b393-4c4f-a00f-9f9722b5a97a-kube-api-access-pdslh" pod "b306ffa9-b393-4c4f-a00f-9f9722b5a97a" (UID: "b306ffa9-b393-4c4f-a00f-9f9722b5a97a"). InnerVolumeSpecName "kube-api-access-pdslh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:16:39.502358 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:39.502223 4927 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b306ffa9-b393-4c4f-a00f-9f9722b5a97a-maas-api-tls\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:16:39.502358 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:39.502258 4927 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b306ffa9-b393-4c4f-a00f-9f9722b5a97a-metrics-certs\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:16:39.502358 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:39.502274 4927 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b306ffa9-b393-4c4f-a00f-9f9722b5a97a-openshift-service-ca\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:16:39.502358 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:39.502289 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pdslh\" (UniqueName: \"kubernetes.io/projected/b306ffa9-b393-4c4f-a00f-9f9722b5a97a-kube-api-access-pdslh\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:16:39.804937 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:39.804897 4927 generic.go:356] "Generic (PLEG): container finished" podID="b306ffa9-b393-4c4f-a00f-9f9722b5a97a" containerID="b4e32f3cba7d477bb4dac18bfa8606f1ca290d495a9612a7461f0f75837097d2" exitCode=0 Oct 31 00:16:39.805468 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:39.804953 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-71b76a-7c9bf5b98c-sr6mg" Oct 31 00:16:39.805468 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:39.805063 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-71b76a-7c9bf5b98c-sr6mg" event={"ID":"b306ffa9-b393-4c4f-a00f-9f9722b5a97a","Type":"ContainerDied","Data":"b4e32f3cba7d477bb4dac18bfa8606f1ca290d495a9612a7461f0f75837097d2"} Oct 31 00:16:39.805468 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:39.805109 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-71b76a-7c9bf5b98c-sr6mg" event={"ID":"b306ffa9-b393-4c4f-a00f-9f9722b5a97a","Type":"ContainerDied","Data":"0bdde3030fbbf95e5ef25b1216e5bf49fe011d24c784da4342282494e7cc29a1"} Oct 31 00:16:39.805468 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:39.805138 4927 scope.go:122] "RemoveContainer" containerID="b4e32f3cba7d477bb4dac18bfa8606f1ca290d495a9612a7461f0f75837097d2" Oct 31 00:16:39.857417 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:39.857391 4927 scope.go:122] "RemoveContainer" containerID="b4e32f3cba7d477bb4dac18bfa8606f1ca290d495a9612a7461f0f75837097d2" Oct 31 00:16:39.857824 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:16:39.857798 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4e32f3cba7d477bb4dac18bfa8606f1ca290d495a9612a7461f0f75837097d2\": container with ID starting with b4e32f3cba7d477bb4dac18bfa8606f1ca290d495a9612a7461f0f75837097d2 not found: ID does not exist" containerID="b4e32f3cba7d477bb4dac18bfa8606f1ca290d495a9612a7461f0f75837097d2" Oct 31 00:16:39.857917 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:39.857841 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4e32f3cba7d477bb4dac18bfa8606f1ca290d495a9612a7461f0f75837097d2"} err="failed to get container status \"b4e32f3cba7d477bb4dac18bfa8606f1ca290d495a9612a7461f0f75837097d2\": rpc error: code = NotFound desc = could not find container \"b4e32f3cba7d477bb4dac18bfa8606f1ca290d495a9612a7461f0f75837097d2\": container with ID starting with b4e32f3cba7d477bb4dac18bfa8606f1ca290d495a9612a7461f0f75837097d2 not found: ID does not exist" Oct 31 00:16:39.982141 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:39.982106 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-71b76a-7c9bf5b98c-sr6mg"] Oct 31 00:16:39.989454 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:39.989417 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-71b76a-7c9bf5b98c-sr6mg"] Oct 31 00:16:40.190799 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:40.190759 4927 patch_prober.go:29] interesting pod/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-fxvfs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.228:15021/healthz/ready\": dial tcp 10.132.0.228:15021: connect: connection refused" start-of-body= Oct 31 00:16:40.191077 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:40.191049 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-fxvfs" podUID="671d81ce-d6d7-4973-a112-ad795eaeac84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.228:15021/healthz/ready\": dial tcp 10.132.0.228:15021: connect: connection refused" Oct 31 00:16:40.195881 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:40.195850 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:16:40.576045 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:40.575950 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:16:40.745522 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:40.745484 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:16:41.089083 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:41.089050 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b306ffa9-b393-4c4f-a00f-9f9722b5a97a" path="/var/lib/kubelet/pods/b306ffa9-b393-4c4f-a00f-9f9722b5a97a/volumes" Oct 31 00:16:41.174598 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:41.174549 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:16:41.179543 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:41.179511 4927 patch_prober.go:29] interesting pod/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-fxvfs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.228:15021/healthz/ready\": dial tcp 10.132.0.228:15021: connect: connection refused" start-of-body= Oct 31 00:16:41.179784 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:41.179765 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-fxvfs" podUID="671d81ce-d6d7-4973-a112-ad795eaeac84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.228:15021/healthz/ready\": dial tcp 10.132.0.228:15021: connect: connection refused" Oct 31 00:16:42.179570 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:42.179534 4927 patch_prober.go:29] interesting pod/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-fxvfs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.228:15021/healthz/ready\": dial tcp 10.132.0.228:15021: connect: connection refused" start-of-body= Oct 31 00:16:42.180061 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:42.179595 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-fxvfs" podUID="671d81ce-d6d7-4973-a112-ad795eaeac84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.228:15021/healthz/ready\": dial tcp 10.132.0.228:15021: connect: connection refused" Oct 31 00:16:43.183839 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:43.183807 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-fxvfs" Oct 31 00:16:43.184385 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:43.184361 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-fxvfs" Oct 31 00:16:43.185017 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:43.184995 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-fxvfs" Oct 31 00:16:45.471571 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:45.471536 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:16:47.372897 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:47.372855 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:16:47.529189 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:47.529145 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:16:47.763784 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:47.763750 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-71b76a-openshift-default-77db967c97-lh4ld"] Oct 31 00:16:47.763980 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:47.763959 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-71b76a-openshift-default-77db967c97-lh4ld" podUID="c51c586a-f2d8-48f9-8813-3b6e4f360e5e" containerName="istio-proxy" containerID="cri-o://2a76af001dfaa02e742381689a56aec54d0870d03c8513bf137af374832cd231" gracePeriod=30 Oct 31 00:16:47.766317 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:47.766289 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:16:48.604018 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:48.603982 4927 patch_prober.go:29] interesting pod/e2e-api-a-71b76a-openshift-default-77db967c97-lh4ld container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 00:16:48.604414 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:48.604051 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-71b76a-openshift-default-77db967c97-lh4ld" podUID="c51c586a-f2d8-48f9-8813-3b6e4f360e5e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:16:49.620144 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:49.620111 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-qmsfx"] Oct 31 00:16:49.620987 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:49.620968 4927 cpu_manager.go:414] "containerMap: removing container" podUID="b306ffa9-b393-4c4f-a00f-9f9722b5a97a" containerName="maas-api" Oct 31 00:16:49.621056 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:49.620989 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b306ffa9-b393-4c4f-a00f-9f9722b5a97a" containerName="maas-api" Oct 31 00:16:49.621119 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:49.621110 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="b306ffa9-b393-4c4f-a00f-9f9722b5a97a" containerName="maas-api" Oct 31 00:16:49.625798 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:49.625778 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-qmsfx" Oct 31 00:16:49.683599 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:16:49.683567 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sqv69" containerName="extractor" Oct 31 00:16:49.702734 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:49.702706 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-qmsfx"] Oct 31 00:16:49.777580 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:49.777547 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b7dab75-7510-49a6-ace3-f72da2adef72-istio-data\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-qmsfx\" (UID: \"4b7dab75-7510-49a6-ace3-f72da2adef72\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-qmsfx" Oct 31 00:16:49.777756 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:49.777616 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b7dab75-7510-49a6-ace3-f72da2adef72-istio-token\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-qmsfx\" (UID: \"4b7dab75-7510-49a6-ace3-f72da2adef72\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-qmsfx" Oct 31 00:16:49.777756 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:49.777707 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kg5lj\" (UniqueName: \"kubernetes.io/projected/4b7dab75-7510-49a6-ace3-f72da2adef72-kube-api-access-kg5lj\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-qmsfx\" (UID: \"4b7dab75-7510-49a6-ace3-f72da2adef72\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-qmsfx" Oct 31 00:16:49.777756 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:49.777750 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b7dab75-7510-49a6-ace3-f72da2adef72-credential-socket\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-qmsfx\" (UID: \"4b7dab75-7510-49a6-ace3-f72da2adef72\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-qmsfx" Oct 31 00:16:49.777907 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:49.777883 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b7dab75-7510-49a6-ace3-f72da2adef72-istio-podinfo\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-qmsfx\" (UID: \"4b7dab75-7510-49a6-ace3-f72da2adef72\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-qmsfx" Oct 31 00:16:49.777946 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:49.777922 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b7dab75-7510-49a6-ace3-f72da2adef72-workload-socket\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-qmsfx\" (UID: \"4b7dab75-7510-49a6-ace3-f72da2adef72\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-qmsfx" Oct 31 00:16:49.777981 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:49.777971 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b7dab75-7510-49a6-ace3-f72da2adef72-workload-certs\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-qmsfx\" (UID: \"4b7dab75-7510-49a6-ace3-f72da2adef72\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-qmsfx" Oct 31 00:16:49.778031 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:49.778020 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b7dab75-7510-49a6-ace3-f72da2adef72-istio-envoy\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-qmsfx\" (UID: \"4b7dab75-7510-49a6-ace3-f72da2adef72\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-qmsfx" Oct 31 00:16:49.778068 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:49.778037 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b7dab75-7510-49a6-ace3-f72da2adef72-istiod-ca-cert\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-qmsfx\" (UID: \"4b7dab75-7510-49a6-ace3-f72da2adef72\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-qmsfx" Oct 31 00:16:49.879119 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:49.879025 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b7dab75-7510-49a6-ace3-f72da2adef72-istio-envoy\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-qmsfx\" (UID: \"4b7dab75-7510-49a6-ace3-f72da2adef72\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-qmsfx" Oct 31 00:16:49.879119 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:49.879067 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b7dab75-7510-49a6-ace3-f72da2adef72-istiod-ca-cert\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-qmsfx\" (UID: \"4b7dab75-7510-49a6-ace3-f72da2adef72\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-qmsfx" Oct 31 00:16:49.879388 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:49.879164 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b7dab75-7510-49a6-ace3-f72da2adef72-istio-data\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-qmsfx\" (UID: \"4b7dab75-7510-49a6-ace3-f72da2adef72\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-qmsfx" Oct 31 00:16:49.879388 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:49.879238 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b7dab75-7510-49a6-ace3-f72da2adef72-istio-token\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-qmsfx\" (UID: \"4b7dab75-7510-49a6-ace3-f72da2adef72\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-qmsfx" Oct 31 00:16:49.879388 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:49.879309 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kg5lj\" (UniqueName: \"kubernetes.io/projected/4b7dab75-7510-49a6-ace3-f72da2adef72-kube-api-access-kg5lj\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-qmsfx\" (UID: \"4b7dab75-7510-49a6-ace3-f72da2adef72\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-qmsfx" Oct 31 00:16:49.879568 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:49.879397 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b7dab75-7510-49a6-ace3-f72da2adef72-credential-socket\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-qmsfx\" (UID: \"4b7dab75-7510-49a6-ace3-f72da2adef72\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-qmsfx" Oct 31 00:16:49.879568 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:49.879467 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b7dab75-7510-49a6-ace3-f72da2adef72-istio-podinfo\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-qmsfx\" (UID: \"4b7dab75-7510-49a6-ace3-f72da2adef72\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-qmsfx" Oct 31 00:16:49.879568 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:49.879505 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b7dab75-7510-49a6-ace3-f72da2adef72-workload-socket\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-qmsfx\" (UID: \"4b7dab75-7510-49a6-ace3-f72da2adef72\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-qmsfx" Oct 31 00:16:49.879568 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:49.879545 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b7dab75-7510-49a6-ace3-f72da2adef72-workload-certs\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-qmsfx\" (UID: \"4b7dab75-7510-49a6-ace3-f72da2adef72\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-qmsfx" Oct 31 00:16:49.879935 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:49.879597 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b7dab75-7510-49a6-ace3-f72da2adef72-istio-data\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-qmsfx\" (UID: \"4b7dab75-7510-49a6-ace3-f72da2adef72\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-qmsfx" Oct 31 00:16:49.879935 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:49.879776 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b7dab75-7510-49a6-ace3-f72da2adef72-credential-socket\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-qmsfx\" (UID: \"4b7dab75-7510-49a6-ace3-f72da2adef72\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-qmsfx" Oct 31 00:16:49.879935 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:49.879835 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b7dab75-7510-49a6-ace3-f72da2adef72-workload-socket\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-qmsfx\" (UID: \"4b7dab75-7510-49a6-ace3-f72da2adef72\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-qmsfx" Oct 31 00:16:49.879935 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:49.879886 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b7dab75-7510-49a6-ace3-f72da2adef72-istiod-ca-cert\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-qmsfx\" (UID: \"4b7dab75-7510-49a6-ace3-f72da2adef72\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-qmsfx" Oct 31 00:16:49.880104 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:49.879944 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b7dab75-7510-49a6-ace3-f72da2adef72-workload-certs\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-qmsfx\" (UID: \"4b7dab75-7510-49a6-ace3-f72da2adef72\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-qmsfx" Oct 31 00:16:49.881846 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:49.881826 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b7dab75-7510-49a6-ace3-f72da2adef72-istio-envoy\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-qmsfx\" (UID: \"4b7dab75-7510-49a6-ace3-f72da2adef72\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-qmsfx" Oct 31 00:16:49.882120 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:49.882099 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b7dab75-7510-49a6-ace3-f72da2adef72-istio-podinfo\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-qmsfx\" (UID: \"4b7dab75-7510-49a6-ace3-f72da2adef72\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-qmsfx" Oct 31 00:16:49.942449 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:49.942419 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b7dab75-7510-49a6-ace3-f72da2adef72-istio-token\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-qmsfx\" (UID: \"4b7dab75-7510-49a6-ace3-f72da2adef72\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-qmsfx" Oct 31 00:16:49.942577 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:49.942552 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kg5lj\" (UniqueName: \"kubernetes.io/projected/4b7dab75-7510-49a6-ace3-f72da2adef72-kube-api-access-kg5lj\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-qmsfx\" (UID: \"4b7dab75-7510-49a6-ace3-f72da2adef72\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-qmsfx" Oct 31 00:16:50.241183 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:50.241134 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-qmsfx" Oct 31 00:16:50.462303 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:50.462263 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:16:50.711971 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:50.711941 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-qmsfx"] Oct 31 00:16:50.715829 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:50.715794 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:16:50.715936 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:50.715879 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:16:50.715936 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:50.715923 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:16:51.044714 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:51.044611 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-qmsfx" event={"ID":"4b7dab75-7510-49a6-ace3-f72da2adef72","Type":"ContainerStarted","Data":"cd1513b92d6b20f19598f3496630c1fbbd4d2c3e11b905dacb8d6cf3212592c3"} Oct 31 00:16:51.044714 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:51.044674 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-qmsfx" event={"ID":"4b7dab75-7510-49a6-ace3-f72da2adef72","Type":"ContainerStarted","Data":"17180e51f67db863588cc9349c62454864303f7dbee8ad9eb78c987e1d2c7d4a"} Oct 31 00:16:51.169634 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:51.169581 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-qmsfx" podStartSLOduration=2.16956198 podStartE2EDuration="2.16956198s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:16:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:16:51.166839162 +0000 UTC m=+2124.203170047" watchObservedRunningTime="2026-10-31 00:16:51.16956198 +0000 UTC m=+2124.205892802" Oct 31 00:16:51.242071 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:51.242029 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-qmsfx" Oct 31 00:16:51.243412 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:51.243381 4927 patch_prober.go:29] interesting pod/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-qmsfx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.229:15021/healthz/ready\": dial tcp 10.132.0.229:15021: connect: connection refused" start-of-body= Oct 31 00:16:51.243569 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:51.243442 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-qmsfx" podUID="4b7dab75-7510-49a6-ace3-f72da2adef72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.229:15021/healthz/ready\": dial tcp 10.132.0.229:15021: connect: connection refused" Oct 31 00:16:52.241989 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:52.241959 4927 patch_prober.go:29] interesting pod/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-qmsfx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.229:15021/healthz/ready\": dial tcp 10.132.0.229:15021: connect: connection refused" start-of-body= Oct 31 00:16:52.242443 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:52.242009 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-qmsfx" podUID="4b7dab75-7510-49a6-ace3-f72da2adef72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.229:15021/healthz/ready\": dial tcp 10.132.0.229:15021: connect: connection refused" Oct 31 00:16:53.079946 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:53.079847 4927 generic.go:356] "Generic (PLEG): container finished" podID="c51c586a-f2d8-48f9-8813-3b6e4f360e5e" containerID="2a76af001dfaa02e742381689a56aec54d0870d03c8513bf137af374832cd231" exitCode=0 Oct 31 00:16:53.095000 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:53.094963 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-71b76a-openshift-default-77db967c97-lh4ld" event={"ID":"c51c586a-f2d8-48f9-8813-3b6e4f360e5e","Type":"ContainerDied","Data":"2a76af001dfaa02e742381689a56aec54d0870d03c8513bf137af374832cd231"} Oct 31 00:16:53.095225 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:53.095207 4927 scope.go:122] "RemoveContainer" containerID="aabe1a791b1aa93cf5e47255b266a37d6cdc8ec03231adcc7bae54204476a0b3" Oct 31 00:16:53.177755 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:53.177728 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-71b76a-openshift-default-77db967c97-lh4ld" Oct 31 00:16:53.243827 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:53.243773 4927 patch_prober.go:29] interesting pod/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-qmsfx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.229:15021/healthz/ready\": dial tcp 10.132.0.229:15021: connect: connection refused" start-of-body= Oct 31 00:16:53.244382 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:53.243848 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-qmsfx" podUID="4b7dab75-7510-49a6-ace3-f72da2adef72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.229:15021/healthz/ready\": dial tcp 10.132.0.229:15021: connect: connection refused" Oct 31 00:16:53.359395 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:53.359287 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c51c586a-f2d8-48f9-8813-3b6e4f360e5e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c51c586a-f2d8-48f9-8813-3b6e4f360e5e-istio-envoy\") pod \"c51c586a-f2d8-48f9-8813-3b6e4f360e5e\" (UID: \"c51c586a-f2d8-48f9-8813-3b6e4f360e5e\") " Oct 31 00:16:53.359551 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:53.359519 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c51c586a-f2d8-48f9-8813-3b6e4f360e5e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c51c586a-f2d8-48f9-8813-3b6e4f360e5e-credential-socket\") pod \"c51c586a-f2d8-48f9-8813-3b6e4f360e5e\" (UID: \"c51c586a-f2d8-48f9-8813-3b6e4f360e5e\") " Oct 31 00:16:53.359609 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:53.359594 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c51c586a-f2d8-48f9-8813-3b6e4f360e5e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c51c586a-f2d8-48f9-8813-3b6e4f360e5e-istio-podinfo\") pod \"c51c586a-f2d8-48f9-8813-3b6e4f360e5e\" (UID: \"c51c586a-f2d8-48f9-8813-3b6e4f360e5e\") " Oct 31 00:16:53.359655 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:53.359630 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c51c586a-f2d8-48f9-8813-3b6e4f360e5e-istio-token\" (UniqueName: \"kubernetes.io/projected/c51c586a-f2d8-48f9-8813-3b6e4f360e5e-istio-token\") pod \"c51c586a-f2d8-48f9-8813-3b6e4f360e5e\" (UID: \"c51c586a-f2d8-48f9-8813-3b6e4f360e5e\") " Oct 31 00:16:53.360664 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:53.360318 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c51c586a-f2d8-48f9-8813-3b6e4f360e5e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c51c586a-f2d8-48f9-8813-3b6e4f360e5e-workload-certs\") pod \"c51c586a-f2d8-48f9-8813-3b6e4f360e5e\" (UID: \"c51c586a-f2d8-48f9-8813-3b6e4f360e5e\") " Oct 31 00:16:53.360816 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:53.360716 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c51c586a-f2d8-48f9-8813-3b6e4f360e5e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c51c586a-f2d8-48f9-8813-3b6e4f360e5e-istiod-ca-cert\") pod \"c51c586a-f2d8-48f9-8813-3b6e4f360e5e\" (UID: \"c51c586a-f2d8-48f9-8813-3b6e4f360e5e\") " Oct 31 00:16:53.360816 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:53.360234 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c51c586a-f2d8-48f9-8813-3b6e4f360e5e-credential-socket" pod "c51c586a-f2d8-48f9-8813-3b6e4f360e5e" (UID: "c51c586a-f2d8-48f9-8813-3b6e4f360e5e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:16:53.360816 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:53.360632 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c51c586a-f2d8-48f9-8813-3b6e4f360e5e-workload-certs" pod "c51c586a-f2d8-48f9-8813-3b6e4f360e5e" (UID: "c51c586a-f2d8-48f9-8813-3b6e4f360e5e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:16:53.361139 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:53.361107 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c51c586a-f2d8-48f9-8813-3b6e4f360e5e-istiod-ca-cert" pod "c51c586a-f2d8-48f9-8813-3b6e4f360e5e" (UID: "c51c586a-f2d8-48f9-8813-3b6e4f360e5e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:16:53.361305 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:53.360798 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c51c586a-f2d8-48f9-8813-3b6e4f360e5e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c51c586a-f2d8-48f9-8813-3b6e4f360e5e-workload-socket\") pod \"c51c586a-f2d8-48f9-8813-3b6e4f360e5e\" (UID: \"c51c586a-f2d8-48f9-8813-3b6e4f360e5e\") " Oct 31 00:16:53.361860 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:53.361728 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c51c586a-f2d8-48f9-8813-3b6e4f360e5e-workload-socket" pod "c51c586a-f2d8-48f9-8813-3b6e4f360e5e" (UID: "c51c586a-f2d8-48f9-8813-3b6e4f360e5e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:16:53.362321 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:53.362304 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c51c586a-f2d8-48f9-8813-3b6e4f360e5e-kube-api-access-fl8ph\" (UniqueName: \"kubernetes.io/projected/c51c586a-f2d8-48f9-8813-3b6e4f360e5e-kube-api-access-fl8ph\") pod \"c51c586a-f2d8-48f9-8813-3b6e4f360e5e\" (UID: \"c51c586a-f2d8-48f9-8813-3b6e4f360e5e\") " Oct 31 00:16:53.362632 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:53.362616 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c51c586a-f2d8-48f9-8813-3b6e4f360e5e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c51c586a-f2d8-48f9-8813-3b6e4f360e5e-istio-data\") pod \"c51c586a-f2d8-48f9-8813-3b6e4f360e5e\" (UID: \"c51c586a-f2d8-48f9-8813-3b6e4f360e5e\") " Oct 31 00:16:53.363087 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:53.362955 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c51c586a-f2d8-48f9-8813-3b6e4f360e5e-istio-data" pod "c51c586a-f2d8-48f9-8813-3b6e4f360e5e" (UID: "c51c586a-f2d8-48f9-8813-3b6e4f360e5e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:16:53.363941 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:53.363916 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c51c586a-f2d8-48f9-8813-3b6e4f360e5e-kube-api-access-fl8ph" pod "c51c586a-f2d8-48f9-8813-3b6e4f360e5e" (UID: "c51c586a-f2d8-48f9-8813-3b6e4f360e5e"). InnerVolumeSpecName "kube-api-access-fl8ph". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:16:53.363941 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:53.363923 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c51c586a-f2d8-48f9-8813-3b6e4f360e5e-istio-envoy" pod "c51c586a-f2d8-48f9-8813-3b6e4f360e5e" (UID: "c51c586a-f2d8-48f9-8813-3b6e4f360e5e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:16:53.364302 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:53.364286 4927 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c51c586a-f2d8-48f9-8813-3b6e4f360e5e-istio-data\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:16:53.364453 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:53.364440 4927 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c51c586a-f2d8-48f9-8813-3b6e4f360e5e-istio-envoy\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:16:53.364555 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:53.364544 4927 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c51c586a-f2d8-48f9-8813-3b6e4f360e5e-credential-socket\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:16:53.364653 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:53.364640 4927 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c51c586a-f2d8-48f9-8813-3b6e4f360e5e-workload-certs\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:16:53.364752 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:53.364741 4927 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c51c586a-f2d8-48f9-8813-3b6e4f360e5e-istiod-ca-cert\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:16:53.364833 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:53.364823 4927 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c51c586a-f2d8-48f9-8813-3b6e4f360e5e-workload-socket\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:16:53.364926 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:53.364915 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fl8ph\" (UniqueName: \"kubernetes.io/projected/c51c586a-f2d8-48f9-8813-3b6e4f360e5e-kube-api-access-fl8ph\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:16:53.367146 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:53.367121 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c51c586a-f2d8-48f9-8813-3b6e4f360e5e-istio-token" pod "c51c586a-f2d8-48f9-8813-3b6e4f360e5e" (UID: "c51c586a-f2d8-48f9-8813-3b6e4f360e5e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:16:53.368003 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:53.367974 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c51c586a-f2d8-48f9-8813-3b6e4f360e5e-istio-podinfo" pod "c51c586a-f2d8-48f9-8813-3b6e4f360e5e" (UID: "c51c586a-f2d8-48f9-8813-3b6e4f360e5e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:16:53.466258 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:53.466213 4927 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c51c586a-f2d8-48f9-8813-3b6e4f360e5e-istio-podinfo\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:16:53.466258 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:53.466244 4927 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c51c586a-f2d8-48f9-8813-3b6e4f360e5e-istio-token\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:16:54.089873 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:54.089846 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-71b76a-openshift-default-77db967c97-lh4ld" Oct 31 00:16:54.090058 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:54.089916 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-71b76a-openshift-default-77db967c97-lh4ld" event={"ID":"c51c586a-f2d8-48f9-8813-3b6e4f360e5e","Type":"ContainerDied","Data":"218f4d1e80a3de938f62ea4a5c245e7eeb4c4eecd358cfaab34d7edc2ff76ebf"} Oct 31 00:16:54.090058 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:54.089974 4927 scope.go:122] "RemoveContainer" containerID="2a76af001dfaa02e742381689a56aec54d0870d03c8513bf137af374832cd231" Oct 31 00:16:54.242375 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:54.241918 4927 patch_prober.go:29] interesting pod/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-qmsfx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.229:15021/healthz/ready\": dial tcp 10.132.0.229:15021: connect: connection refused" start-of-body= Oct 31 00:16:54.242375 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:54.241991 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-qmsfx" podUID="4b7dab75-7510-49a6-ace3-f72da2adef72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.229:15021/healthz/ready\": dial tcp 10.132.0.229:15021: connect: connection refused" Oct 31 00:16:54.384564 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:54.384414 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-71b76a-openshift-default-77db967c97-lh4ld"] Oct 31 00:16:54.394543 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:54.394500 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-71b76a-openshift-default-77db967c97-lh4ld"] Oct 31 00:16:55.088801 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:55.088769 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c51c586a-f2d8-48f9-8813-3b6e4f360e5e" path="/var/lib/kubelet/pods/c51c586a-f2d8-48f9-8813-3b6e4f360e5e/volumes" Oct 31 00:16:55.241958 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:55.241917 4927 patch_prober.go:29] interesting pod/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-qmsfx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.229:15021/healthz/ready\": dial tcp 10.132.0.229:15021: connect: connection refused" start-of-body= Oct 31 00:16:55.242164 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:55.242001 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-qmsfx" podUID="4b7dab75-7510-49a6-ace3-f72da2adef72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.229:15021/healthz/ready\": dial tcp 10.132.0.229:15021: connect: connection refused" Oct 31 00:16:55.491553 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:55.491506 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:16:56.242512 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:56.242358 4927 patch_prober.go:29] interesting pod/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-qmsfx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.229:15021/healthz/ready\": dial tcp 10.132.0.229:15021: connect: connection refused" start-of-body= Oct 31 00:16:56.242512 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:56.242425 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-qmsfx" podUID="4b7dab75-7510-49a6-ace3-f72da2adef72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.229:15021/healthz/ready\": dial tcp 10.132.0.229:15021: connect: connection refused" Oct 31 00:16:57.247047 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:57.247018 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-qmsfx" Oct 31 00:16:57.247476 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:57.247449 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-qmsfx" Oct 31 00:16:57.248260 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:57.248240 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-qmsfx" Oct 31 00:16:58.176116 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:58.176084 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:16:58.391765 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:16:58.391728 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:17:00.126344 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:00.126302 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-348a33-6987f947f-cqp2m"] Oct 31 00:17:00.381945 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:00.381863 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:17:00.561946 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:00.561909 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:17:00.720312 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:00.720275 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:17:03.540227 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:03.540091 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-39e3a8-77d67d47f8-msm6p"] Oct 31 00:17:03.650387 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:03.650348 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-39e3a8-545496b975-zvh7k"] Oct 31 00:17:03.872684 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:03.872599 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:17:04.038804 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:04.038768 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:17:04.968905 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:04.968867 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:17:05.158355 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:05.157754 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-348a33-6987f947f-cqp2m" podUID="31bc5ff3-08ec-470f-b1f9-713946258110" containerName="payload-pre-processing" containerID="cri-o://77daa4a37e090eaed08df3549337816e2d553dd41bcefbf7a0a2eb08897d6743" gracePeriod=25 Oct 31 00:17:05.844611 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:05.844576 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-348a33-6987f947f-cqp2m" Oct 31 00:17:05.999802 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:05.999651 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31bc5ff3-08ec-470f-b1f9-713946258110-kube-api-access-zzpt7\" (UniqueName: \"kubernetes.io/projected/31bc5ff3-08ec-470f-b1f9-713946258110-kube-api-access-zzpt7\") pod \"31bc5ff3-08ec-470f-b1f9-713946258110\" (UID: \"31bc5ff3-08ec-470f-b1f9-713946258110\") " Oct 31 00:17:06.000987 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:05.999948 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/31bc5ff3-08ec-470f-b1f9-713946258110-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31bc5ff3-08ec-470f-b1f9-713946258110-plugins-config-volume\") pod \"31bc5ff3-08ec-470f-b1f9-713946258110\" (UID: \"31bc5ff3-08ec-470f-b1f9-713946258110\") " Oct 31 00:17:06.001163 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:06.001124 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31bc5ff3-08ec-470f-b1f9-713946258110-plugins-config-volume" pod "31bc5ff3-08ec-470f-b1f9-713946258110" (UID: "31bc5ff3-08ec-470f-b1f9-713946258110"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:17:06.008926 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:06.008883 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31bc5ff3-08ec-470f-b1f9-713946258110-kube-api-access-zzpt7" pod "31bc5ff3-08ec-470f-b1f9-713946258110" (UID: "31bc5ff3-08ec-470f-b1f9-713946258110"). InnerVolumeSpecName "kube-api-access-zzpt7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:17:06.101741 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:06.101706 4927 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31bc5ff3-08ec-470f-b1f9-713946258110-plugins-config-volume\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:17:06.101741 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:06.101735 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zzpt7\" (UniqueName: \"kubernetes.io/projected/31bc5ff3-08ec-470f-b1f9-713946258110-kube-api-access-zzpt7\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:17:06.270274 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:06.270186 4927 generic.go:356] "Generic (PLEG): container finished" podID="31bc5ff3-08ec-470f-b1f9-713946258110" containerID="77daa4a37e090eaed08df3549337816e2d553dd41bcefbf7a0a2eb08897d6743" exitCode=0 Oct 31 00:17:06.270274 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:06.270238 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-348a33-6987f947f-cqp2m" Oct 31 00:17:06.270274 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:06.270247 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-348a33-6987f947f-cqp2m" event={"ID":"31bc5ff3-08ec-470f-b1f9-713946258110","Type":"ContainerDied","Data":"77daa4a37e090eaed08df3549337816e2d553dd41bcefbf7a0a2eb08897d6743"} Oct 31 00:17:06.270609 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:06.270298 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-348a33-6987f947f-cqp2m" event={"ID":"31bc5ff3-08ec-470f-b1f9-713946258110","Type":"ContainerDied","Data":"a71611b508fd833c32f61a19f3fd2cf78d5c0b6f8306f59eaf052c8faf7cddc3"} Oct 31 00:17:06.270609 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:06.270344 4927 scope.go:122] "RemoveContainer" containerID="77daa4a37e090eaed08df3549337816e2d553dd41bcefbf7a0a2eb08897d6743" Oct 31 00:17:06.282028 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:06.282006 4927 scope.go:122] "RemoveContainer" containerID="77daa4a37e090eaed08df3549337816e2d553dd41bcefbf7a0a2eb08897d6743" Oct 31 00:17:06.282351 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:17:06.282307 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"77daa4a37e090eaed08df3549337816e2d553dd41bcefbf7a0a2eb08897d6743\": container with ID starting with 77daa4a37e090eaed08df3549337816e2d553dd41bcefbf7a0a2eb08897d6743 not found: ID does not exist" containerID="77daa4a37e090eaed08df3549337816e2d553dd41bcefbf7a0a2eb08897d6743" Oct 31 00:17:06.282458 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:06.282363 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77daa4a37e090eaed08df3549337816e2d553dd41bcefbf7a0a2eb08897d6743"} err="failed to get container status \"77daa4a37e090eaed08df3549337816e2d553dd41bcefbf7a0a2eb08897d6743\": rpc error: code = NotFound desc = could not find container \"77daa4a37e090eaed08df3549337816e2d553dd41bcefbf7a0a2eb08897d6743\": container with ID starting with 77daa4a37e090eaed08df3549337816e2d553dd41bcefbf7a0a2eb08897d6743 not found: ID does not exist" Oct 31 00:17:06.457281 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:06.457244 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-348a33-6987f947f-cqp2m"] Oct 31 00:17:06.508787 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:06.508749 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-348a33-6987f947f-cqp2m"] Oct 31 00:17:06.621206 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:06.621112 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:17:06.936131 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:06.936091 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:17:07.090792 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:07.090755 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="31bc5ff3-08ec-470f-b1f9-713946258110" path="/var/lib/kubelet/pods/31bc5ff3-08ec-470f-b1f9-713946258110/volumes" Oct 31 00:17:07.095380 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:07.095348 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:17:07.565246 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:07.565209 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:17:08.552355 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:08.548863 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-39e3a8-77d67d47f8-msm6p" podUID="d639aac3-8752-495f-8c71-fdc050a03016" containerName="payload-processing" containerID="cri-o://f0cf05276ff8f0af48fa34cdc41c1996d82b68c2f0f9dd0ef839a5dde96cfeae" gracePeriod=25 Oct 31 00:17:08.651491 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:08.651407 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-39e3a8-545496b975-zvh7k" podUID="bb4e1d22-509e-4c3f-b877-8094320fc9ed" containerName="payload-pre-processing" containerID="cri-o://3b73a43624872dcf2600dbd50de0b019e2514dbf3c1fafb527f23ec7f5ce5fd0" gracePeriod=25 Oct 31 00:17:08.902507 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:08.902461 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-39e3a8-77d67d47f8-msm6p" Oct 31 00:17:08.971095 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:08.969868 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d639aac3-8752-495f-8c71-fdc050a03016-kube-api-access-qhcnk\" (UniqueName: \"kubernetes.io/projected/d639aac3-8752-495f-8c71-fdc050a03016-kube-api-access-qhcnk\") pod \"d639aac3-8752-495f-8c71-fdc050a03016\" (UID: \"d639aac3-8752-495f-8c71-fdc050a03016\") " Oct 31 00:17:08.971095 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:08.970082 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d639aac3-8752-495f-8c71-fdc050a03016-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d639aac3-8752-495f-8c71-fdc050a03016-plugins-config-volume\") pod \"d639aac3-8752-495f-8c71-fdc050a03016\" (UID: \"d639aac3-8752-495f-8c71-fdc050a03016\") " Oct 31 00:17:08.971095 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:08.970997 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d639aac3-8752-495f-8c71-fdc050a03016-plugins-config-volume" pod "d639aac3-8752-495f-8c71-fdc050a03016" (UID: "d639aac3-8752-495f-8c71-fdc050a03016"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:17:08.981971 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:08.981935 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d639aac3-8752-495f-8c71-fdc050a03016-kube-api-access-qhcnk" pod "d639aac3-8752-495f-8c71-fdc050a03016" (UID: "d639aac3-8752-495f-8c71-fdc050a03016"). InnerVolumeSpecName "kube-api-access-qhcnk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:17:09.073427 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:09.073392 4927 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d639aac3-8752-495f-8c71-fdc050a03016-plugins-config-volume\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:17:09.073427 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:09.073428 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qhcnk\" (UniqueName: \"kubernetes.io/projected/d639aac3-8752-495f-8c71-fdc050a03016-kube-api-access-qhcnk\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:17:09.089189 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:09.089162 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-39e3a8-545496b975-zvh7k" Oct 31 00:17:09.188002 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:09.187957 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb4e1d22-509e-4c3f-b877-8094320fc9ed-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb4e1d22-509e-4c3f-b877-8094320fc9ed-plugins-config-volume\") pod \"bb4e1d22-509e-4c3f-b877-8094320fc9ed\" (UID: \"bb4e1d22-509e-4c3f-b877-8094320fc9ed\") " Oct 31 00:17:09.188520 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:09.188495 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb4e1d22-509e-4c3f-b877-8094320fc9ed-kube-api-access-8ml6x\" (UniqueName: \"kubernetes.io/projected/bb4e1d22-509e-4c3f-b877-8094320fc9ed-kube-api-access-8ml6x\") pod \"bb4e1d22-509e-4c3f-b877-8094320fc9ed\" (UID: \"bb4e1d22-509e-4c3f-b877-8094320fc9ed\") " Oct 31 00:17:09.188992 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:09.188590 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb4e1d22-509e-4c3f-b877-8094320fc9ed-plugins-config-volume" pod "bb4e1d22-509e-4c3f-b877-8094320fc9ed" (UID: "bb4e1d22-509e-4c3f-b877-8094320fc9ed"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:17:09.189413 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:09.189382 4927 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb4e1d22-509e-4c3f-b877-8094320fc9ed-plugins-config-volume\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:17:09.191573 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:09.191539 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb4e1d22-509e-4c3f-b877-8094320fc9ed-kube-api-access-8ml6x" pod "bb4e1d22-509e-4c3f-b877-8094320fc9ed" (UID: "bb4e1d22-509e-4c3f-b877-8094320fc9ed"). InnerVolumeSpecName "kube-api-access-8ml6x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:17:09.291720 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:09.291679 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8ml6x\" (UniqueName: \"kubernetes.io/projected/bb4e1d22-509e-4c3f-b877-8094320fc9ed-kube-api-access-8ml6x\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:17:09.296830 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:09.296793 4927 generic.go:356] "Generic (PLEG): container finished" podID="d639aac3-8752-495f-8c71-fdc050a03016" containerID="f0cf05276ff8f0af48fa34cdc41c1996d82b68c2f0f9dd0ef839a5dde96cfeae" exitCode=0 Oct 31 00:17:09.297010 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:09.296866 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-39e3a8-77d67d47f8-msm6p" Oct 31 00:17:09.297010 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:09.296883 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-39e3a8-77d67d47f8-msm6p" event={"ID":"d639aac3-8752-495f-8c71-fdc050a03016","Type":"ContainerDied","Data":"f0cf05276ff8f0af48fa34cdc41c1996d82b68c2f0f9dd0ef839a5dde96cfeae"} Oct 31 00:17:09.297010 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:09.296983 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-39e3a8-77d67d47f8-msm6p" event={"ID":"d639aac3-8752-495f-8c71-fdc050a03016","Type":"ContainerDied","Data":"a0cf8d283bcd81efd941211c6f197286fae82c1a733d877604b2eea34894b6fe"} Oct 31 00:17:09.297188 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:09.297014 4927 scope.go:122] "RemoveContainer" containerID="f0cf05276ff8f0af48fa34cdc41c1996d82b68c2f0f9dd0ef839a5dde96cfeae" Oct 31 00:17:09.299649 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:09.299625 4927 generic.go:356] "Generic (PLEG): container finished" podID="bb4e1d22-509e-4c3f-b877-8094320fc9ed" containerID="3b73a43624872dcf2600dbd50de0b019e2514dbf3c1fafb527f23ec7f5ce5fd0" exitCode=0 Oct 31 00:17:09.299874 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:09.299856 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-39e3a8-545496b975-zvh7k" event={"ID":"bb4e1d22-509e-4c3f-b877-8094320fc9ed","Type":"ContainerDied","Data":"3b73a43624872dcf2600dbd50de0b019e2514dbf3c1fafb527f23ec7f5ce5fd0"} Oct 31 00:17:09.299951 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:09.299901 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-39e3a8-545496b975-zvh7k" event={"ID":"bb4e1d22-509e-4c3f-b877-8094320fc9ed","Type":"ContainerDied","Data":"df2cc1246d1ecfd32bacbe85effb277e14f1ffea2f8920cdd98be7d613c00bc3"} Oct 31 00:17:09.300017 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:09.299984 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-39e3a8-545496b975-zvh7k" Oct 31 00:17:09.314356 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:09.314299 4927 scope.go:122] "RemoveContainer" containerID="f0cf05276ff8f0af48fa34cdc41c1996d82b68c2f0f9dd0ef839a5dde96cfeae" Oct 31 00:17:09.314777 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:17:09.314750 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0cf05276ff8f0af48fa34cdc41c1996d82b68c2f0f9dd0ef839a5dde96cfeae\": container with ID starting with f0cf05276ff8f0af48fa34cdc41c1996d82b68c2f0f9dd0ef839a5dde96cfeae not found: ID does not exist" containerID="f0cf05276ff8f0af48fa34cdc41c1996d82b68c2f0f9dd0ef839a5dde96cfeae" Oct 31 00:17:09.314864 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:09.314787 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0cf05276ff8f0af48fa34cdc41c1996d82b68c2f0f9dd0ef839a5dde96cfeae"} err="failed to get container status \"f0cf05276ff8f0af48fa34cdc41c1996d82b68c2f0f9dd0ef839a5dde96cfeae\": rpc error: code = NotFound desc = could not find container \"f0cf05276ff8f0af48fa34cdc41c1996d82b68c2f0f9dd0ef839a5dde96cfeae\": container with ID starting with f0cf05276ff8f0af48fa34cdc41c1996d82b68c2f0f9dd0ef839a5dde96cfeae not found: ID does not exist" Oct 31 00:17:09.314864 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:09.314814 4927 scope.go:122] "RemoveContainer" containerID="3b73a43624872dcf2600dbd50de0b019e2514dbf3c1fafb527f23ec7f5ce5fd0" Oct 31 00:17:09.327141 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:09.327122 4927 scope.go:122] "RemoveContainer" containerID="3b73a43624872dcf2600dbd50de0b019e2514dbf3c1fafb527f23ec7f5ce5fd0" Oct 31 00:17:09.327498 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:17:09.327466 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b73a43624872dcf2600dbd50de0b019e2514dbf3c1fafb527f23ec7f5ce5fd0\": container with ID starting with 3b73a43624872dcf2600dbd50de0b019e2514dbf3c1fafb527f23ec7f5ce5fd0 not found: ID does not exist" containerID="3b73a43624872dcf2600dbd50de0b019e2514dbf3c1fafb527f23ec7f5ce5fd0" Oct 31 00:17:09.327614 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:09.327507 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b73a43624872dcf2600dbd50de0b019e2514dbf3c1fafb527f23ec7f5ce5fd0"} err="failed to get container status \"3b73a43624872dcf2600dbd50de0b019e2514dbf3c1fafb527f23ec7f5ce5fd0\": rpc error: code = NotFound desc = could not find container \"3b73a43624872dcf2600dbd50de0b019e2514dbf3c1fafb527f23ec7f5ce5fd0\": container with ID starting with 3b73a43624872dcf2600dbd50de0b019e2514dbf3c1fafb527f23ec7f5ce5fd0 not found: ID does not exist" Oct 31 00:17:09.507871 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:09.507784 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-39e3a8-77d67d47f8-msm6p"] Oct 31 00:17:09.516135 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:09.516101 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-39e3a8-77d67d47f8-msm6p"] Oct 31 00:17:09.646426 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:09.646385 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-39e3a8-545496b975-zvh7k"] Oct 31 00:17:09.648624 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:09.648579 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-39e3a8-545496b975-zvh7k"] Oct 31 00:17:11.088610 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:11.088576 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb4e1d22-509e-4c3f-b877-8094320fc9ed" path="/var/lib/kubelet/pods/bb4e1d22-509e-4c3f-b877-8094320fc9ed/volumes" Oct 31 00:17:11.089079 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:11.089063 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d639aac3-8752-495f-8c71-fdc050a03016" path="/var/lib/kubelet/pods/d639aac3-8752-495f-8c71-fdc050a03016/volumes" Oct 31 00:17:12.366358 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:12.356426 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:17:12.836530 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:12.836489 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:17:13.027942 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:13.027909 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:17:14.168499 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:14.161897 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:17:14.323928 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:14.323892 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:17:14.869355 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:14.868701 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:17:16.289420 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:16.289381 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-03c703-587b89bbdd-vzfkm"] Oct 31 00:17:16.398988 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:16.398955 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-03c703-68ccf588f9-957fw"] Oct 31 00:17:16.423404 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:16.420646 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-ab0ac9-5b7888b747-pxq8l"] Oct 31 00:17:16.423404 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:16.420886 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-ab0ac9-5b7888b747-pxq8l" podUID="e8587ee7-1b21-409a-8c2d-ca9d7b69be4a" containerName="maas-api" containerID="cri-o://555021ce9e0b1d669136f3a2152a57aa9794f2ab916744e8f011247599f01083" gracePeriod=30 Oct 31 00:17:16.488556 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:16.488523 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:17:16.694200 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:16.694154 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:17:16.770747 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:16.770715 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-ab0ac9-5b7888b747-pxq8l" Oct 31 00:17:16.847409 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:16.845126 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:17:16.875356 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:16.866230 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8587ee7-1b21-409a-8c2d-ca9d7b69be4a-kube-api-access-lz649\" (UniqueName: \"kubernetes.io/projected/e8587ee7-1b21-409a-8c2d-ca9d7b69be4a-kube-api-access-lz649\") pod \"e8587ee7-1b21-409a-8c2d-ca9d7b69be4a\" (UID: \"e8587ee7-1b21-409a-8c2d-ca9d7b69be4a\") " Oct 31 00:17:16.875356 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:16.866519 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e8587ee7-1b21-409a-8c2d-ca9d7b69be4a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e8587ee7-1b21-409a-8c2d-ca9d7b69be4a-metrics-certs\") pod \"e8587ee7-1b21-409a-8c2d-ca9d7b69be4a\" (UID: \"e8587ee7-1b21-409a-8c2d-ca9d7b69be4a\") " Oct 31 00:17:16.875356 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:16.866746 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e8587ee7-1b21-409a-8c2d-ca9d7b69be4a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e8587ee7-1b21-409a-8c2d-ca9d7b69be4a-maas-api-tls\") pod \"e8587ee7-1b21-409a-8c2d-ca9d7b69be4a\" (UID: \"e8587ee7-1b21-409a-8c2d-ca9d7b69be4a\") " Oct 31 00:17:16.875356 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:16.867293 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e8587ee7-1b21-409a-8c2d-ca9d7b69be4a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8587ee7-1b21-409a-8c2d-ca9d7b69be4a-openshift-service-ca\") pod \"e8587ee7-1b21-409a-8c2d-ca9d7b69be4a\" (UID: \"e8587ee7-1b21-409a-8c2d-ca9d7b69be4a\") " Oct 31 00:17:16.897151 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:16.893653 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8587ee7-1b21-409a-8c2d-ca9d7b69be4a-openshift-service-ca" pod "e8587ee7-1b21-409a-8c2d-ca9d7b69be4a" (UID: "e8587ee7-1b21-409a-8c2d-ca9d7b69be4a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:17:16.916582 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:16.915513 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8587ee7-1b21-409a-8c2d-ca9d7b69be4a-maas-api-tls" pod "e8587ee7-1b21-409a-8c2d-ca9d7b69be4a" (UID: "e8587ee7-1b21-409a-8c2d-ca9d7b69be4a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:17:16.963641 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:16.963595 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8587ee7-1b21-409a-8c2d-ca9d7b69be4a-metrics-certs" pod "e8587ee7-1b21-409a-8c2d-ca9d7b69be4a" (UID: "e8587ee7-1b21-409a-8c2d-ca9d7b69be4a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:17:16.963820 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:16.963746 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8587ee7-1b21-409a-8c2d-ca9d7b69be4a-kube-api-access-lz649" pod "e8587ee7-1b21-409a-8c2d-ca9d7b69be4a" (UID: "e8587ee7-1b21-409a-8c2d-ca9d7b69be4a"). InnerVolumeSpecName "kube-api-access-lz649". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:17:16.965492 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:16.964091 4927 generic.go:356] "Generic (PLEG): container finished" podID="e8587ee7-1b21-409a-8c2d-ca9d7b69be4a" containerID="555021ce9e0b1d669136f3a2152a57aa9794f2ab916744e8f011247599f01083" exitCode=0 Oct 31 00:17:16.965492 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:16.964392 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-ab0ac9-5b7888b747-pxq8l" event={"ID":"e8587ee7-1b21-409a-8c2d-ca9d7b69be4a","Type":"ContainerDied","Data":"555021ce9e0b1d669136f3a2152a57aa9794f2ab916744e8f011247599f01083"} Oct 31 00:17:16.965492 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:16.964510 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-ab0ac9-5b7888b747-pxq8l" event={"ID":"e8587ee7-1b21-409a-8c2d-ca9d7b69be4a","Type":"ContainerDied","Data":"b31911c0a5f877bcec2d78aefd91e7a7321bea32bd24859b600c4a749ae8d7ba"} Oct 31 00:17:16.965492 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:16.964543 4927 scope.go:122] "RemoveContainer" containerID="555021ce9e0b1d669136f3a2152a57aa9794f2ab916744e8f011247599f01083" Oct 31 00:17:16.965492 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:16.964778 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-ab0ac9-5b7888b747-pxq8l" Oct 31 00:17:16.975670 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:16.973788 4927 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8587ee7-1b21-409a-8c2d-ca9d7b69be4a-openshift-service-ca\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:17:16.975670 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:16.973823 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lz649\" (UniqueName: \"kubernetes.io/projected/e8587ee7-1b21-409a-8c2d-ca9d7b69be4a-kube-api-access-lz649\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:17:16.975670 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:16.973843 4927 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e8587ee7-1b21-409a-8c2d-ca9d7b69be4a-metrics-certs\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:17:16.975670 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:16.973858 4927 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e8587ee7-1b21-409a-8c2d-ca9d7b69be4a-maas-api-tls\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:17:17.004356 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:17.003240 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:17:17.040248 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:17.040182 4927 scope.go:122] "RemoveContainer" containerID="555021ce9e0b1d669136f3a2152a57aa9794f2ab916744e8f011247599f01083" Oct 31 00:17:17.041110 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:17:17.040727 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"555021ce9e0b1d669136f3a2152a57aa9794f2ab916744e8f011247599f01083\": container with ID starting with 555021ce9e0b1d669136f3a2152a57aa9794f2ab916744e8f011247599f01083 not found: ID does not exist" containerID="555021ce9e0b1d669136f3a2152a57aa9794f2ab916744e8f011247599f01083" Oct 31 00:17:17.041110 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:17.040766 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"555021ce9e0b1d669136f3a2152a57aa9794f2ab916744e8f011247599f01083"} err="failed to get container status \"555021ce9e0b1d669136f3a2152a57aa9794f2ab916744e8f011247599f01083\": rpc error: code = NotFound desc = could not find container \"555021ce9e0b1d669136f3a2152a57aa9794f2ab916744e8f011247599f01083\": container with ID starting with 555021ce9e0b1d669136f3a2152a57aa9794f2ab916744e8f011247599f01083 not found: ID does not exist" Oct 31 00:17:17.139752 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:17.139667 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-ab0ac9-5b7888b747-pxq8l"] Oct 31 00:17:17.145701 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:17.145671 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-ab0ac9-5b7888b747-pxq8l"] Oct 31 00:17:18.284815 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:18.284770 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:17:18.491301 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:18.491253 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:17:18.683355 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:18.680226 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:17:19.089635 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:19.089558 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e8587ee7-1b21-409a-8c2d-ca9d7b69be4a" path="/var/lib/kubelet/pods/e8587ee7-1b21-409a-8c2d-ca9d7b69be4a/volumes" Oct 31 00:17:19.213362 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:19.213307 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:17:19.683146 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:17:19.683109 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sqv69" containerName="extractor" Oct 31 00:17:21.289769 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:21.289690 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-03c703-587b89bbdd-vzfkm" podUID="b9106a2b-acad-4af9-af60-757055c5d135" containerName="payload-processing" containerID="cri-o://b3f8af41a9ca14245a70502788a0cb1772c90d0d3ebb633769bb736c20ec3d75" gracePeriod=25 Oct 31 00:17:21.399612 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:21.399529 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-03c703-68ccf588f9-957fw" podUID="a7afae12-c562-4a37-adf2-506d8780f93f" containerName="payload-pre-processing" containerID="cri-o://f85af1aee7b46a0f95e7cf09c0f45dccb852bc978228e81c8a84f8915cf3b05e" gracePeriod=25 Oct 31 00:17:21.456949 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:21.456914 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-27e82b/distinct-w2-27e82b-kserve-75f475fcd9-mccsz"] Oct 31 00:17:21.618121 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:21.618085 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-03c703-587b89bbdd-vzfkm" Oct 31 00:17:21.650898 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:21.650815 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:17:21.708186 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:21.708152 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-03c703-68ccf588f9-957fw" Oct 31 00:17:21.743468 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:21.743424 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9106a2b-acad-4af9-af60-757055c5d135-kube-api-access-dhdff\" (UniqueName: \"kubernetes.io/projected/b9106a2b-acad-4af9-af60-757055c5d135-kube-api-access-dhdff\") pod \"b9106a2b-acad-4af9-af60-757055c5d135\" (UID: \"b9106a2b-acad-4af9-af60-757055c5d135\") " Oct 31 00:17:21.743668 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:21.743619 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b9106a2b-acad-4af9-af60-757055c5d135-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9106a2b-acad-4af9-af60-757055c5d135-plugins-config-volume\") pod \"b9106a2b-acad-4af9-af60-757055c5d135\" (UID: \"b9106a2b-acad-4af9-af60-757055c5d135\") " Oct 31 00:17:21.746463 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:21.746433 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9106a2b-acad-4af9-af60-757055c5d135-kube-api-access-dhdff" pod "b9106a2b-acad-4af9-af60-757055c5d135" (UID: "b9106a2b-acad-4af9-af60-757055c5d135"). InnerVolumeSpecName "kube-api-access-dhdff". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:17:21.746661 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:21.746633 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9106a2b-acad-4af9-af60-757055c5d135-plugins-config-volume" pod "b9106a2b-acad-4af9-af60-757055c5d135" (UID: "b9106a2b-acad-4af9-af60-757055c5d135"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:17:21.845043 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:21.845002 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7afae12-c562-4a37-adf2-506d8780f93f-kube-api-access-cmsql\" (UniqueName: \"kubernetes.io/projected/a7afae12-c562-4a37-adf2-506d8780f93f-kube-api-access-cmsql\") pod \"a7afae12-c562-4a37-adf2-506d8780f93f\" (UID: \"a7afae12-c562-4a37-adf2-506d8780f93f\") " Oct 31 00:17:21.845270 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:21.845252 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a7afae12-c562-4a37-adf2-506d8780f93f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7afae12-c562-4a37-adf2-506d8780f93f-plugins-config-volume\") pod \"a7afae12-c562-4a37-adf2-506d8780f93f\" (UID: \"a7afae12-c562-4a37-adf2-506d8780f93f\") " Oct 31 00:17:21.845642 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:21.845619 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7afae12-c562-4a37-adf2-506d8780f93f-plugins-config-volume" pod "a7afae12-c562-4a37-adf2-506d8780f93f" (UID: "a7afae12-c562-4a37-adf2-506d8780f93f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:17:21.845910 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:21.845888 4927 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9106a2b-acad-4af9-af60-757055c5d135-plugins-config-volume\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:17:21.845953 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:21.845917 4927 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7afae12-c562-4a37-adf2-506d8780f93f-plugins-config-volume\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:17:21.845953 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:21.845933 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dhdff\" (UniqueName: \"kubernetes.io/projected/b9106a2b-acad-4af9-af60-757055c5d135-kube-api-access-dhdff\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:17:21.847200 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:21.847176 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7afae12-c562-4a37-adf2-506d8780f93f-kube-api-access-cmsql" pod "a7afae12-c562-4a37-adf2-506d8780f93f" (UID: "a7afae12-c562-4a37-adf2-506d8780f93f"). InnerVolumeSpecName "kube-api-access-cmsql". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:17:21.947307 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:21.947274 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cmsql\" (UniqueName: \"kubernetes.io/projected/a7afae12-c562-4a37-adf2-506d8780f93f-kube-api-access-cmsql\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:17:22.081835 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:22.081801 4927 generic.go:356] "Generic (PLEG): container finished" podID="b9106a2b-acad-4af9-af60-757055c5d135" containerID="b3f8af41a9ca14245a70502788a0cb1772c90d0d3ebb633769bb736c20ec3d75" exitCode=0 Oct 31 00:17:22.082023 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:22.081836 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-03c703-587b89bbdd-vzfkm" Oct 31 00:17:22.082023 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:22.081883 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-03c703-587b89bbdd-vzfkm" event={"ID":"b9106a2b-acad-4af9-af60-757055c5d135","Type":"ContainerDied","Data":"b3f8af41a9ca14245a70502788a0cb1772c90d0d3ebb633769bb736c20ec3d75"} Oct 31 00:17:22.082023 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:22.081929 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-03c703-587b89bbdd-vzfkm" event={"ID":"b9106a2b-acad-4af9-af60-757055c5d135","Type":"ContainerDied","Data":"273fbf1ef2735f96a5606cac324fd3de7024c8e612275e2862eaf0b78b0d5971"} Oct 31 00:17:22.082023 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:22.081956 4927 scope.go:122] "RemoveContainer" containerID="b3f8af41a9ca14245a70502788a0cb1772c90d0d3ebb633769bb736c20ec3d75" Oct 31 00:17:22.084120 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:22.084092 4927 generic.go:356] "Generic (PLEG): container finished" podID="3898ce0c-61e4-4888-9e54-3185c86a5326" containerID="c14ecd91a809659c013aabbf0dd11ba620bb7e763cd825cc1fbdf6357bc6bc56" exitCode=0 Oct 31 00:17:22.084247 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:22.084174 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c9c21e-openshift-default-749bcfc96c-wjzsh" event={"ID":"3898ce0c-61e4-4888-9e54-3185c86a5326","Type":"ContainerDied","Data":"c14ecd91a809659c013aabbf0dd11ba620bb7e763cd825cc1fbdf6357bc6bc56"} Oct 31 00:17:22.084802 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:22.084671 4927 scope.go:122] "RemoveContainer" containerID="c14ecd91a809659c013aabbf0dd11ba620bb7e763cd825cc1fbdf6357bc6bc56" Oct 31 00:17:22.086290 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:22.086266 4927 generic.go:356] "Generic (PLEG): container finished" podID="a7afae12-c562-4a37-adf2-506d8780f93f" containerID="f85af1aee7b46a0f95e7cf09c0f45dccb852bc978228e81c8a84f8915cf3b05e" exitCode=0 Oct 31 00:17:22.086416 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:22.086298 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-03c703-68ccf588f9-957fw" Oct 31 00:17:22.086483 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:22.086449 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-03c703-68ccf588f9-957fw" event={"ID":"a7afae12-c562-4a37-adf2-506d8780f93f","Type":"ContainerDied","Data":"f85af1aee7b46a0f95e7cf09c0f45dccb852bc978228e81c8a84f8915cf3b05e"} Oct 31 00:17:22.086542 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:22.086490 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-03c703-68ccf588f9-957fw" event={"ID":"a7afae12-c562-4a37-adf2-506d8780f93f","Type":"ContainerDied","Data":"4f4d6d409c0326af1d7af95c57711f2a7a20eab8a6389eb2e7302a7f192b5c61"} Oct 31 00:17:22.086796 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:22.086750 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:17:22.086887 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:22.086845 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:17:22.086950 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:22.086900 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:17:22.095282 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:22.095264 4927 scope.go:122] "RemoveContainer" containerID="b3f8af41a9ca14245a70502788a0cb1772c90d0d3ebb633769bb736c20ec3d75" Oct 31 00:17:22.095594 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:17:22.095572 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b3f8af41a9ca14245a70502788a0cb1772c90d0d3ebb633769bb736c20ec3d75\": container with ID starting with b3f8af41a9ca14245a70502788a0cb1772c90d0d3ebb633769bb736c20ec3d75 not found: ID does not exist" containerID="b3f8af41a9ca14245a70502788a0cb1772c90d0d3ebb633769bb736c20ec3d75" Oct 31 00:17:22.095692 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:22.095599 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3f8af41a9ca14245a70502788a0cb1772c90d0d3ebb633769bb736c20ec3d75"} err="failed to get container status \"b3f8af41a9ca14245a70502788a0cb1772c90d0d3ebb633769bb736c20ec3d75\": rpc error: code = NotFound desc = could not find container \"b3f8af41a9ca14245a70502788a0cb1772c90d0d3ebb633769bb736c20ec3d75\": container with ID starting with b3f8af41a9ca14245a70502788a0cb1772c90d0d3ebb633769bb736c20ec3d75 not found: ID does not exist" Oct 31 00:17:22.095692 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:22.095622 4927 scope.go:122] "RemoveContainer" containerID="f85af1aee7b46a0f95e7cf09c0f45dccb852bc978228e81c8a84f8915cf3b05e" Oct 31 00:17:22.109739 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:22.109702 4927 scope.go:122] "RemoveContainer" containerID="f85af1aee7b46a0f95e7cf09c0f45dccb852bc978228e81c8a84f8915cf3b05e" Oct 31 00:17:22.110104 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:17:22.110069 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f85af1aee7b46a0f95e7cf09c0f45dccb852bc978228e81c8a84f8915cf3b05e\": container with ID starting with f85af1aee7b46a0f95e7cf09c0f45dccb852bc978228e81c8a84f8915cf3b05e not found: ID does not exist" containerID="f85af1aee7b46a0f95e7cf09c0f45dccb852bc978228e81c8a84f8915cf3b05e" Oct 31 00:17:22.110194 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:22.110106 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f85af1aee7b46a0f95e7cf09c0f45dccb852bc978228e81c8a84f8915cf3b05e"} err="failed to get container status \"f85af1aee7b46a0f95e7cf09c0f45dccb852bc978228e81c8a84f8915cf3b05e\": rpc error: code = NotFound desc = could not find container \"f85af1aee7b46a0f95e7cf09c0f45dccb852bc978228e81c8a84f8915cf3b05e\": container with ID starting with f85af1aee7b46a0f95e7cf09c0f45dccb852bc978228e81c8a84f8915cf3b05e not found: ID does not exist" Oct 31 00:17:22.261473 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:22.261382 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-03c703-587b89bbdd-vzfkm"] Oct 31 00:17:22.268312 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:22.268280 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-03c703-587b89bbdd-vzfkm"] Oct 31 00:17:22.380980 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:22.380946 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-03c703-68ccf588f9-957fw"] Oct 31 00:17:22.388055 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:22.388023 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-03c703-68ccf588f9-957fw"] Oct 31 00:17:23.090492 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:23.090444 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a7afae12-c562-4a37-adf2-506d8780f93f" path="/var/lib/kubelet/pods/a7afae12-c562-4a37-adf2-506d8780f93f/volumes" Oct 31 00:17:23.090945 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:23.090927 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b9106a2b-acad-4af9-af60-757055c5d135" path="/var/lib/kubelet/pods/b9106a2b-acad-4af9-af60-757055c5d135/volumes" Oct 31 00:17:23.099979 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:23.099952 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c9c21e-openshift-default-749bcfc96c-wjzsh" event={"ID":"3898ce0c-61e4-4888-9e54-3185c86a5326","Type":"ContainerStarted","Data":"47cbb5f450fa0df6d752b192904af4ab0983057b7c61403a5bf24363a3d906dc"} Oct 31 00:17:23.680172 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:23.680132 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-c9c21e-openshift-default-749bcfc96c-wjzsh" Oct 31 00:17:23.681192 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:23.681160 4927 patch_prober.go:29] interesting pod/e2e-api-a-c9c21e-openshift-default-749bcfc96c-wjzsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" start-of-body= Oct 31 00:17:23.681287 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:23.681218 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c9c21e-openshift-default-749bcfc96c-wjzsh" podUID="3898ce0c-61e4-4888-9e54-3185c86a5326" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" Oct 31 00:17:23.736461 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:23.736425 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:17:24.032414 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:24.031418 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-c9c21e-openshift-default-749bcfc96c-wjzsh" Oct 31 00:17:24.679596 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:24.679560 4927 patch_prober.go:29] interesting pod/e2e-api-a-c9c21e-openshift-default-749bcfc96c-wjzsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" start-of-body= Oct 31 00:17:24.679801 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:24.679622 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c9c21e-openshift-default-749bcfc96c-wjzsh" podUID="3898ce0c-61e4-4888-9e54-3185c86a5326" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" Oct 31 00:17:25.545104 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:25.545069 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:17:25.680268 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:25.680209 4927 patch_prober.go:29] interesting pod/e2e-api-a-c9c21e-openshift-default-749bcfc96c-wjzsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" start-of-body= Oct 31 00:17:25.680511 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:25.680288 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c9c21e-openshift-default-749bcfc96c-wjzsh" podUID="3898ce0c-61e4-4888-9e54-3185c86a5326" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" Oct 31 00:17:25.740745 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:25.740659 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:17:25.877614 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:25.877527 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-ab0ac9-openshift-default-54c6546bdf-lxwwq"] Oct 31 00:17:25.877854 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:25.877825 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-ab0ac9-openshift-default-54c6546bdf-lxwwq" podUID="3b669d7f-c767-4c4e-a524-6e774fc69034" containerName="istio-proxy" containerID="cri-o://e8dc89e23fffca41c6652fcb3bfbab07c7fbf700202efaf16e98f640624ae2ba" gracePeriod=30 Oct 31 00:17:25.989352 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:25.987767 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:17:26.680440 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:26.680404 4927 patch_prober.go:29] interesting pod/e2e-api-a-c9c21e-openshift-default-749bcfc96c-wjzsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" start-of-body= Oct 31 00:17:26.680839 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:26.680456 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c9c21e-openshift-default-749bcfc96c-wjzsh" podUID="3898ce0c-61e4-4888-9e54-3185c86a5326" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" Oct 31 00:17:27.679700 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:27.679663 4927 patch_prober.go:29] interesting pod/e2e-api-a-c9c21e-openshift-default-749bcfc96c-wjzsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" start-of-body= Oct 31 00:17:27.679877 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:27.679718 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c9c21e-openshift-default-749bcfc96c-wjzsh" podUID="3898ce0c-61e4-4888-9e54-3185c86a5326" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" Oct 31 00:17:28.685109 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:28.685079 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-c9c21e-openshift-default-749bcfc96c-wjzsh" Oct 31 00:17:28.686174 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:28.686151 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-c9c21e-openshift-default-749bcfc96c-wjzsh" Oct 31 00:17:28.797798 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:28.797753 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:17:29.706035 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:29.705999 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-27e82b-64f94b9586-wb68f"] Oct 31 00:17:29.817662 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:29.817622 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-27e82b-5968f9f875-t9qz2"] Oct 31 00:17:30.069320 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:30.069226 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:17:30.281796 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:30.281739 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:17:30.494290 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:30.494254 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:17:31.190432 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:31.190407 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-ab0ac9-openshift-default-54c6546bdf-lxwwq" Oct 31 00:17:31.197795 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:31.196323 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b669d7f-c767-4c4e-a524-6e774fc69034-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b669d7f-c767-4c4e-a524-6e774fc69034-istio-data\") pod \"3b669d7f-c767-4c4e-a524-6e774fc69034\" (UID: \"3b669d7f-c767-4c4e-a524-6e774fc69034\") " Oct 31 00:17:31.197795 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:31.196424 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b669d7f-c767-4c4e-a524-6e774fc69034-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b669d7f-c767-4c4e-a524-6e774fc69034-istio-envoy\") pod \"3b669d7f-c767-4c4e-a524-6e774fc69034\" (UID: \"3b669d7f-c767-4c4e-a524-6e774fc69034\") " Oct 31 00:17:31.197795 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:31.196496 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b669d7f-c767-4c4e-a524-6e774fc69034-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b669d7f-c767-4c4e-a524-6e774fc69034-istiod-ca-cert\") pod \"3b669d7f-c767-4c4e-a524-6e774fc69034\" (UID: \"3b669d7f-c767-4c4e-a524-6e774fc69034\") " Oct 31 00:17:31.197795 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:31.196538 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3b669d7f-c767-4c4e-a524-6e774fc69034-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b669d7f-c767-4c4e-a524-6e774fc69034-istio-podinfo\") pod \"3b669d7f-c767-4c4e-a524-6e774fc69034\" (UID: \"3b669d7f-c767-4c4e-a524-6e774fc69034\") " Oct 31 00:17:31.197795 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:31.196574 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b669d7f-c767-4c4e-a524-6e774fc69034-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b669d7f-c767-4c4e-a524-6e774fc69034-credential-socket\") pod \"3b669d7f-c767-4c4e-a524-6e774fc69034\" (UID: \"3b669d7f-c767-4c4e-a524-6e774fc69034\") " Oct 31 00:17:31.197795 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:31.196624 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b669d7f-c767-4c4e-a524-6e774fc69034-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b669d7f-c767-4c4e-a524-6e774fc69034-workload-socket\") pod \"3b669d7f-c767-4c4e-a524-6e774fc69034\" (UID: \"3b669d7f-c767-4c4e-a524-6e774fc69034\") " Oct 31 00:17:31.197795 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:31.196741 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b669d7f-c767-4c4e-a524-6e774fc69034-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b669d7f-c767-4c4e-a524-6e774fc69034-workload-certs\") pod \"3b669d7f-c767-4c4e-a524-6e774fc69034\" (UID: \"3b669d7f-c767-4c4e-a524-6e774fc69034\") " Oct 31 00:17:31.197795 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:31.196840 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b669d7f-c767-4c4e-a524-6e774fc69034-kube-api-access-smzd7\" (UniqueName: \"kubernetes.io/projected/3b669d7f-c767-4c4e-a524-6e774fc69034-kube-api-access-smzd7\") pod \"3b669d7f-c767-4c4e-a524-6e774fc69034\" (UID: \"3b669d7f-c767-4c4e-a524-6e774fc69034\") " Oct 31 00:17:31.197795 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:31.196898 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b669d7f-c767-4c4e-a524-6e774fc69034-istio-token\" (UniqueName: \"kubernetes.io/projected/3b669d7f-c767-4c4e-a524-6e774fc69034-istio-token\") pod \"3b669d7f-c767-4c4e-a524-6e774fc69034\" (UID: \"3b669d7f-c767-4c4e-a524-6e774fc69034\") " Oct 31 00:17:31.208922 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:31.200793 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b669d7f-c767-4c4e-a524-6e774fc69034-istio-token" pod "3b669d7f-c767-4c4e-a524-6e774fc69034" (UID: "3b669d7f-c767-4c4e-a524-6e774fc69034"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:17:31.208922 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:31.200807 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b669d7f-c767-4c4e-a524-6e774fc69034-istio-data" pod "3b669d7f-c767-4c4e-a524-6e774fc69034" (UID: "3b669d7f-c767-4c4e-a524-6e774fc69034"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:17:31.208922 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:31.201031 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b669d7f-c767-4c4e-a524-6e774fc69034-credential-socket" pod "3b669d7f-c767-4c4e-a524-6e774fc69034" (UID: "3b669d7f-c767-4c4e-a524-6e774fc69034"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:17:31.208922 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:31.201050 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b669d7f-c767-4c4e-a524-6e774fc69034-workload-certs" pod "3b669d7f-c767-4c4e-a524-6e774fc69034" (UID: "3b669d7f-c767-4c4e-a524-6e774fc69034"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:17:31.208922 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:31.201283 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b669d7f-c767-4c4e-a524-6e774fc69034-workload-socket" pod "3b669d7f-c767-4c4e-a524-6e774fc69034" (UID: "3b669d7f-c767-4c4e-a524-6e774fc69034"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:17:31.208922 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:31.201371 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b669d7f-c767-4c4e-a524-6e774fc69034-istiod-ca-cert" pod "3b669d7f-c767-4c4e-a524-6e774fc69034" (UID: "3b669d7f-c767-4c4e-a524-6e774fc69034"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:17:31.208922 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:31.203266 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b669d7f-c767-4c4e-a524-6e774fc69034-istio-envoy" pod "3b669d7f-c767-4c4e-a524-6e774fc69034" (UID: "3b669d7f-c767-4c4e-a524-6e774fc69034"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:17:31.211296 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:31.211265 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b669d7f-c767-4c4e-a524-6e774fc69034-kube-api-access-smzd7" pod "3b669d7f-c767-4c4e-a524-6e774fc69034" (UID: "3b669d7f-c767-4c4e-a524-6e774fc69034"). InnerVolumeSpecName "kube-api-access-smzd7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:17:31.212137 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:31.212108 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3b669d7f-c767-4c4e-a524-6e774fc69034-istio-podinfo" pod "3b669d7f-c767-4c4e-a524-6e774fc69034" (UID: "3b669d7f-c767-4c4e-a524-6e774fc69034"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:17:31.221506 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:31.221474 4927 generic.go:356] "Generic (PLEG): container finished" podID="3b669d7f-c767-4c4e-a524-6e774fc69034" containerID="e8dc89e23fffca41c6652fcb3bfbab07c7fbf700202efaf16e98f640624ae2ba" exitCode=0 Oct 31 00:17:31.221615 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:31.221544 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-ab0ac9-openshift-default-54c6546bdf-lxwwq" Oct 31 00:17:31.221615 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:31.221555 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-ab0ac9-openshift-default-54c6546bdf-lxwwq" event={"ID":"3b669d7f-c767-4c4e-a524-6e774fc69034","Type":"ContainerDied","Data":"e8dc89e23fffca41c6652fcb3bfbab07c7fbf700202efaf16e98f640624ae2ba"} Oct 31 00:17:31.221748 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:31.221650 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-ab0ac9-openshift-default-54c6546bdf-lxwwq" event={"ID":"3b669d7f-c767-4c4e-a524-6e774fc69034","Type":"ContainerDied","Data":"2b98a1f2be9d1217e59058d87a39b60061352fdea6585f2400892a4fedcf1da5"} Oct 31 00:17:31.221748 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:31.221682 4927 scope.go:122] "RemoveContainer" containerID="e8dc89e23fffca41c6652fcb3bfbab07c7fbf700202efaf16e98f640624ae2ba" Oct 31 00:17:31.239712 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:31.239689 4927 scope.go:122] "RemoveContainer" containerID="e2cdeea856504f4f745d34fae7ed87897794b11dec004edbaf555fde8f4bb24e" Oct 31 00:17:31.254648 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:31.254626 4927 scope.go:122] "RemoveContainer" containerID="e8dc89e23fffca41c6652fcb3bfbab07c7fbf700202efaf16e98f640624ae2ba" Oct 31 00:17:31.255077 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:17:31.255056 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8dc89e23fffca41c6652fcb3bfbab07c7fbf700202efaf16e98f640624ae2ba\": container with ID starting with e8dc89e23fffca41c6652fcb3bfbab07c7fbf700202efaf16e98f640624ae2ba not found: ID does not exist" containerID="e8dc89e23fffca41c6652fcb3bfbab07c7fbf700202efaf16e98f640624ae2ba" Oct 31 00:17:31.255154 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:31.255086 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8dc89e23fffca41c6652fcb3bfbab07c7fbf700202efaf16e98f640624ae2ba"} err="failed to get container status \"e8dc89e23fffca41c6652fcb3bfbab07c7fbf700202efaf16e98f640624ae2ba\": rpc error: code = NotFound desc = could not find container \"e8dc89e23fffca41c6652fcb3bfbab07c7fbf700202efaf16e98f640624ae2ba\": container with ID starting with e8dc89e23fffca41c6652fcb3bfbab07c7fbf700202efaf16e98f640624ae2ba not found: ID does not exist" Oct 31 00:17:31.255154 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:31.255108 4927 scope.go:122] "RemoveContainer" containerID="e2cdeea856504f4f745d34fae7ed87897794b11dec004edbaf555fde8f4bb24e" Oct 31 00:17:31.255441 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:17:31.255421 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2cdeea856504f4f745d34fae7ed87897794b11dec004edbaf555fde8f4bb24e\": container with ID starting with e2cdeea856504f4f745d34fae7ed87897794b11dec004edbaf555fde8f4bb24e not found: ID does not exist" containerID="e2cdeea856504f4f745d34fae7ed87897794b11dec004edbaf555fde8f4bb24e" Oct 31 00:17:31.255541 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:31.255448 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2cdeea856504f4f745d34fae7ed87897794b11dec004edbaf555fde8f4bb24e"} err="failed to get container status \"e2cdeea856504f4f745d34fae7ed87897794b11dec004edbaf555fde8f4bb24e\": rpc error: code = NotFound desc = could not find container \"e2cdeea856504f4f745d34fae7ed87897794b11dec004edbaf555fde8f4bb24e\": container with ID starting with e2cdeea856504f4f745d34fae7ed87897794b11dec004edbaf555fde8f4bb24e not found: ID does not exist" Oct 31 00:17:31.299623 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:31.299587 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-smzd7\" (UniqueName: \"kubernetes.io/projected/3b669d7f-c767-4c4e-a524-6e774fc69034-kube-api-access-smzd7\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:17:31.299623 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:31.299619 4927 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b669d7f-c767-4c4e-a524-6e774fc69034-istio-token\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:17:31.299623 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:31.299629 4927 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b669d7f-c767-4c4e-a524-6e774fc69034-istio-data\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:17:31.299831 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:31.299637 4927 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b669d7f-c767-4c4e-a524-6e774fc69034-istio-envoy\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:17:31.299831 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:31.299646 4927 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b669d7f-c767-4c4e-a524-6e774fc69034-istiod-ca-cert\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:17:31.299831 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:31.299656 4927 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b669d7f-c767-4c4e-a524-6e774fc69034-istio-podinfo\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:17:31.299831 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:31.299667 4927 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b669d7f-c767-4c4e-a524-6e774fc69034-credential-socket\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:17:31.299831 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:31.299680 4927 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b669d7f-c767-4c4e-a524-6e774fc69034-workload-socket\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:17:31.299831 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:31.299689 4927 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b669d7f-c767-4c4e-a524-6e774fc69034-workload-certs\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:17:31.585654 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:31.563751 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-ab0ac9-openshift-default-54c6546bdf-lxwwq"] Oct 31 00:17:31.593237 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:31.592180 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-ab0ac9-openshift-default-54c6546bdf-lxwwq"] Oct 31 00:17:32.063707 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:32.063661 4927 patch_prober.go:29] interesting pod/e2e-shared-a-w3-ab0ac9-openshift-default-54c6546bdf-lxwwq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.197:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:17:32.063895 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:32.063734 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w3-ab0ac9-openshift-default-54c6546bdf-lxwwq" podUID="3b669d7f-c767-4c4e-a524-6e774fc69034" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.197:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:17:33.087637 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:33.087561 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b669d7f-c767-4c4e-a524-6e774fc69034" path="/var/lib/kubelet/pods/3b669d7f-c767-4c4e-a524-6e774fc69034/volumes" Oct 31 00:17:33.701443 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:33.701401 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:17:34.706759 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:34.706713 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w2-27e82b-64f94b9586-wb68f" podUID="59ccea66-1d77-43cd-bd56-e88b44a5e621" containerName="payload-processing" containerID="cri-o://a8eef16879d922846f405a93c664865feabb85ecddcfb489be76f82b84fef7c9" gracePeriod=25 Oct 31 00:17:34.818117 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:34.818069 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-27e82b-5968f9f875-t9qz2" podUID="687684ad-5b0d-493a-9fa4-cbbe640229fe" containerName="payload-pre-processing" containerID="cri-o://97af6b4131cff76a3d0bf2dd637d64c4132121feb08d0a1acf9a6e4ea6429ee6" gracePeriod=25 Oct 31 00:17:36.479999 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:36.479929 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-27e82b/distinct-w2-27e82b-kserve-75f475fcd9-mccsz" podUID="4ba8ced4-e93f-46ac-87c5-3d0b7fadc02d" containerName="main" containerID="cri-o://823eae6a03e711e6eef4109ae26cf244ea9989cfd22ead0df68af2c8e41e5076" gracePeriod=45 Oct 31 00:17:36.778880 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:36.778848 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-27e82b/distinct-w2-27e82b-kserve-75f475fcd9-mccsz" Oct 31 00:17:36.819072 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:36.819040 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ba8ced4-e93f-46ac-87c5-3d0b7fadc02d-home\" (UniqueName: \"kubernetes.io/empty-dir/4ba8ced4-e93f-46ac-87c5-3d0b7fadc02d-home\") pod \"4ba8ced4-e93f-46ac-87c5-3d0b7fadc02d\" (UID: \"4ba8ced4-e93f-46ac-87c5-3d0b7fadc02d\") " Oct 31 00:17:36.819223 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:36.819174 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ba8ced4-e93f-46ac-87c5-3d0b7fadc02d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ba8ced4-e93f-46ac-87c5-3d0b7fadc02d-model-cache\") pod \"4ba8ced4-e93f-46ac-87c5-3d0b7fadc02d\" (UID: \"4ba8ced4-e93f-46ac-87c5-3d0b7fadc02d\") " Oct 31 00:17:36.819311 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:36.819251 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ba8ced4-e93f-46ac-87c5-3d0b7fadc02d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ba8ced4-e93f-46ac-87c5-3d0b7fadc02d-dshm\") pod \"4ba8ced4-e93f-46ac-87c5-3d0b7fadc02d\" (UID: \"4ba8ced4-e93f-46ac-87c5-3d0b7fadc02d\") " Oct 31 00:17:36.819401 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:36.819314 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4ba8ced4-e93f-46ac-87c5-3d0b7fadc02d-tls-certs\" (UniqueName: \"kubernetes.io/secret/4ba8ced4-e93f-46ac-87c5-3d0b7fadc02d-tls-certs\") pod \"4ba8ced4-e93f-46ac-87c5-3d0b7fadc02d\" (UID: \"4ba8ced4-e93f-46ac-87c5-3d0b7fadc02d\") " Oct 31 00:17:36.819401 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:36.819378 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ba8ced4-e93f-46ac-87c5-3d0b7fadc02d-kube-api-access-hg497\" (UniqueName: \"kubernetes.io/projected/4ba8ced4-e93f-46ac-87c5-3d0b7fadc02d-kube-api-access-hg497\") pod \"4ba8ced4-e93f-46ac-87c5-3d0b7fadc02d\" (UID: \"4ba8ced4-e93f-46ac-87c5-3d0b7fadc02d\") " Oct 31 00:17:36.819401 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:36.819376 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ba8ced4-e93f-46ac-87c5-3d0b7fadc02d-home" pod "4ba8ced4-e93f-46ac-87c5-3d0b7fadc02d" (UID: "4ba8ced4-e93f-46ac-87c5-3d0b7fadc02d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:17:36.819563 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:36.819459 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ba8ced4-e93f-46ac-87c5-3d0b7fadc02d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ba8ced4-e93f-46ac-87c5-3d0b7fadc02d-tmp-dir\") pod \"4ba8ced4-e93f-46ac-87c5-3d0b7fadc02d\" (UID: \"4ba8ced4-e93f-46ac-87c5-3d0b7fadc02d\") " Oct 31 00:17:36.819563 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:36.819486 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ba8ced4-e93f-46ac-87c5-3d0b7fadc02d-model-cache" pod "4ba8ced4-e93f-46ac-87c5-3d0b7fadc02d" (UID: "4ba8ced4-e93f-46ac-87c5-3d0b7fadc02d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:17:36.819785 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:36.819757 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ba8ced4-e93f-46ac-87c5-3d0b7fadc02d-tmp-dir" pod "4ba8ced4-e93f-46ac-87c5-3d0b7fadc02d" (UID: "4ba8ced4-e93f-46ac-87c5-3d0b7fadc02d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:17:36.820172 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:36.820152 4927 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ba8ced4-e93f-46ac-87c5-3d0b7fadc02d-tmp-dir\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:17:36.820270 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:36.820174 4927 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ba8ced4-e93f-46ac-87c5-3d0b7fadc02d-home\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:17:36.820270 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:36.820187 4927 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ba8ced4-e93f-46ac-87c5-3d0b7fadc02d-model-cache\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:17:36.822116 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:36.822082 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ba8ced4-e93f-46ac-87c5-3d0b7fadc02d-dshm" pod "4ba8ced4-e93f-46ac-87c5-3d0b7fadc02d" (UID: "4ba8ced4-e93f-46ac-87c5-3d0b7fadc02d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:17:36.822239 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:36.822183 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ba8ced4-e93f-46ac-87c5-3d0b7fadc02d-kube-api-access-hg497" pod "4ba8ced4-e93f-46ac-87c5-3d0b7fadc02d" (UID: "4ba8ced4-e93f-46ac-87c5-3d0b7fadc02d"). InnerVolumeSpecName "kube-api-access-hg497". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:17:36.822239 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:36.822184 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ba8ced4-e93f-46ac-87c5-3d0b7fadc02d-tls-certs" pod "4ba8ced4-e93f-46ac-87c5-3d0b7fadc02d" (UID: "4ba8ced4-e93f-46ac-87c5-3d0b7fadc02d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:17:36.921517 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:36.921484 4927 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ba8ced4-e93f-46ac-87c5-3d0b7fadc02d-dshm\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:17:36.921517 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:36.921514 4927 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ba8ced4-e93f-46ac-87c5-3d0b7fadc02d-tls-certs\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:17:36.921517 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:36.921523 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hg497\" (UniqueName: \"kubernetes.io/projected/4ba8ced4-e93f-46ac-87c5-3d0b7fadc02d-kube-api-access-hg497\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:17:37.242573 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:37.242535 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:17:37.283844 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:37.283807 4927 generic.go:356] "Generic (PLEG): container finished" podID="4ba8ced4-e93f-46ac-87c5-3d0b7fadc02d" containerID="823eae6a03e711e6eef4109ae26cf244ea9989cfd22ead0df68af2c8e41e5076" exitCode=0 Oct 31 00:17:37.284075 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:37.283864 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-27e82b/distinct-w2-27e82b-kserve-75f475fcd9-mccsz" event={"ID":"4ba8ced4-e93f-46ac-87c5-3d0b7fadc02d","Type":"ContainerDied","Data":"823eae6a03e711e6eef4109ae26cf244ea9989cfd22ead0df68af2c8e41e5076"} Oct 31 00:17:37.284075 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:37.283901 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-27e82b/distinct-w2-27e82b-kserve-75f475fcd9-mccsz" event={"ID":"4ba8ced4-e93f-46ac-87c5-3d0b7fadc02d","Type":"ContainerDied","Data":"db62b5e71af88671d8e269905bd21aae59d4a08e06fd8d900c901628effa1bf7"} Oct 31 00:17:37.284075 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:37.283923 4927 scope.go:122] "RemoveContainer" containerID="823eae6a03e711e6eef4109ae26cf244ea9989cfd22ead0df68af2c8e41e5076" Oct 31 00:17:37.284075 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:37.284064 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-27e82b/distinct-w2-27e82b-kserve-75f475fcd9-mccsz" Oct 31 00:17:37.301367 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:37.301322 4927 scope.go:122] "RemoveContainer" containerID="823eae6a03e711e6eef4109ae26cf244ea9989cfd22ead0df68af2c8e41e5076" Oct 31 00:17:37.301636 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:17:37.301615 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"823eae6a03e711e6eef4109ae26cf244ea9989cfd22ead0df68af2c8e41e5076\": container with ID starting with 823eae6a03e711e6eef4109ae26cf244ea9989cfd22ead0df68af2c8e41e5076 not found: ID does not exist" containerID="823eae6a03e711e6eef4109ae26cf244ea9989cfd22ead0df68af2c8e41e5076" Oct 31 00:17:37.301719 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:37.301643 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"823eae6a03e711e6eef4109ae26cf244ea9989cfd22ead0df68af2c8e41e5076"} err="failed to get container status \"823eae6a03e711e6eef4109ae26cf244ea9989cfd22ead0df68af2c8e41e5076\": rpc error: code = NotFound desc = could not find container \"823eae6a03e711e6eef4109ae26cf244ea9989cfd22ead0df68af2c8e41e5076\": container with ID starting with 823eae6a03e711e6eef4109ae26cf244ea9989cfd22ead0df68af2c8e41e5076 not found: ID does not exist" Oct 31 00:17:37.393979 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:37.393883 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:17:37.501359 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:37.492908 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-27e82b/distinct-w2-27e82b-kserve-75f475fcd9-mccsz"] Oct 31 00:17:37.501359 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:37.501190 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-27e82b/distinct-w2-27e82b-kserve-75f475fcd9-mccsz"] Oct 31 00:17:38.582033 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:38.581993 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:17:39.088327 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:39.088292 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ba8ced4-e93f-46ac-87c5-3d0b7fadc02d" path="/var/lib/kubelet/pods/4ba8ced4-e93f-46ac-87c5-3d0b7fadc02d/volumes" Oct 31 00:17:39.205025 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:39.204995 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-d9e782-67987686-hwzmf"] Oct 31 00:17:39.597205 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:39.597167 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:17:39.809806 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:39.809777 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:17:40.918491 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:40.918441 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-27e82b-5968f9f875-t9qz2" podUID="687684ad-5b0d-493a-9fa4-cbbe640229fe" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 00:17:41.887281 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:41.887243 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-27e82b-64f94b9586-wb68f" podUID="59ccea66-1d77-43cd-bd56-e88b44a5e621" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 00:17:42.519978 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:42.519942 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-c9c21e-68d69b8994-hnwg4"] Oct 31 00:17:42.520511 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:42.520208 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c9c21e-68d69b8994-hnwg4" podUID="dbec9857-9469-47ff-8977-0f912e5d0a9c" containerName="maas-api" containerID="cri-o://940cdd8be4499fe8a28e46a34be6582ec41c7eaa033c5b80b90ac40c8e3ad083" gracePeriod=30 Oct 31 00:17:42.618877 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:42.618835 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:17:43.042424 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:43.040994 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c9c21e-68d69b8994-hnwg4" Oct 31 00:17:43.144462 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:43.144428 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dbec9857-9469-47ff-8977-0f912e5d0a9c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dbec9857-9469-47ff-8977-0f912e5d0a9c-maas-api-tls\") pod \"dbec9857-9469-47ff-8977-0f912e5d0a9c\" (UID: \"dbec9857-9469-47ff-8977-0f912e5d0a9c\") " Oct 31 00:17:43.144684 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:43.144664 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dbec9857-9469-47ff-8977-0f912e5d0a9c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/dbec9857-9469-47ff-8977-0f912e5d0a9c-metrics-certs\") pod \"dbec9857-9469-47ff-8977-0f912e5d0a9c\" (UID: \"dbec9857-9469-47ff-8977-0f912e5d0a9c\") " Oct 31 00:17:43.144757 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:43.144722 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dbec9857-9469-47ff-8977-0f912e5d0a9c-kube-api-access-7k79d\" (UniqueName: \"kubernetes.io/projected/dbec9857-9469-47ff-8977-0f912e5d0a9c-kube-api-access-7k79d\") pod \"dbec9857-9469-47ff-8977-0f912e5d0a9c\" (UID: \"dbec9857-9469-47ff-8977-0f912e5d0a9c\") " Oct 31 00:17:43.144817 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:43.144799 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dbec9857-9469-47ff-8977-0f912e5d0a9c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dbec9857-9469-47ff-8977-0f912e5d0a9c-openshift-service-ca\") pod \"dbec9857-9469-47ff-8977-0f912e5d0a9c\" (UID: \"dbec9857-9469-47ff-8977-0f912e5d0a9c\") " Oct 31 00:17:43.145203 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:43.145165 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dbec9857-9469-47ff-8977-0f912e5d0a9c-openshift-service-ca" pod "dbec9857-9469-47ff-8977-0f912e5d0a9c" (UID: "dbec9857-9469-47ff-8977-0f912e5d0a9c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:17:43.145774 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:43.145546 4927 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dbec9857-9469-47ff-8977-0f912e5d0a9c-openshift-service-ca\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:17:43.147200 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:43.147178 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dbec9857-9469-47ff-8977-0f912e5d0a9c-maas-api-tls" pod "dbec9857-9469-47ff-8977-0f912e5d0a9c" (UID: "dbec9857-9469-47ff-8977-0f912e5d0a9c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:17:43.147404 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:43.147383 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbec9857-9469-47ff-8977-0f912e5d0a9c-kube-api-access-7k79d" pod "dbec9857-9469-47ff-8977-0f912e5d0a9c" (UID: "dbec9857-9469-47ff-8977-0f912e5d0a9c"). InnerVolumeSpecName "kube-api-access-7k79d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:17:43.148444 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:43.148421 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dbec9857-9469-47ff-8977-0f912e5d0a9c-metrics-certs" pod "dbec9857-9469-47ff-8977-0f912e5d0a9c" (UID: "dbec9857-9469-47ff-8977-0f912e5d0a9c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:17:43.246624 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:43.246540 4927 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dbec9857-9469-47ff-8977-0f912e5d0a9c-metrics-certs\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:17:43.246624 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:43.246572 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7k79d\" (UniqueName: \"kubernetes.io/projected/dbec9857-9469-47ff-8977-0f912e5d0a9c-kube-api-access-7k79d\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:17:43.246624 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:43.246583 4927 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dbec9857-9469-47ff-8977-0f912e5d0a9c-maas-api-tls\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:17:43.551083 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:43.550996 4927 generic.go:356] "Generic (PLEG): container finished" podID="dbec9857-9469-47ff-8977-0f912e5d0a9c" containerID="940cdd8be4499fe8a28e46a34be6582ec41c7eaa033c5b80b90ac40c8e3ad083" exitCode=0 Oct 31 00:17:43.551083 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:43.551035 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c9c21e-68d69b8994-hnwg4" Oct 31 00:17:43.551083 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:43.551071 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c9c21e-68d69b8994-hnwg4" event={"ID":"dbec9857-9469-47ff-8977-0f912e5d0a9c","Type":"ContainerDied","Data":"940cdd8be4499fe8a28e46a34be6582ec41c7eaa033c5b80b90ac40c8e3ad083"} Oct 31 00:17:43.551611 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:43.551123 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c9c21e-68d69b8994-hnwg4" event={"ID":"dbec9857-9469-47ff-8977-0f912e5d0a9c","Type":"ContainerDied","Data":"69ff9641ddad1615b0bc1b55d6d9b1f82fdd0179e092830ec55f4b7cb3adc74e"} Oct 31 00:17:43.551611 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:43.551154 4927 scope.go:122] "RemoveContainer" containerID="940cdd8be4499fe8a28e46a34be6582ec41c7eaa033c5b80b90ac40c8e3ad083" Oct 31 00:17:43.563438 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:43.563412 4927 scope.go:122] "RemoveContainer" containerID="940cdd8be4499fe8a28e46a34be6582ec41c7eaa033c5b80b90ac40c8e3ad083" Oct 31 00:17:43.563720 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:17:43.563699 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"940cdd8be4499fe8a28e46a34be6582ec41c7eaa033c5b80b90ac40c8e3ad083\": container with ID starting with 940cdd8be4499fe8a28e46a34be6582ec41c7eaa033c5b80b90ac40c8e3ad083 not found: ID does not exist" containerID="940cdd8be4499fe8a28e46a34be6582ec41c7eaa033c5b80b90ac40c8e3ad083" Oct 31 00:17:43.563812 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:43.563727 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"940cdd8be4499fe8a28e46a34be6582ec41c7eaa033c5b80b90ac40c8e3ad083"} err="failed to get container status \"940cdd8be4499fe8a28e46a34be6582ec41c7eaa033c5b80b90ac40c8e3ad083\": rpc error: code = NotFound desc = could not find container \"940cdd8be4499fe8a28e46a34be6582ec41c7eaa033c5b80b90ac40c8e3ad083\": container with ID starting with 940cdd8be4499fe8a28e46a34be6582ec41c7eaa033c5b80b90ac40c8e3ad083 not found: ID does not exist" Oct 31 00:17:43.725123 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:43.725089 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-c9c21e-68d69b8994-hnwg4"] Oct 31 00:17:43.731640 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:43.731611 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-c9c21e-68d69b8994-hnwg4"] Oct 31 00:17:44.049590 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:44.049554 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:17:44.205347 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:44.205296 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-d9e782-67987686-hwzmf" podUID="b45aa3c1-e11b-4d72-b93c-2248b7916834" containerName="payload-processing" containerID="cri-o://61958775a787dc49f2a0ae9730b259296d1852329d78b4640794111c16ae5d86" gracePeriod=25 Oct 31 00:17:44.275138 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:44.275100 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:17:44.462702 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:44.462665 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:17:44.551579 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:44.551552 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-d9e782-67987686-hwzmf" Oct 31 00:17:44.561252 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:44.561219 4927 generic.go:356] "Generic (PLEG): container finished" podID="b45aa3c1-e11b-4d72-b93c-2248b7916834" containerID="61958775a787dc49f2a0ae9730b259296d1852329d78b4640794111c16ae5d86" exitCode=0 Oct 31 00:17:44.561422 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:44.561269 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-d9e782-67987686-hwzmf" Oct 31 00:17:44.561422 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:44.561296 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-d9e782-67987686-hwzmf" event={"ID":"b45aa3c1-e11b-4d72-b93c-2248b7916834","Type":"ContainerDied","Data":"61958775a787dc49f2a0ae9730b259296d1852329d78b4640794111c16ae5d86"} Oct 31 00:17:44.561422 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:44.561363 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-d9e782-67987686-hwzmf" event={"ID":"b45aa3c1-e11b-4d72-b93c-2248b7916834","Type":"ContainerDied","Data":"90c2c0e0dd845d2c1501a23ec861d40f5bf8f049804d2fb60c886145da412826"} Oct 31 00:17:44.561422 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:44.561388 4927 scope.go:122] "RemoveContainer" containerID="61958775a787dc49f2a0ae9730b259296d1852329d78b4640794111c16ae5d86" Oct 31 00:17:44.565441 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:44.565419 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b45aa3c1-e11b-4d72-b93c-2248b7916834-kube-api-access-8j9lw\" (UniqueName: \"kubernetes.io/projected/b45aa3c1-e11b-4d72-b93c-2248b7916834-kube-api-access-8j9lw\") pod \"b45aa3c1-e11b-4d72-b93c-2248b7916834\" (UID: \"b45aa3c1-e11b-4d72-b93c-2248b7916834\") " Oct 31 00:17:44.567784 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:44.567755 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b45aa3c1-e11b-4d72-b93c-2248b7916834-kube-api-access-8j9lw" pod "b45aa3c1-e11b-4d72-b93c-2248b7916834" (UID: "b45aa3c1-e11b-4d72-b93c-2248b7916834"). InnerVolumeSpecName "kube-api-access-8j9lw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:17:44.573909 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:44.573887 4927 scope.go:122] "RemoveContainer" containerID="61958775a787dc49f2a0ae9730b259296d1852329d78b4640794111c16ae5d86" Oct 31 00:17:44.574232 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:17:44.574183 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61958775a787dc49f2a0ae9730b259296d1852329d78b4640794111c16ae5d86\": container with ID starting with 61958775a787dc49f2a0ae9730b259296d1852329d78b4640794111c16ae5d86 not found: ID does not exist" containerID="61958775a787dc49f2a0ae9730b259296d1852329d78b4640794111c16ae5d86" Oct 31 00:17:44.574232 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:44.574210 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61958775a787dc49f2a0ae9730b259296d1852329d78b4640794111c16ae5d86"} err="failed to get container status \"61958775a787dc49f2a0ae9730b259296d1852329d78b4640794111c16ae5d86\": rpc error: code = NotFound desc = could not find container \"61958775a787dc49f2a0ae9730b259296d1852329d78b4640794111c16ae5d86\": container with ID starting with 61958775a787dc49f2a0ae9730b259296d1852329d78b4640794111c16ae5d86 not found: ID does not exist" Oct 31 00:17:44.666625 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:44.666594 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b45aa3c1-e11b-4d72-b93c-2248b7916834-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b45aa3c1-e11b-4d72-b93c-2248b7916834-plugins-config-volume\") pod \"b45aa3c1-e11b-4d72-b93c-2248b7916834\" (UID: \"b45aa3c1-e11b-4d72-b93c-2248b7916834\") " Oct 31 00:17:44.667046 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:44.667018 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b45aa3c1-e11b-4d72-b93c-2248b7916834-plugins-config-volume" pod "b45aa3c1-e11b-4d72-b93c-2248b7916834" (UID: "b45aa3c1-e11b-4d72-b93c-2248b7916834"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:17:44.667500 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:44.667474 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8j9lw\" (UniqueName: \"kubernetes.io/projected/b45aa3c1-e11b-4d72-b93c-2248b7916834-kube-api-access-8j9lw\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:17:44.667626 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:44.667504 4927 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b45aa3c1-e11b-4d72-b93c-2248b7916834-plugins-config-volume\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:17:44.975029 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:44.974024 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:17:44.986306 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:44.986272 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-d9e782-67987686-hwzmf"] Oct 31 00:17:44.990912 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:44.990882 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-d9e782-67987686-hwzmf"] Oct 31 00:17:45.088675 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:45.088636 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b45aa3c1-e11b-4d72-b93c-2248b7916834" path="/var/lib/kubelet/pods/b45aa3c1-e11b-4d72-b93c-2248b7916834/volumes" Oct 31 00:17:45.089107 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:45.089092 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dbec9857-9469-47ff-8977-0f912e5d0a9c" path="/var/lib/kubelet/pods/dbec9857-9469-47ff-8977-0f912e5d0a9c/volumes" Oct 31 00:17:48.422644 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:48.422608 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-4a3dba-744d479576-fx5nx"] Oct 31 00:17:48.423129 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:48.422822 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-4a3dba-744d479576-fx5nx" podUID="f0295476-7278-4b04-af13-d09dafe0b5b6" containerName="maas-api" containerID="cri-o://2d8ef2f89979a6d233b1d612e90d708121b6b44568cc97de08635e8ea03d4674" gracePeriod=30 Oct 31 00:17:48.488788 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:48.488738 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-4a3dba-744d479576-fx5nx" podUID="f0295476-7278-4b04-af13-d09dafe0b5b6" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.216:8443/health\": dial tcp 10.132.0.216:8443: connect: connection refused" Oct 31 00:17:48.571954 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:48.571912 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:17:48.598653 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:48.598621 4927 generic.go:356] "Generic (PLEG): container finished" podID="f0295476-7278-4b04-af13-d09dafe0b5b6" containerID="2d8ef2f89979a6d233b1d612e90d708121b6b44568cc97de08635e8ea03d4674" exitCode=0 Oct 31 00:17:48.598818 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:48.598704 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-4a3dba-744d479576-fx5nx" event={"ID":"f0295476-7278-4b04-af13-d09dafe0b5b6","Type":"ContainerDied","Data":"2d8ef2f89979a6d233b1d612e90d708121b6b44568cc97de08635e8ea03d4674"} Oct 31 00:17:48.917526 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:48.909839 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-4a3dba-744d479576-fx5nx" Oct 31 00:17:48.955389 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:48.952753 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f0295476-7278-4b04-af13-d09dafe0b5b6-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f0295476-7278-4b04-af13-d09dafe0b5b6-maas-api-tls\") pod \"f0295476-7278-4b04-af13-d09dafe0b5b6\" (UID: \"f0295476-7278-4b04-af13-d09dafe0b5b6\") " Oct 31 00:17:48.955389 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:48.952824 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f0295476-7278-4b04-af13-d09dafe0b5b6-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0295476-7278-4b04-af13-d09dafe0b5b6-openshift-service-ca\") pod \"f0295476-7278-4b04-af13-d09dafe0b5b6\" (UID: \"f0295476-7278-4b04-af13-d09dafe0b5b6\") " Oct 31 00:17:48.955389 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:48.952933 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f0295476-7278-4b04-af13-d09dafe0b5b6-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f0295476-7278-4b04-af13-d09dafe0b5b6-metrics-certs\") pod \"f0295476-7278-4b04-af13-d09dafe0b5b6\" (UID: \"f0295476-7278-4b04-af13-d09dafe0b5b6\") " Oct 31 00:17:48.955389 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:48.953026 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0295476-7278-4b04-af13-d09dafe0b5b6-kube-api-access-6bt8r\" (UniqueName: \"kubernetes.io/projected/f0295476-7278-4b04-af13-d09dafe0b5b6-kube-api-access-6bt8r\") pod \"f0295476-7278-4b04-af13-d09dafe0b5b6\" (UID: \"f0295476-7278-4b04-af13-d09dafe0b5b6\") " Oct 31 00:17:48.955389 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:48.954401 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0295476-7278-4b04-af13-d09dafe0b5b6-openshift-service-ca" pod "f0295476-7278-4b04-af13-d09dafe0b5b6" (UID: "f0295476-7278-4b04-af13-d09dafe0b5b6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:17:48.957504 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:48.957471 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f0295476-7278-4b04-af13-d09dafe0b5b6-maas-api-tls" pod "f0295476-7278-4b04-af13-d09dafe0b5b6" (UID: "f0295476-7278-4b04-af13-d09dafe0b5b6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:17:48.957641 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:48.957511 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f0295476-7278-4b04-af13-d09dafe0b5b6-metrics-certs" pod "f0295476-7278-4b04-af13-d09dafe0b5b6" (UID: "f0295476-7278-4b04-af13-d09dafe0b5b6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:17:48.957771 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:48.957749 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0295476-7278-4b04-af13-d09dafe0b5b6-kube-api-access-6bt8r" pod "f0295476-7278-4b04-af13-d09dafe0b5b6" (UID: "f0295476-7278-4b04-af13-d09dafe0b5b6"). InnerVolumeSpecName "kube-api-access-6bt8r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:17:49.055217 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:49.055127 4927 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f0295476-7278-4b04-af13-d09dafe0b5b6-maas-api-tls\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:17:49.055567 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:49.055543 4927 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0295476-7278-4b04-af13-d09dafe0b5b6-openshift-service-ca\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:17:49.055761 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:49.055730 4927 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f0295476-7278-4b04-af13-d09dafe0b5b6-metrics-certs\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:17:49.055880 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:49.055864 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6bt8r\" (UniqueName: \"kubernetes.io/projected/f0295476-7278-4b04-af13-d09dafe0b5b6-kube-api-access-6bt8r\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:17:49.607263 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:49.607226 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-4a3dba-744d479576-fx5nx" Oct 31 00:17:49.607263 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:49.607245 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-4a3dba-744d479576-fx5nx" event={"ID":"f0295476-7278-4b04-af13-d09dafe0b5b6","Type":"ContainerDied","Data":"00bbaf6706ea5db937ea575ce594963bf4252a95da2c55102e93e5840fd56f65"} Oct 31 00:17:49.607841 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:49.607354 4927 scope.go:122] "RemoveContainer" containerID="2d8ef2f89979a6d233b1d612e90d708121b6b44568cc97de08635e8ea03d4674" Oct 31 00:17:49.679707 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:17:49.679672 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sqv69" containerName="extractor" Oct 31 00:17:49.778567 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:49.778534 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-4a3dba-744d479576-fx5nx"] Oct 31 00:17:49.832394 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:49.832357 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-4a3dba-744d479576-fx5nx"] Oct 31 00:17:50.240373 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:50.240306 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:17:50.521221 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:50.521127 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:17:50.679733 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:50.679698 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:17:50.881623 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:50.881529 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-27e82b-5968f9f875-t9qz2" podUID="687684ad-5b0d-493a-9fa4-cbbe640229fe" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 00:17:51.088526 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:51.088492 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f0295476-7278-4b04-af13-d09dafe0b5b6" path="/var/lib/kubelet/pods/f0295476-7278-4b04-af13-d09dafe0b5b6/volumes" Oct 31 00:17:51.181412 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:51.181369 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:17:51.400398 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:51.400314 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:17:51.563118 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:51.563034 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:17:51.748591 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:51.748556 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-c9c21e-openshift-default-749bcfc96c-wjzsh"] Oct 31 00:17:51.749160 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:51.748807 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-c9c21e-openshift-default-749bcfc96c-wjzsh" podUID="3898ce0c-61e4-4888-9e54-3185c86a5326" containerName="istio-proxy" containerID="cri-o://47cbb5f450fa0df6d752b192904af4ab0983057b7c61403a5bf24363a3d906dc" gracePeriod=30 Oct 31 00:17:51.768570 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:51.768535 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:17:51.936361 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:51.931448 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-27e82b-64f94b9586-wb68f" podUID="59ccea66-1d77-43cd-bd56-e88b44a5e621" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 00:17:53.954704 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:53.954664 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:17:54.494813 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:54.494772 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-76b073-openshift-default-697d759dd5-ql7sd"] Oct 31 00:17:54.495568 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:54.495550 4927 cpu_manager.go:414] "containerMap: removing container" podUID="e8587ee7-1b21-409a-8c2d-ca9d7b69be4a" containerName="maas-api" Oct 31 00:17:54.495651 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:54.495570 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8587ee7-1b21-409a-8c2d-ca9d7b69be4a" containerName="maas-api" Oct 31 00:17:54.495651 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:54.495590 4927 cpu_manager.go:414] "containerMap: removing container" podUID="3b669d7f-c767-4c4e-a524-6e774fc69034" containerName="istio-proxy" Oct 31 00:17:54.495651 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:54.495596 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b669d7f-c767-4c4e-a524-6e774fc69034" containerName="istio-proxy" Oct 31 00:17:54.495651 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:54.495609 4927 cpu_manager.go:414] "containerMap: removing container" podUID="bb4e1d22-509e-4c3f-b877-8094320fc9ed" containerName="payload-pre-processing" Oct 31 00:17:54.495651 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:54.495616 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb4e1d22-509e-4c3f-b877-8094320fc9ed" containerName="payload-pre-processing" Oct 31 00:17:54.495651 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:54.495626 4927 cpu_manager.go:414] "containerMap: removing container" podUID="31bc5ff3-08ec-470f-b1f9-713946258110" containerName="payload-pre-processing" Oct 31 00:17:54.495651 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:54.495631 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31bc5ff3-08ec-470f-b1f9-713946258110" containerName="payload-pre-processing" Oct 31 00:17:54.495651 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:54.495639 4927 cpu_manager.go:414] "containerMap: removing container" podUID="dbec9857-9469-47ff-8977-0f912e5d0a9c" containerName="maas-api" Oct 31 00:17:54.495651 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:54.495644 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dbec9857-9469-47ff-8977-0f912e5d0a9c" containerName="maas-api" Oct 31 00:17:54.495651 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:54.495656 4927 cpu_manager.go:414] "containerMap: removing container" podUID="f0295476-7278-4b04-af13-d09dafe0b5b6" containerName="maas-api" Oct 31 00:17:54.495980 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:54.495661 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0295476-7278-4b04-af13-d09dafe0b5b6" containerName="maas-api" Oct 31 00:17:54.495980 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:54.495671 4927 cpu_manager.go:414] "containerMap: removing container" podUID="4ba8ced4-e93f-46ac-87c5-3d0b7fadc02d" containerName="main" Oct 31 00:17:54.495980 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:54.495675 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ba8ced4-e93f-46ac-87c5-3d0b7fadc02d" containerName="main" Oct 31 00:17:54.495980 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:54.495687 4927 cpu_manager.go:414] "containerMap: removing container" podUID="c51c586a-f2d8-48f9-8813-3b6e4f360e5e" containerName="istio-proxy" Oct 31 00:17:54.495980 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:54.495692 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c51c586a-f2d8-48f9-8813-3b6e4f360e5e" containerName="istio-proxy" Oct 31 00:17:54.495980 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:54.495699 4927 cpu_manager.go:414] "containerMap: removing container" podUID="a7afae12-c562-4a37-adf2-506d8780f93f" containerName="payload-pre-processing" Oct 31 00:17:54.495980 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:54.495704 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7afae12-c562-4a37-adf2-506d8780f93f" containerName="payload-pre-processing" Oct 31 00:17:54.495980 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:54.495720 4927 cpu_manager.go:414] "containerMap: removing container" podUID="d639aac3-8752-495f-8c71-fdc050a03016" containerName="payload-processing" Oct 31 00:17:54.495980 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:54.495725 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d639aac3-8752-495f-8c71-fdc050a03016" containerName="payload-processing" Oct 31 00:17:54.495980 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:54.495735 4927 cpu_manager.go:414] "containerMap: removing container" podUID="c51c586a-f2d8-48f9-8813-3b6e4f360e5e" containerName="istio-proxy" Oct 31 00:17:54.495980 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:54.495739 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c51c586a-f2d8-48f9-8813-3b6e4f360e5e" containerName="istio-proxy" Oct 31 00:17:54.495980 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:54.495746 4927 cpu_manager.go:414] "containerMap: removing container" podUID="b9106a2b-acad-4af9-af60-757055c5d135" containerName="payload-processing" Oct 31 00:17:54.495980 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:54.495750 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9106a2b-acad-4af9-af60-757055c5d135" containerName="payload-processing" Oct 31 00:17:54.495980 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:54.495766 4927 cpu_manager.go:414] "containerMap: removing container" podUID="b45aa3c1-e11b-4d72-b93c-2248b7916834" containerName="payload-processing" Oct 31 00:17:54.495980 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:54.495771 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b45aa3c1-e11b-4d72-b93c-2248b7916834" containerName="payload-processing" Oct 31 00:17:54.495980 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:54.495868 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0295476-7278-4b04-af13-d09dafe0b5b6" containerName="maas-api" Oct 31 00:17:54.495980 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:54.495877 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ba8ced4-e93f-46ac-87c5-3d0b7fadc02d" containerName="main" Oct 31 00:17:54.495980 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:54.495884 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="c51c586a-f2d8-48f9-8813-3b6e4f360e5e" containerName="istio-proxy" Oct 31 00:17:54.495980 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:54.495890 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7afae12-c562-4a37-adf2-506d8780f93f" containerName="payload-pre-processing" Oct 31 00:17:54.495980 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:54.495899 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b669d7f-c767-4c4e-a524-6e774fc69034" containerName="istio-proxy" Oct 31 00:17:54.495980 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:54.495905 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="d639aac3-8752-495f-8c71-fdc050a03016" containerName="payload-processing" Oct 31 00:17:54.495980 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:54.495913 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="c51c586a-f2d8-48f9-8813-3b6e4f360e5e" containerName="istio-proxy" Oct 31 00:17:54.495980 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:54.495918 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9106a2b-acad-4af9-af60-757055c5d135" containerName="payload-processing" Oct 31 00:17:54.495980 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:54.495927 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="b45aa3c1-e11b-4d72-b93c-2248b7916834" containerName="payload-processing" Oct 31 00:17:54.495980 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:54.495935 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8587ee7-1b21-409a-8c2d-ca9d7b69be4a" containerName="maas-api" Oct 31 00:17:54.495980 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:54.495943 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb4e1d22-509e-4c3f-b877-8094320fc9ed" containerName="payload-pre-processing" Oct 31 00:17:54.495980 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:54.495949 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="31bc5ff3-08ec-470f-b1f9-713946258110" containerName="payload-pre-processing" Oct 31 00:17:54.495980 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:54.495955 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="dbec9857-9469-47ff-8977-0f912e5d0a9c" containerName="maas-api" Oct 31 00:17:54.503808 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:54.501074 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-76b073-openshift-default-697d759dd5-ql7sd" Oct 31 00:17:54.609864 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:54.588206 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-76b073-openshift-default-dockercfg-tq5gk\"" Oct 31 00:17:54.621666 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:54.621626 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:17:54.741774 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:54.737131 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cdqhf\" (UniqueName: \"kubernetes.io/projected/78b89047-2bad-472e-a80d-0ab8f1755877-kube-api-access-cdqhf\") pod \"e2e-inf-a-76b073-openshift-default-697d759dd5-ql7sd\" (UID: \"78b89047-2bad-472e-a80d-0ab8f1755877\") " pod="openshift-ingress/e2e-inf-a-76b073-openshift-default-697d759dd5-ql7sd" Oct 31 00:17:54.741774 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:54.737300 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78b89047-2bad-472e-a80d-0ab8f1755877-workload-socket\") pod \"e2e-inf-a-76b073-openshift-default-697d759dd5-ql7sd\" (UID: \"78b89047-2bad-472e-a80d-0ab8f1755877\") " pod="openshift-ingress/e2e-inf-a-76b073-openshift-default-697d759dd5-ql7sd" Oct 31 00:17:54.741774 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:54.737370 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78b89047-2bad-472e-a80d-0ab8f1755877-workload-certs\") pod \"e2e-inf-a-76b073-openshift-default-697d759dd5-ql7sd\" (UID: \"78b89047-2bad-472e-a80d-0ab8f1755877\") " pod="openshift-ingress/e2e-inf-a-76b073-openshift-default-697d759dd5-ql7sd" Oct 31 00:17:54.741774 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:54.737470 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78b89047-2bad-472e-a80d-0ab8f1755877-istio-envoy\") pod \"e2e-inf-a-76b073-openshift-default-697d759dd5-ql7sd\" (UID: \"78b89047-2bad-472e-a80d-0ab8f1755877\") " pod="openshift-ingress/e2e-inf-a-76b073-openshift-default-697d759dd5-ql7sd" Oct 31 00:17:54.741774 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:54.737548 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/78b89047-2bad-472e-a80d-0ab8f1755877-istio-token\") pod \"e2e-inf-a-76b073-openshift-default-697d759dd5-ql7sd\" (UID: \"78b89047-2bad-472e-a80d-0ab8f1755877\") " pod="openshift-ingress/e2e-inf-a-76b073-openshift-default-697d759dd5-ql7sd" Oct 31 00:17:54.741774 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:54.737600 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78b89047-2bad-472e-a80d-0ab8f1755877-istio-podinfo\") pod \"e2e-inf-a-76b073-openshift-default-697d759dd5-ql7sd\" (UID: \"78b89047-2bad-472e-a80d-0ab8f1755877\") " pod="openshift-ingress/e2e-inf-a-76b073-openshift-default-697d759dd5-ql7sd" Oct 31 00:17:54.741774 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:54.737678 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78b89047-2bad-472e-a80d-0ab8f1755877-istiod-ca-cert\") pod \"e2e-inf-a-76b073-openshift-default-697d759dd5-ql7sd\" (UID: \"78b89047-2bad-472e-a80d-0ab8f1755877\") " pod="openshift-ingress/e2e-inf-a-76b073-openshift-default-697d759dd5-ql7sd" Oct 31 00:17:54.741774 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:54.737781 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78b89047-2bad-472e-a80d-0ab8f1755877-istio-data\") pod \"e2e-inf-a-76b073-openshift-default-697d759dd5-ql7sd\" (UID: \"78b89047-2bad-472e-a80d-0ab8f1755877\") " pod="openshift-ingress/e2e-inf-a-76b073-openshift-default-697d759dd5-ql7sd" Oct 31 00:17:54.741774 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:54.737834 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78b89047-2bad-472e-a80d-0ab8f1755877-credential-socket\") pod \"e2e-inf-a-76b073-openshift-default-697d759dd5-ql7sd\" (UID: \"78b89047-2bad-472e-a80d-0ab8f1755877\") " pod="openshift-ingress/e2e-inf-a-76b073-openshift-default-697d759dd5-ql7sd" Oct 31 00:17:54.776888 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:54.767215 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-76b073-openshift-default-697d759dd5-ql7sd"] Oct 31 00:17:54.843404 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:54.842228 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78b89047-2bad-472e-a80d-0ab8f1755877-istio-data\") pod \"e2e-inf-a-76b073-openshift-default-697d759dd5-ql7sd\" (UID: \"78b89047-2bad-472e-a80d-0ab8f1755877\") " pod="openshift-ingress/e2e-inf-a-76b073-openshift-default-697d759dd5-ql7sd" Oct 31 00:17:54.843716 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:54.843689 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78b89047-2bad-472e-a80d-0ab8f1755877-credential-socket\") pod \"e2e-inf-a-76b073-openshift-default-697d759dd5-ql7sd\" (UID: \"78b89047-2bad-472e-a80d-0ab8f1755877\") " pod="openshift-ingress/e2e-inf-a-76b073-openshift-default-697d759dd5-ql7sd" Oct 31 00:17:54.844237 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:54.844212 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cdqhf\" (UniqueName: \"kubernetes.io/projected/78b89047-2bad-472e-a80d-0ab8f1755877-kube-api-access-cdqhf\") pod \"e2e-inf-a-76b073-openshift-default-697d759dd5-ql7sd\" (UID: \"78b89047-2bad-472e-a80d-0ab8f1755877\") " pod="openshift-ingress/e2e-inf-a-76b073-openshift-default-697d759dd5-ql7sd" Oct 31 00:17:54.844868 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:54.844844 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78b89047-2bad-472e-a80d-0ab8f1755877-workload-socket\") pod \"e2e-inf-a-76b073-openshift-default-697d759dd5-ql7sd\" (UID: \"78b89047-2bad-472e-a80d-0ab8f1755877\") " pod="openshift-ingress/e2e-inf-a-76b073-openshift-default-697d759dd5-ql7sd" Oct 31 00:17:54.845326 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:54.845309 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78b89047-2bad-472e-a80d-0ab8f1755877-workload-certs\") pod \"e2e-inf-a-76b073-openshift-default-697d759dd5-ql7sd\" (UID: \"78b89047-2bad-472e-a80d-0ab8f1755877\") " pod="openshift-ingress/e2e-inf-a-76b073-openshift-default-697d759dd5-ql7sd" Oct 31 00:17:54.845831 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:54.845811 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78b89047-2bad-472e-a80d-0ab8f1755877-istio-envoy\") pod \"e2e-inf-a-76b073-openshift-default-697d759dd5-ql7sd\" (UID: \"78b89047-2bad-472e-a80d-0ab8f1755877\") " pod="openshift-ingress/e2e-inf-a-76b073-openshift-default-697d759dd5-ql7sd" Oct 31 00:17:54.846325 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:54.846306 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/78b89047-2bad-472e-a80d-0ab8f1755877-istio-token\") pod \"e2e-inf-a-76b073-openshift-default-697d759dd5-ql7sd\" (UID: \"78b89047-2bad-472e-a80d-0ab8f1755877\") " pod="openshift-ingress/e2e-inf-a-76b073-openshift-default-697d759dd5-ql7sd" Oct 31 00:17:54.846738 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:54.846718 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78b89047-2bad-472e-a80d-0ab8f1755877-istio-podinfo\") pod \"e2e-inf-a-76b073-openshift-default-697d759dd5-ql7sd\" (UID: \"78b89047-2bad-472e-a80d-0ab8f1755877\") " pod="openshift-ingress/e2e-inf-a-76b073-openshift-default-697d759dd5-ql7sd" Oct 31 00:17:54.847768 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:54.847747 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78b89047-2bad-472e-a80d-0ab8f1755877-istiod-ca-cert\") pod \"e2e-inf-a-76b073-openshift-default-697d759dd5-ql7sd\" (UID: \"78b89047-2bad-472e-a80d-0ab8f1755877\") " pod="openshift-ingress/e2e-inf-a-76b073-openshift-default-697d759dd5-ql7sd" Oct 31 00:17:54.848020 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:54.844098 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78b89047-2bad-472e-a80d-0ab8f1755877-credential-socket\") pod \"e2e-inf-a-76b073-openshift-default-697d759dd5-ql7sd\" (UID: \"78b89047-2bad-472e-a80d-0ab8f1755877\") " pod="openshift-ingress/e2e-inf-a-76b073-openshift-default-697d759dd5-ql7sd" Oct 31 00:17:54.848923 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:54.845244 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78b89047-2bad-472e-a80d-0ab8f1755877-workload-socket\") pod \"e2e-inf-a-76b073-openshift-default-697d759dd5-ql7sd\" (UID: \"78b89047-2bad-472e-a80d-0ab8f1755877\") " pod="openshift-ingress/e2e-inf-a-76b073-openshift-default-697d759dd5-ql7sd" Oct 31 00:17:54.849061 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:54.843252 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78b89047-2bad-472e-a80d-0ab8f1755877-istio-data\") pod \"e2e-inf-a-76b073-openshift-default-697d759dd5-ql7sd\" (UID: \"78b89047-2bad-472e-a80d-0ab8f1755877\") " pod="openshift-ingress/e2e-inf-a-76b073-openshift-default-697d759dd5-ql7sd" Oct 31 00:17:54.901546 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:54.845697 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78b89047-2bad-472e-a80d-0ab8f1755877-workload-certs\") pod \"e2e-inf-a-76b073-openshift-default-697d759dd5-ql7sd\" (UID: \"78b89047-2bad-472e-a80d-0ab8f1755877\") " pod="openshift-ingress/e2e-inf-a-76b073-openshift-default-697d759dd5-ql7sd" Oct 31 00:17:54.920879 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:54.848867 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78b89047-2bad-472e-a80d-0ab8f1755877-istiod-ca-cert\") pod \"e2e-inf-a-76b073-openshift-default-697d759dd5-ql7sd\" (UID: \"78b89047-2bad-472e-a80d-0ab8f1755877\") " pod="openshift-ingress/e2e-inf-a-76b073-openshift-default-697d759dd5-ql7sd" Oct 31 00:17:54.920976 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:54.896740 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78b89047-2bad-472e-a80d-0ab8f1755877-istio-envoy\") pod \"e2e-inf-a-76b073-openshift-default-697d759dd5-ql7sd\" (UID: \"78b89047-2bad-472e-a80d-0ab8f1755877\") " pod="openshift-ingress/e2e-inf-a-76b073-openshift-default-697d759dd5-ql7sd" Oct 31 00:17:54.921038 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:54.903949 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78b89047-2bad-472e-a80d-0ab8f1755877-istio-podinfo\") pod \"e2e-inf-a-76b073-openshift-default-697d759dd5-ql7sd\" (UID: \"78b89047-2bad-472e-a80d-0ab8f1755877\") " pod="openshift-ingress/e2e-inf-a-76b073-openshift-default-697d759dd5-ql7sd" Oct 31 00:17:54.937300 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:54.932946 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/78b89047-2bad-472e-a80d-0ab8f1755877-istio-token\") pod \"e2e-inf-a-76b073-openshift-default-697d759dd5-ql7sd\" (UID: \"78b89047-2bad-472e-a80d-0ab8f1755877\") " pod="openshift-ingress/e2e-inf-a-76b073-openshift-default-697d759dd5-ql7sd" Oct 31 00:17:54.937300 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:54.936063 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cdqhf\" (UniqueName: \"kubernetes.io/projected/78b89047-2bad-472e-a80d-0ab8f1755877-kube-api-access-cdqhf\") pod \"e2e-inf-a-76b073-openshift-default-697d759dd5-ql7sd\" (UID: \"78b89047-2bad-472e-a80d-0ab8f1755877\") " pod="openshift-ingress/e2e-inf-a-76b073-openshift-default-697d759dd5-ql7sd" Oct 31 00:17:54.946431 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:54.946309 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:17:55.115197 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:55.115174 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-27e82b-64f94b9586-wb68f" Oct 31 00:17:55.139017 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:55.138984 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-76b073-openshift-default-697d759dd5-ql7sd" Oct 31 00:17:55.178534 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:55.178504 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-27e82b-5968f9f875-t9qz2" Oct 31 00:17:55.254710 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:55.253660 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/687684ad-5b0d-493a-9fa4-cbbe640229fe-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/687684ad-5b0d-493a-9fa4-cbbe640229fe-plugins-config-volume\") pod \"687684ad-5b0d-493a-9fa4-cbbe640229fe\" (UID: \"687684ad-5b0d-493a-9fa4-cbbe640229fe\") " Oct 31 00:17:55.254710 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:55.253707 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/59ccea66-1d77-43cd-bd56-e88b44a5e621-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59ccea66-1d77-43cd-bd56-e88b44a5e621-plugins-config-volume\") pod \"59ccea66-1d77-43cd-bd56-e88b44a5e621\" (UID: \"59ccea66-1d77-43cd-bd56-e88b44a5e621\") " Oct 31 00:17:55.254710 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:55.253747 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59ccea66-1d77-43cd-bd56-e88b44a5e621-kube-api-access-ckl8m\" (UniqueName: \"kubernetes.io/projected/59ccea66-1d77-43cd-bd56-e88b44a5e621-kube-api-access-ckl8m\") pod \"59ccea66-1d77-43cd-bd56-e88b44a5e621\" (UID: \"59ccea66-1d77-43cd-bd56-e88b44a5e621\") " Oct 31 00:17:55.254710 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:55.253812 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/687684ad-5b0d-493a-9fa4-cbbe640229fe-kube-api-access-65chd\" (UniqueName: \"kubernetes.io/projected/687684ad-5b0d-493a-9fa4-cbbe640229fe-kube-api-access-65chd\") pod \"687684ad-5b0d-493a-9fa4-cbbe640229fe\" (UID: \"687684ad-5b0d-493a-9fa4-cbbe640229fe\") " Oct 31 00:17:55.255058 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:55.254815 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59ccea66-1d77-43cd-bd56-e88b44a5e621-plugins-config-volume" pod "59ccea66-1d77-43cd-bd56-e88b44a5e621" (UID: "59ccea66-1d77-43cd-bd56-e88b44a5e621"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:17:55.255058 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:55.254818 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/687684ad-5b0d-493a-9fa4-cbbe640229fe-plugins-config-volume" pod "687684ad-5b0d-493a-9fa4-cbbe640229fe" (UID: "687684ad-5b0d-493a-9fa4-cbbe640229fe"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:17:55.256629 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:55.256600 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59ccea66-1d77-43cd-bd56-e88b44a5e621-kube-api-access-ckl8m" pod "59ccea66-1d77-43cd-bd56-e88b44a5e621" (UID: "59ccea66-1d77-43cd-bd56-e88b44a5e621"). InnerVolumeSpecName "kube-api-access-ckl8m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:17:55.257663 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:55.257639 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/687684ad-5b0d-493a-9fa4-cbbe640229fe-kube-api-access-65chd" pod "687684ad-5b0d-493a-9fa4-cbbe640229fe" (UID: "687684ad-5b0d-493a-9fa4-cbbe640229fe"). InnerVolumeSpecName "kube-api-access-65chd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:17:55.356390 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:55.356292 4927 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/687684ad-5b0d-493a-9fa4-cbbe640229fe-plugins-config-volume\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:17:55.356390 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:55.356347 4927 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59ccea66-1d77-43cd-bd56-e88b44a5e621-plugins-config-volume\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:17:55.356390 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:55.356365 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ckl8m\" (UniqueName: \"kubernetes.io/projected/59ccea66-1d77-43cd-bd56-e88b44a5e621-kube-api-access-ckl8m\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:17:55.356390 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:55.356383 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-65chd\" (UniqueName: \"kubernetes.io/projected/687684ad-5b0d-493a-9fa4-cbbe640229fe-kube-api-access-65chd\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:17:55.454282 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:55.454250 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:17:55.583190 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:55.583159 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-76b073-openshift-default-697d759dd5-ql7sd"] Oct 31 00:17:55.583919 ip-10-0-141-180 kubenswrapper[4927]: W1031 00:17:55.583892 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78b89047_2bad_472e_a80d_0ab8f1755877.slice/crio-c4906276f547c7ece3d9b2f5e76254af194ba639ea5bedf8417cc3abc6cb6078 WatchSource:0}: Error finding container c4906276f547c7ece3d9b2f5e76254af194ba639ea5bedf8417cc3abc6cb6078: Status 404 returned error can't find the container with id c4906276f547c7ece3d9b2f5e76254af194ba639ea5bedf8417cc3abc6cb6078 Oct 31 00:17:55.586582 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:55.586545 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:17:55.586667 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:55.586618 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:17:55.586667 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:55.586646 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:17:55.907178 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:55.907092 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-76b073-openshift-default-697d759dd5-ql7sd" event={"ID":"78b89047-2bad-472e-a80d-0ab8f1755877","Type":"ContainerStarted","Data":"cf6775d7b3f36612239f5a7641c6558beb5c7e657831a0ecd62d672653c6541e"} Oct 31 00:17:55.911012 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:55.910971 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-76b073-openshift-default-697d759dd5-ql7sd" event={"ID":"78b89047-2bad-472e-a80d-0ab8f1755877","Type":"ContainerStarted","Data":"c4906276f547c7ece3d9b2f5e76254af194ba639ea5bedf8417cc3abc6cb6078"} Oct 31 00:17:55.934200 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:55.934157 4927 generic.go:356] "Generic (PLEG): container finished" podID="687684ad-5b0d-493a-9fa4-cbbe640229fe" containerID="97af6b4131cff76a3d0bf2dd637d64c4132121feb08d0a1acf9a6e4ea6429ee6" exitCode=0 Oct 31 00:17:55.934624 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:55.934594 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-27e82b-5968f9f875-t9qz2" event={"ID":"687684ad-5b0d-493a-9fa4-cbbe640229fe","Type":"ContainerDied","Data":"97af6b4131cff76a3d0bf2dd637d64c4132121feb08d0a1acf9a6e4ea6429ee6"} Oct 31 00:17:55.934807 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:55.934789 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-27e82b-5968f9f875-t9qz2" event={"ID":"687684ad-5b0d-493a-9fa4-cbbe640229fe","Type":"ContainerDied","Data":"ac57ee797300ddb624140b3badd499d024599e7cf270505d3058c55385829085"} Oct 31 00:17:55.934943 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:55.934929 4927 scope.go:122] "RemoveContainer" containerID="97af6b4131cff76a3d0bf2dd637d64c4132121feb08d0a1acf9a6e4ea6429ee6" Oct 31 00:17:55.957714 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:55.951432 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-27e82b-5968f9f875-t9qz2" Oct 31 00:17:55.961437 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:55.961405 4927 generic.go:356] "Generic (PLEG): container finished" podID="59ccea66-1d77-43cd-bd56-e88b44a5e621" containerID="a8eef16879d922846f405a93c664865feabb85ecddcfb489be76f82b84fef7c9" exitCode=0 Oct 31 00:17:55.961736 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:55.961712 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-27e82b-64f94b9586-wb68f" event={"ID":"59ccea66-1d77-43cd-bd56-e88b44a5e621","Type":"ContainerDied","Data":"a8eef16879d922846f405a93c664865feabb85ecddcfb489be76f82b84fef7c9"} Oct 31 00:17:55.961935 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:55.961902 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-27e82b-64f94b9586-wb68f" event={"ID":"59ccea66-1d77-43cd-bd56-e88b44a5e621","Type":"ContainerDied","Data":"bb81b7b3ed577dc8e2d039c0020ea560729abb232c159242f5a0bd4135987103"} Oct 31 00:17:55.962158 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:55.962143 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-27e82b-64f94b9586-wb68f" Oct 31 00:17:56.025695 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:56.025666 4927 scope.go:122] "RemoveContainer" containerID="97af6b4131cff76a3d0bf2dd637d64c4132121feb08d0a1acf9a6e4ea6429ee6" Oct 31 00:17:56.026152 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:17:56.026123 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"97af6b4131cff76a3d0bf2dd637d64c4132121feb08d0a1acf9a6e4ea6429ee6\": container with ID starting with 97af6b4131cff76a3d0bf2dd637d64c4132121feb08d0a1acf9a6e4ea6429ee6 not found: ID does not exist" containerID="97af6b4131cff76a3d0bf2dd637d64c4132121feb08d0a1acf9a6e4ea6429ee6" Oct 31 00:17:56.026246 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:56.026162 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97af6b4131cff76a3d0bf2dd637d64c4132121feb08d0a1acf9a6e4ea6429ee6"} err="failed to get container status \"97af6b4131cff76a3d0bf2dd637d64c4132121feb08d0a1acf9a6e4ea6429ee6\": rpc error: code = NotFound desc = could not find container \"97af6b4131cff76a3d0bf2dd637d64c4132121feb08d0a1acf9a6e4ea6429ee6\": container with ID starting with 97af6b4131cff76a3d0bf2dd637d64c4132121feb08d0a1acf9a6e4ea6429ee6 not found: ID does not exist" Oct 31 00:17:56.026246 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:56.026190 4927 scope.go:122] "RemoveContainer" containerID="a8eef16879d922846f405a93c664865feabb85ecddcfb489be76f82b84fef7c9" Oct 31 00:17:56.040148 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:56.040128 4927 scope.go:122] "RemoveContainer" containerID="a8eef16879d922846f405a93c664865feabb85ecddcfb489be76f82b84fef7c9" Oct 31 00:17:56.041117 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:17:56.041093 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8eef16879d922846f405a93c664865feabb85ecddcfb489be76f82b84fef7c9\": container with ID starting with a8eef16879d922846f405a93c664865feabb85ecddcfb489be76f82b84fef7c9 not found: ID does not exist" containerID="a8eef16879d922846f405a93c664865feabb85ecddcfb489be76f82b84fef7c9" Oct 31 00:17:56.041235 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:56.041129 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8eef16879d922846f405a93c664865feabb85ecddcfb489be76f82b84fef7c9"} err="failed to get container status \"a8eef16879d922846f405a93c664865feabb85ecddcfb489be76f82b84fef7c9\": rpc error: code = NotFound desc = could not find container \"a8eef16879d922846f405a93c664865feabb85ecddcfb489be76f82b84fef7c9\": container with ID starting with a8eef16879d922846f405a93c664865feabb85ecddcfb489be76f82b84fef7c9 not found: ID does not exist" Oct 31 00:17:56.053910 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:56.053864 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-76b073-openshift-default-697d759dd5-ql7sd" podStartSLOduration=2.053850931 podStartE2EDuration="2.053850931s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:17:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:17:56.051595461 +0000 UTC m=+2189.087926282" watchObservedRunningTime="2026-10-31 00:17:56.053850931 +0000 UTC m=+2189.090181752" Oct 31 00:17:56.139408 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:56.139373 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-76b073-openshift-default-697d759dd5-ql7sd" Oct 31 00:17:56.141376 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:56.141349 4927 patch_prober.go:29] interesting pod/e2e-inf-a-76b073-openshift-default-697d759dd5-ql7sd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.230:15021/healthz/ready\": dial tcp 10.132.0.230:15021: connect: connection refused" start-of-body= Oct 31 00:17:56.141500 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:56.141399 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-76b073-openshift-default-697d759dd5-ql7sd" podUID="78b89047-2bad-472e-a80d-0ab8f1755877" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.230:15021/healthz/ready\": dial tcp 10.132.0.230:15021: connect: connection refused" Oct 31 00:17:56.185404 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:56.185281 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-27e82b-5968f9f875-t9qz2"] Oct 31 00:17:56.208138 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:56.208089 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-27e82b-5968f9f875-t9qz2"] Oct 31 00:17:56.289788 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:56.289753 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-27e82b-64f94b9586-wb68f"] Oct 31 00:17:56.293617 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:56.293593 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-27e82b-64f94b9586-wb68f"] Oct 31 00:17:56.980586 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:56.980556 4927 generic.go:356] "Generic (PLEG): container finished" podID="3898ce0c-61e4-4888-9e54-3185c86a5326" containerID="47cbb5f450fa0df6d752b192904af4ab0983057b7c61403a5bf24363a3d906dc" exitCode=0 Oct 31 00:17:56.980786 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:56.980638 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c9c21e-openshift-default-749bcfc96c-wjzsh" event={"ID":"3898ce0c-61e4-4888-9e54-3185c86a5326","Type":"ContainerDied","Data":"47cbb5f450fa0df6d752b192904af4ab0983057b7c61403a5bf24363a3d906dc"} Oct 31 00:17:56.980786 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:56.980715 4927 scope.go:122] "RemoveContainer" containerID="c14ecd91a809659c013aabbf0dd11ba620bb7e763cd825cc1fbdf6357bc6bc56" Oct 31 00:17:57.041974 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:57.041946 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-c9c21e-openshift-default-749bcfc96c-wjzsh" Oct 31 00:17:57.088726 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:57.088693 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="59ccea66-1d77-43cd-bd56-e88b44a5e621" path="/var/lib/kubelet/pods/59ccea66-1d77-43cd-bd56-e88b44a5e621/volumes" Oct 31 00:17:57.089125 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:57.089111 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="687684ad-5b0d-493a-9fa4-cbbe640229fe" path="/var/lib/kubelet/pods/687684ad-5b0d-493a-9fa4-cbbe640229fe/volumes" Oct 31 00:17:57.144574 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:57.144545 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-76b073-openshift-default-697d759dd5-ql7sd" Oct 31 00:17:57.215816 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:57.215777 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3898ce0c-61e4-4888-9e54-3185c86a5326-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3898ce0c-61e4-4888-9e54-3185c86a5326-istio-podinfo\") pod \"3898ce0c-61e4-4888-9e54-3185c86a5326\" (UID: \"3898ce0c-61e4-4888-9e54-3185c86a5326\") " Oct 31 00:17:57.215994 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:57.215836 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3898ce0c-61e4-4888-9e54-3185c86a5326-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3898ce0c-61e4-4888-9e54-3185c86a5326-credential-socket\") pod \"3898ce0c-61e4-4888-9e54-3185c86a5326\" (UID: \"3898ce0c-61e4-4888-9e54-3185c86a5326\") " Oct 31 00:17:57.215994 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:57.215967 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3898ce0c-61e4-4888-9e54-3185c86a5326-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3898ce0c-61e4-4888-9e54-3185c86a5326-istio-envoy\") pod \"3898ce0c-61e4-4888-9e54-3185c86a5326\" (UID: \"3898ce0c-61e4-4888-9e54-3185c86a5326\") " Oct 31 00:17:57.216680 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:57.216571 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3898ce0c-61e4-4888-9e54-3185c86a5326-credential-socket" pod "3898ce0c-61e4-4888-9e54-3185c86a5326" (UID: "3898ce0c-61e4-4888-9e54-3185c86a5326"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:17:57.216832 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:57.216716 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3898ce0c-61e4-4888-9e54-3185c86a5326-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3898ce0c-61e4-4888-9e54-3185c86a5326-istio-data\") pod \"3898ce0c-61e4-4888-9e54-3185c86a5326\" (UID: \"3898ce0c-61e4-4888-9e54-3185c86a5326\") " Oct 31 00:17:57.216905 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:57.216843 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3898ce0c-61e4-4888-9e54-3185c86a5326-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3898ce0c-61e4-4888-9e54-3185c86a5326-istiod-ca-cert\") pod \"3898ce0c-61e4-4888-9e54-3185c86a5326\" (UID: \"3898ce0c-61e4-4888-9e54-3185c86a5326\") " Oct 31 00:17:57.216971 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:57.216946 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3898ce0c-61e4-4888-9e54-3185c86a5326-kube-api-access-j2mlm\" (UniqueName: \"kubernetes.io/projected/3898ce0c-61e4-4888-9e54-3185c86a5326-kube-api-access-j2mlm\") pod \"3898ce0c-61e4-4888-9e54-3185c86a5326\" (UID: \"3898ce0c-61e4-4888-9e54-3185c86a5326\") " Oct 31 00:17:57.217295 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:57.217268 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3898ce0c-61e4-4888-9e54-3185c86a5326-istio-data" pod "3898ce0c-61e4-4888-9e54-3185c86a5326" (UID: "3898ce0c-61e4-4888-9e54-3185c86a5326"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:17:57.217863 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:57.217839 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3898ce0c-61e4-4888-9e54-3185c86a5326-istiod-ca-cert" pod "3898ce0c-61e4-4888-9e54-3185c86a5326" (UID: "3898ce0c-61e4-4888-9e54-3185c86a5326"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:17:57.230395 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:57.218108 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3898ce0c-61e4-4888-9e54-3185c86a5326-istio-token\" (UniqueName: \"kubernetes.io/projected/3898ce0c-61e4-4888-9e54-3185c86a5326-istio-token\") pod \"3898ce0c-61e4-4888-9e54-3185c86a5326\" (UID: \"3898ce0c-61e4-4888-9e54-3185c86a5326\") " Oct 31 00:17:57.230395 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:57.218217 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3898ce0c-61e4-4888-9e54-3185c86a5326-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3898ce0c-61e4-4888-9e54-3185c86a5326-workload-certs\") pod \"3898ce0c-61e4-4888-9e54-3185c86a5326\" (UID: \"3898ce0c-61e4-4888-9e54-3185c86a5326\") " Oct 31 00:17:57.230395 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:57.218264 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3898ce0c-61e4-4888-9e54-3185c86a5326-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3898ce0c-61e4-4888-9e54-3185c86a5326-workload-socket\") pod \"3898ce0c-61e4-4888-9e54-3185c86a5326\" (UID: \"3898ce0c-61e4-4888-9e54-3185c86a5326\") " Oct 31 00:17:57.230395 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:57.219488 4927 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3898ce0c-61e4-4888-9e54-3185c86a5326-istio-data\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:17:57.230395 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:57.219495 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3898ce0c-61e4-4888-9e54-3185c86a5326-istio-podinfo" pod "3898ce0c-61e4-4888-9e54-3185c86a5326" (UID: "3898ce0c-61e4-4888-9e54-3185c86a5326"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:17:57.230395 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:57.219522 4927 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3898ce0c-61e4-4888-9e54-3185c86a5326-istiod-ca-cert\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:17:57.230395 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:57.219539 4927 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3898ce0c-61e4-4888-9e54-3185c86a5326-credential-socket\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:17:57.230395 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:57.219634 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3898ce0c-61e4-4888-9e54-3185c86a5326-workload-certs" pod "3898ce0c-61e4-4888-9e54-3185c86a5326" (UID: "3898ce0c-61e4-4888-9e54-3185c86a5326"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:17:57.230395 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:57.219830 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3898ce0c-61e4-4888-9e54-3185c86a5326-workload-socket" pod "3898ce0c-61e4-4888-9e54-3185c86a5326" (UID: "3898ce0c-61e4-4888-9e54-3185c86a5326"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:17:57.230395 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:57.222094 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3898ce0c-61e4-4888-9e54-3185c86a5326-istio-envoy" pod "3898ce0c-61e4-4888-9e54-3185c86a5326" (UID: "3898ce0c-61e4-4888-9e54-3185c86a5326"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:17:57.237000 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:57.236970 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3898ce0c-61e4-4888-9e54-3185c86a5326-istio-token" pod "3898ce0c-61e4-4888-9e54-3185c86a5326" (UID: "3898ce0c-61e4-4888-9e54-3185c86a5326"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:17:57.237684 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:57.237655 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3898ce0c-61e4-4888-9e54-3185c86a5326-kube-api-access-j2mlm" pod "3898ce0c-61e4-4888-9e54-3185c86a5326" (UID: "3898ce0c-61e4-4888-9e54-3185c86a5326"). InnerVolumeSpecName "kube-api-access-j2mlm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:17:57.321073 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:57.320996 4927 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3898ce0c-61e4-4888-9e54-3185c86a5326-istio-token\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:17:57.321073 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:57.321036 4927 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3898ce0c-61e4-4888-9e54-3185c86a5326-workload-certs\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:17:57.321073 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:57.321053 4927 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3898ce0c-61e4-4888-9e54-3185c86a5326-workload-socket\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:17:57.321073 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:57.321068 4927 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3898ce0c-61e4-4888-9e54-3185c86a5326-istio-podinfo\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:17:57.321380 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:57.321082 4927 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3898ce0c-61e4-4888-9e54-3185c86a5326-istio-envoy\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:17:57.321380 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:57.321096 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j2mlm\" (UniqueName: \"kubernetes.io/projected/3898ce0c-61e4-4888-9e54-3185c86a5326-kube-api-access-j2mlm\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:17:57.381541 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:57.381505 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:17:57.581020 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:57.580922 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:17:57.703659 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:57.703620 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-4a3dba-openshift-default-dc9664475-zxb5s"] Oct 31 00:17:57.703921 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:57.703894 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-4a3dba-openshift-default-dc9664475-zxb5s" podUID="1b1c447d-a95f-4edb-9c9d-8987eea5eaf4" containerName="istio-proxy" containerID="cri-o://963a5aa1c641ed6f16929af7e3016f135429161948921c103d3981e14fd8ab1d" gracePeriod=30 Oct 31 00:17:57.741501 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:57.741467 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:17:57.988396 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:57.985907 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:17:58.048416 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:58.048381 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-c9c21e-openshift-default-749bcfc96c-wjzsh" Oct 31 00:17:58.059883 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:58.057619 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c9c21e-openshift-default-749bcfc96c-wjzsh" event={"ID":"3898ce0c-61e4-4888-9e54-3185c86a5326","Type":"ContainerDied","Data":"5253d1b7b1a0d9094a0d5bf28ca0f7db9a2c1c023511f7f2f8b283547e72d521"} Oct 31 00:17:58.059883 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:58.057698 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-76b073-openshift-default-697d759dd5-ql7sd" Oct 31 00:17:58.059883 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:58.057761 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-76b073-openshift-default-697d759dd5-ql7sd" Oct 31 00:17:58.059883 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:58.057791 4927 scope.go:122] "RemoveContainer" containerID="47cbb5f450fa0df6d752b192904af4ab0983057b7c61403a5bf24363a3d906dc" Oct 31 00:17:58.221300 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:58.221264 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:17:58.384240 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:58.382288 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-c9c21e-openshift-default-749bcfc96c-wjzsh"] Oct 31 00:17:58.387601 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:58.387572 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-c9c21e-openshift-default-749bcfc96c-wjzsh"] Oct 31 00:17:58.653889 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:58.653810 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:17:58.995102 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:58.995066 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:17:59.088683 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:59.088648 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3898ce0c-61e4-4888-9e54-3185c86a5326" path="/var/lib/kubelet/pods/3898ce0c-61e4-4888-9e54-3185c86a5326/volumes" Oct 31 00:17:59.166240 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:59.166159 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:17:59.303543 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:59.303452 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-92e375-openshift-default-6987df58bc-jzxns"] Oct 31 00:17:59.304437 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:59.304409 4927 cpu_manager.go:414] "containerMap: removing container" podUID="687684ad-5b0d-493a-9fa4-cbbe640229fe" containerName="payload-pre-processing" Oct 31 00:17:59.304437 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:59.304433 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="687684ad-5b0d-493a-9fa4-cbbe640229fe" containerName="payload-pre-processing" Oct 31 00:17:59.304665 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:59.304466 4927 cpu_manager.go:414] "containerMap: removing container" podUID="3898ce0c-61e4-4888-9e54-3185c86a5326" containerName="istio-proxy" Oct 31 00:17:59.304665 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:59.304474 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3898ce0c-61e4-4888-9e54-3185c86a5326" containerName="istio-proxy" Oct 31 00:17:59.304665 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:59.304487 4927 cpu_manager.go:414] "containerMap: removing container" podUID="59ccea66-1d77-43cd-bd56-e88b44a5e621" containerName="payload-processing" Oct 31 00:17:59.304665 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:59.304494 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59ccea66-1d77-43cd-bd56-e88b44a5e621" containerName="payload-processing" Oct 31 00:17:59.304665 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:59.304534 4927 cpu_manager.go:414] "containerMap: removing container" podUID="3898ce0c-61e4-4888-9e54-3185c86a5326" containerName="istio-proxy" Oct 31 00:17:59.304665 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:59.304540 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3898ce0c-61e4-4888-9e54-3185c86a5326" containerName="istio-proxy" Oct 31 00:17:59.304665 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:59.304552 4927 cpu_manager.go:414] "containerMap: removing container" podUID="3b669d7f-c767-4c4e-a524-6e774fc69034" containerName="istio-proxy" Oct 31 00:17:59.304665 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:59.304557 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b669d7f-c767-4c4e-a524-6e774fc69034" containerName="istio-proxy" Oct 31 00:17:59.305089 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:59.304683 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="687684ad-5b0d-493a-9fa4-cbbe640229fe" containerName="payload-pre-processing" Oct 31 00:17:59.305089 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:59.304694 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b669d7f-c767-4c4e-a524-6e774fc69034" containerName="istio-proxy" Oct 31 00:17:59.305089 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:59.304702 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="3898ce0c-61e4-4888-9e54-3185c86a5326" containerName="istio-proxy" Oct 31 00:17:59.305089 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:59.304707 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="59ccea66-1d77-43cd-bd56-e88b44a5e621" containerName="payload-processing" Oct 31 00:17:59.305089 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:59.304716 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="3898ce0c-61e4-4888-9e54-3185c86a5326" containerName="istio-proxy" Oct 31 00:17:59.309927 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:59.309906 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-92e375-openshift-default-6987df58bc-jzxns" Oct 31 00:17:59.391390 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:59.374954 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0802e5e7-e2ef-4a19-8663-6ba9b29ec0c2-istio-envoy\") pod \"e2e-api-a-92e375-openshift-default-6987df58bc-jzxns\" (UID: \"0802e5e7-e2ef-4a19-8663-6ba9b29ec0c2\") " pod="openshift-ingress/e2e-api-a-92e375-openshift-default-6987df58bc-jzxns" Oct 31 00:17:59.420463 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:59.412295 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0802e5e7-e2ef-4a19-8663-6ba9b29ec0c2-credential-socket\") pod \"e2e-api-a-92e375-openshift-default-6987df58bc-jzxns\" (UID: \"0802e5e7-e2ef-4a19-8663-6ba9b29ec0c2\") " pod="openshift-ingress/e2e-api-a-92e375-openshift-default-6987df58bc-jzxns" Oct 31 00:17:59.420843 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:59.420802 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0802e5e7-e2ef-4a19-8663-6ba9b29ec0c2-istio-podinfo\") pod \"e2e-api-a-92e375-openshift-default-6987df58bc-jzxns\" (UID: \"0802e5e7-e2ef-4a19-8663-6ba9b29ec0c2\") " pod="openshift-ingress/e2e-api-a-92e375-openshift-default-6987df58bc-jzxns" Oct 31 00:17:59.428573 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:59.428528 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0802e5e7-e2ef-4a19-8663-6ba9b29ec0c2-istiod-ca-cert\") pod \"e2e-api-a-92e375-openshift-default-6987df58bc-jzxns\" (UID: \"0802e5e7-e2ef-4a19-8663-6ba9b29ec0c2\") " pod="openshift-ingress/e2e-api-a-92e375-openshift-default-6987df58bc-jzxns" Oct 31 00:17:59.463388 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:59.449638 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-92e375-openshift-default-dockercfg-bzdx2\"" Oct 31 00:17:59.463388 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:59.458515 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0802e5e7-e2ef-4a19-8663-6ba9b29ec0c2-workload-socket\") pod \"e2e-api-a-92e375-openshift-default-6987df58bc-jzxns\" (UID: \"0802e5e7-e2ef-4a19-8663-6ba9b29ec0c2\") " pod="openshift-ingress/e2e-api-a-92e375-openshift-default-6987df58bc-jzxns" Oct 31 00:17:59.463388 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:59.458729 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6pxmc\" (UniqueName: \"kubernetes.io/projected/0802e5e7-e2ef-4a19-8663-6ba9b29ec0c2-kube-api-access-6pxmc\") pod \"e2e-api-a-92e375-openshift-default-6987df58bc-jzxns\" (UID: \"0802e5e7-e2ef-4a19-8663-6ba9b29ec0c2\") " pod="openshift-ingress/e2e-api-a-92e375-openshift-default-6987df58bc-jzxns" Oct 31 00:17:59.463388 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:59.458763 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0802e5e7-e2ef-4a19-8663-6ba9b29ec0c2-istio-data\") pod \"e2e-api-a-92e375-openshift-default-6987df58bc-jzxns\" (UID: \"0802e5e7-e2ef-4a19-8663-6ba9b29ec0c2\") " pod="openshift-ingress/e2e-api-a-92e375-openshift-default-6987df58bc-jzxns" Oct 31 00:17:59.463388 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:59.458788 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0802e5e7-e2ef-4a19-8663-6ba9b29ec0c2-istio-token\") pod \"e2e-api-a-92e375-openshift-default-6987df58bc-jzxns\" (UID: \"0802e5e7-e2ef-4a19-8663-6ba9b29ec0c2\") " pod="openshift-ingress/e2e-api-a-92e375-openshift-default-6987df58bc-jzxns" Oct 31 00:17:59.463388 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:59.458848 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0802e5e7-e2ef-4a19-8663-6ba9b29ec0c2-workload-certs\") pod \"e2e-api-a-92e375-openshift-default-6987df58bc-jzxns\" (UID: \"0802e5e7-e2ef-4a19-8663-6ba9b29ec0c2\") " pod="openshift-ingress/e2e-api-a-92e375-openshift-default-6987df58bc-jzxns" Oct 31 00:17:59.527776 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:59.527732 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:17:59.575453 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:59.561233 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6pxmc\" (UniqueName: \"kubernetes.io/projected/0802e5e7-e2ef-4a19-8663-6ba9b29ec0c2-kube-api-access-6pxmc\") pod \"e2e-api-a-92e375-openshift-default-6987df58bc-jzxns\" (UID: \"0802e5e7-e2ef-4a19-8663-6ba9b29ec0c2\") " pod="openshift-ingress/e2e-api-a-92e375-openshift-default-6987df58bc-jzxns" Oct 31 00:17:59.616605 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:59.616573 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0802e5e7-e2ef-4a19-8663-6ba9b29ec0c2-istio-data\") pod \"e2e-api-a-92e375-openshift-default-6987df58bc-jzxns\" (UID: \"0802e5e7-e2ef-4a19-8663-6ba9b29ec0c2\") " pod="openshift-ingress/e2e-api-a-92e375-openshift-default-6987df58bc-jzxns" Oct 31 00:17:59.621281 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:59.621252 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0802e5e7-e2ef-4a19-8663-6ba9b29ec0c2-istio-token\") pod \"e2e-api-a-92e375-openshift-default-6987df58bc-jzxns\" (UID: \"0802e5e7-e2ef-4a19-8663-6ba9b29ec0c2\") " pod="openshift-ingress/e2e-api-a-92e375-openshift-default-6987df58bc-jzxns" Oct 31 00:17:59.623560 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:59.623533 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0802e5e7-e2ef-4a19-8663-6ba9b29ec0c2-workload-certs\") pod \"e2e-api-a-92e375-openshift-default-6987df58bc-jzxns\" (UID: \"0802e5e7-e2ef-4a19-8663-6ba9b29ec0c2\") " pod="openshift-ingress/e2e-api-a-92e375-openshift-default-6987df58bc-jzxns" Oct 31 00:17:59.624255 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:59.624234 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0802e5e7-e2ef-4a19-8663-6ba9b29ec0c2-istio-envoy\") pod \"e2e-api-a-92e375-openshift-default-6987df58bc-jzxns\" (UID: \"0802e5e7-e2ef-4a19-8663-6ba9b29ec0c2\") " pod="openshift-ingress/e2e-api-a-92e375-openshift-default-6987df58bc-jzxns" Oct 31 00:17:59.624480 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:59.624464 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0802e5e7-e2ef-4a19-8663-6ba9b29ec0c2-credential-socket\") pod \"e2e-api-a-92e375-openshift-default-6987df58bc-jzxns\" (UID: \"0802e5e7-e2ef-4a19-8663-6ba9b29ec0c2\") " pod="openshift-ingress/e2e-api-a-92e375-openshift-default-6987df58bc-jzxns" Oct 31 00:17:59.625208 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:59.625189 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0802e5e7-e2ef-4a19-8663-6ba9b29ec0c2-istio-podinfo\") pod \"e2e-api-a-92e375-openshift-default-6987df58bc-jzxns\" (UID: \"0802e5e7-e2ef-4a19-8663-6ba9b29ec0c2\") " pod="openshift-ingress/e2e-api-a-92e375-openshift-default-6987df58bc-jzxns" Oct 31 00:17:59.625768 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:59.625748 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0802e5e7-e2ef-4a19-8663-6ba9b29ec0c2-istiod-ca-cert\") pod \"e2e-api-a-92e375-openshift-default-6987df58bc-jzxns\" (UID: \"0802e5e7-e2ef-4a19-8663-6ba9b29ec0c2\") " pod="openshift-ingress/e2e-api-a-92e375-openshift-default-6987df58bc-jzxns" Oct 31 00:17:59.626606 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:59.626587 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0802e5e7-e2ef-4a19-8663-6ba9b29ec0c2-workload-socket\") pod \"e2e-api-a-92e375-openshift-default-6987df58bc-jzxns\" (UID: \"0802e5e7-e2ef-4a19-8663-6ba9b29ec0c2\") " pod="openshift-ingress/e2e-api-a-92e375-openshift-default-6987df58bc-jzxns" Oct 31 00:17:59.627345 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:59.624066 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0802e5e7-e2ef-4a19-8663-6ba9b29ec0c2-workload-certs\") pod \"e2e-api-a-92e375-openshift-default-6987df58bc-jzxns\" (UID: \"0802e5e7-e2ef-4a19-8663-6ba9b29ec0c2\") " pod="openshift-ingress/e2e-api-a-92e375-openshift-default-6987df58bc-jzxns" Oct 31 00:17:59.627773 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:59.626525 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0802e5e7-e2ef-4a19-8663-6ba9b29ec0c2-istiod-ca-cert\") pod \"e2e-api-a-92e375-openshift-default-6987df58bc-jzxns\" (UID: \"0802e5e7-e2ef-4a19-8663-6ba9b29ec0c2\") " pod="openshift-ingress/e2e-api-a-92e375-openshift-default-6987df58bc-jzxns" Oct 31 00:17:59.627926 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:59.625127 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0802e5e7-e2ef-4a19-8663-6ba9b29ec0c2-credential-socket\") pod \"e2e-api-a-92e375-openshift-default-6987df58bc-jzxns\" (UID: \"0802e5e7-e2ef-4a19-8663-6ba9b29ec0c2\") " pod="openshift-ingress/e2e-api-a-92e375-openshift-default-6987df58bc-jzxns" Oct 31 00:17:59.628025 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:59.621191 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0802e5e7-e2ef-4a19-8663-6ba9b29ec0c2-istio-data\") pod \"e2e-api-a-92e375-openshift-default-6987df58bc-jzxns\" (UID: \"0802e5e7-e2ef-4a19-8663-6ba9b29ec0c2\") " pod="openshift-ingress/e2e-api-a-92e375-openshift-default-6987df58bc-jzxns" Oct 31 00:17:59.628122 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:59.627041 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0802e5e7-e2ef-4a19-8663-6ba9b29ec0c2-workload-socket\") pod \"e2e-api-a-92e375-openshift-default-6987df58bc-jzxns\" (UID: \"0802e5e7-e2ef-4a19-8663-6ba9b29ec0c2\") " pod="openshift-ingress/e2e-api-a-92e375-openshift-default-6987df58bc-jzxns" Oct 31 00:17:59.628208 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:59.627714 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0802e5e7-e2ef-4a19-8663-6ba9b29ec0c2-istio-envoy\") pod \"e2e-api-a-92e375-openshift-default-6987df58bc-jzxns\" (UID: \"0802e5e7-e2ef-4a19-8663-6ba9b29ec0c2\") " pod="openshift-ingress/e2e-api-a-92e375-openshift-default-6987df58bc-jzxns" Oct 31 00:17:59.631804 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:59.631782 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-92e375-openshift-default-6987df58bc-jzxns"] Oct 31 00:17:59.665840 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:59.664557 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0802e5e7-e2ef-4a19-8663-6ba9b29ec0c2-istio-podinfo\") pod \"e2e-api-a-92e375-openshift-default-6987df58bc-jzxns\" (UID: \"0802e5e7-e2ef-4a19-8663-6ba9b29ec0c2\") " pod="openshift-ingress/e2e-api-a-92e375-openshift-default-6987df58bc-jzxns" Oct 31 00:17:59.682805 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:59.682764 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6pxmc\" (UniqueName: \"kubernetes.io/projected/0802e5e7-e2ef-4a19-8663-6ba9b29ec0c2-kube-api-access-6pxmc\") pod \"e2e-api-a-92e375-openshift-default-6987df58bc-jzxns\" (UID: \"0802e5e7-e2ef-4a19-8663-6ba9b29ec0c2\") " pod="openshift-ingress/e2e-api-a-92e375-openshift-default-6987df58bc-jzxns" Oct 31 00:17:59.700388 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:59.699871 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0802e5e7-e2ef-4a19-8663-6ba9b29ec0c2-istio-token\") pod \"e2e-api-a-92e375-openshift-default-6987df58bc-jzxns\" (UID: \"0802e5e7-e2ef-4a19-8663-6ba9b29ec0c2\") " pod="openshift-ingress/e2e-api-a-92e375-openshift-default-6987df58bc-jzxns" Oct 31 00:17:59.778775 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:59.778044 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:17:59.935105 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:17:59.935074 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-92e375-openshift-default-6987df58bc-jzxns" Oct 31 00:18:00.381650 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:00.381618 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-92e375-openshift-default-6987df58bc-jzxns"] Oct 31 00:18:00.381989 ip-10-0-141-180 kubenswrapper[4927]: W1031 00:18:00.381653 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0802e5e7_e2ef_4a19_8663_6ba9b29ec0c2.slice/crio-a47efad15df9bcb79f44211b1ddf40865d3a22e9712944dbb0dfbbbc77106e8a WatchSource:0}: Error finding container a47efad15df9bcb79f44211b1ddf40865d3a22e9712944dbb0dfbbbc77106e8a: Status 404 returned error can't find the container with id a47efad15df9bcb79f44211b1ddf40865d3a22e9712944dbb0dfbbbc77106e8a Oct 31 00:18:00.384456 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:00.384423 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:18:00.384578 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:00.384484 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:18:00.384578 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:00.384515 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:18:01.009181 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:01.009133 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:18:01.118654 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:01.118617 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-92e375-openshift-default-6987df58bc-jzxns" event={"ID":"0802e5e7-e2ef-4a19-8663-6ba9b29ec0c2","Type":"ContainerStarted","Data":"678ab560f58ffa4cbda83d6e146dea152a309dabcc7fbecda4d40d0b9cf12dc7"} Oct 31 00:18:01.118839 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:01.118674 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-92e375-openshift-default-6987df58bc-jzxns" event={"ID":"0802e5e7-e2ef-4a19-8663-6ba9b29ec0c2","Type":"ContainerStarted","Data":"a47efad15df9bcb79f44211b1ddf40865d3a22e9712944dbb0dfbbbc77106e8a"} Oct 31 00:18:01.245238 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:01.245176 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-92e375-openshift-default-6987df58bc-jzxns" podStartSLOduration=2.245155541 podStartE2EDuration="2.245155541s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:17:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:18:01.243148275 +0000 UTC m=+2194.279479118" watchObservedRunningTime="2026-10-31 00:18:01.245155541 +0000 UTC m=+2194.281486365" Oct 31 00:18:01.316646 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:01.316562 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:18:01.935455 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:01.935406 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-92e375-openshift-default-6987df58bc-jzxns" Oct 31 00:18:01.942263 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:01.942233 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-92e375-openshift-default-6987df58bc-jzxns" Oct 31 00:18:02.126833 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:02.126796 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-92e375-openshift-default-6987df58bc-jzxns" Oct 31 00:18:02.127873 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:02.127850 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-92e375-openshift-default-6987df58bc-jzxns" Oct 31 00:18:02.130210 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:02.130180 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:18:02.461363 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:02.460739 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:18:02.561355 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:02.559884 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-76b073-68467cb4b7-pf7qq"] Oct 31 00:18:02.702355 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:02.699168 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-76b073-68467cb4b7-pf7qq" Oct 31 00:18:02.825830 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:02.823488 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-76b073-dockercfg-kh889\"" Oct 31 00:18:02.831271 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:02.830191 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-76b073\"" Oct 31 00:18:02.856414 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:02.856381 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-76b073-68467cb4b7-pf7qq"] Oct 31 00:18:02.882652 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:02.882620 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-76b073-78787bc78-d6l68"] Oct 31 00:18:02.887901 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:02.887864 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e63da6f1-8b8f-451d-84c1-9b39411a2a59-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-76b073-68467cb4b7-pf7qq\" (UID: \"e63da6f1-8b8f-451d-84c1-9b39411a2a59\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-76b073-68467cb4b7-pf7qq" Oct 31 00:18:02.888066 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:02.887912 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-245lg\" (UniqueName: \"kubernetes.io/projected/e63da6f1-8b8f-451d-84c1-9b39411a2a59-kube-api-access-245lg\") pod \"payload-pre-processing-e2e-inf-a-76b073-68467cb4b7-pf7qq\" (UID: \"e63da6f1-8b8f-451d-84c1-9b39411a2a59\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-76b073-68467cb4b7-pf7qq" Oct 31 00:18:02.890282 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:02.888971 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-76b073-78787bc78-d6l68" Oct 31 00:18:02.899591 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:02.898858 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-76b073-78787bc78-d6l68"] Oct 31 00:18:02.989555 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:02.989470 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e63da6f1-8b8f-451d-84c1-9b39411a2a59-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-76b073-68467cb4b7-pf7qq\" (UID: \"e63da6f1-8b8f-451d-84c1-9b39411a2a59\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-76b073-68467cb4b7-pf7qq" Oct 31 00:18:02.989555 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:02.989518 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-245lg\" (UniqueName: \"kubernetes.io/projected/e63da6f1-8b8f-451d-84c1-9b39411a2a59-kube-api-access-245lg\") pod \"payload-pre-processing-e2e-inf-a-76b073-68467cb4b7-pf7qq\" (UID: \"e63da6f1-8b8f-451d-84c1-9b39411a2a59\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-76b073-68467cb4b7-pf7qq" Oct 31 00:18:02.990080 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:02.989644 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f69c734-4a5f-427d-a682-c5bc7ebaa0b1-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-76b073-78787bc78-d6l68\" (UID: \"0f69c734-4a5f-427d-a682-c5bc7ebaa0b1\") " pod="openshift-ingress/payload-processing-e2e-inf-a-76b073-78787bc78-d6l68" Oct 31 00:18:02.990080 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:02.989801 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fzrsk\" (UniqueName: \"kubernetes.io/projected/0f69c734-4a5f-427d-a682-c5bc7ebaa0b1-kube-api-access-fzrsk\") pod \"payload-processing-e2e-inf-a-76b073-78787bc78-d6l68\" (UID: \"0f69c734-4a5f-427d-a682-c5bc7ebaa0b1\") " pod="openshift-ingress/payload-processing-e2e-inf-a-76b073-78787bc78-d6l68" Oct 31 00:18:02.990266 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:02.990242 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e63da6f1-8b8f-451d-84c1-9b39411a2a59-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-76b073-68467cb4b7-pf7qq\" (UID: \"e63da6f1-8b8f-451d-84c1-9b39411a2a59\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-76b073-68467cb4b7-pf7qq" Oct 31 00:18:03.052822 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:03.052791 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-245lg\" (UniqueName: \"kubernetes.io/projected/e63da6f1-8b8f-451d-84c1-9b39411a2a59-kube-api-access-245lg\") pod \"payload-pre-processing-e2e-inf-a-76b073-68467cb4b7-pf7qq\" (UID: \"e63da6f1-8b8f-451d-84c1-9b39411a2a59\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-76b073-68467cb4b7-pf7qq" Oct 31 00:18:03.091244 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:03.091209 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fzrsk\" (UniqueName: \"kubernetes.io/projected/0f69c734-4a5f-427d-a682-c5bc7ebaa0b1-kube-api-access-fzrsk\") pod \"payload-processing-e2e-inf-a-76b073-78787bc78-d6l68\" (UID: \"0f69c734-4a5f-427d-a682-c5bc7ebaa0b1\") " pod="openshift-ingress/payload-processing-e2e-inf-a-76b073-78787bc78-d6l68" Oct 31 00:18:03.091474 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:03.091458 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f69c734-4a5f-427d-a682-c5bc7ebaa0b1-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-76b073-78787bc78-d6l68\" (UID: \"0f69c734-4a5f-427d-a682-c5bc7ebaa0b1\") " pod="openshift-ingress/payload-processing-e2e-inf-a-76b073-78787bc78-d6l68" Oct 31 00:18:03.091764 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:03.091747 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-4a3dba-openshift-default-dc9664475-zxb5s" Oct 31 00:18:03.092357 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:03.092307 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f69c734-4a5f-427d-a682-c5bc7ebaa0b1-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-76b073-78787bc78-d6l68\" (UID: \"0f69c734-4a5f-427d-a682-c5bc7ebaa0b1\") " pod="openshift-ingress/payload-processing-e2e-inf-a-76b073-78787bc78-d6l68" Oct 31 00:18:03.138950 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:03.138908 4927 generic.go:356] "Generic (PLEG): container finished" podID="1b1c447d-a95f-4edb-9c9d-8987eea5eaf4" containerID="963a5aa1c641ed6f16929af7e3016f135429161948921c103d3981e14fd8ab1d" exitCode=0 Oct 31 00:18:03.139163 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:03.138965 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-4a3dba-openshift-default-dc9664475-zxb5s" Oct 31 00:18:03.139163 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:03.138974 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-4a3dba-openshift-default-dc9664475-zxb5s" event={"ID":"1b1c447d-a95f-4edb-9c9d-8987eea5eaf4","Type":"ContainerDied","Data":"963a5aa1c641ed6f16929af7e3016f135429161948921c103d3981e14fd8ab1d"} Oct 31 00:18:03.139163 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:03.139048 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-4a3dba-openshift-default-dc9664475-zxb5s" event={"ID":"1b1c447d-a95f-4edb-9c9d-8987eea5eaf4","Type":"ContainerDied","Data":"422ea89a0d08389286efbc4d7e02c55e7ade5829834719ea8e7c68849e422dff"} Oct 31 00:18:03.139163 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:03.139079 4927 scope.go:122] "RemoveContainer" containerID="963a5aa1c641ed6f16929af7e3016f135429161948921c103d3981e14fd8ab1d" Oct 31 00:18:03.153721 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:03.153695 4927 scope.go:122] "RemoveContainer" containerID="d750bd6d9be7f3a0cecd96781cede0a31a4438375a563f22bd141188270130da" Oct 31 00:18:03.153721 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:03.153713 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fzrsk\" (UniqueName: \"kubernetes.io/projected/0f69c734-4a5f-427d-a682-c5bc7ebaa0b1-kube-api-access-fzrsk\") pod \"payload-processing-e2e-inf-a-76b073-78787bc78-d6l68\" (UID: \"0f69c734-4a5f-427d-a682-c5bc7ebaa0b1\") " pod="openshift-ingress/payload-processing-e2e-inf-a-76b073-78787bc78-d6l68" Oct 31 00:18:03.172934 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:03.172898 4927 scope.go:122] "RemoveContainer" containerID="963a5aa1c641ed6f16929af7e3016f135429161948921c103d3981e14fd8ab1d" Oct 31 00:18:03.173283 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:18:03.173255 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"963a5aa1c641ed6f16929af7e3016f135429161948921c103d3981e14fd8ab1d\": container with ID starting with 963a5aa1c641ed6f16929af7e3016f135429161948921c103d3981e14fd8ab1d not found: ID does not exist" containerID="963a5aa1c641ed6f16929af7e3016f135429161948921c103d3981e14fd8ab1d" Oct 31 00:18:03.173465 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:03.173290 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"963a5aa1c641ed6f16929af7e3016f135429161948921c103d3981e14fd8ab1d"} err="failed to get container status \"963a5aa1c641ed6f16929af7e3016f135429161948921c103d3981e14fd8ab1d\": rpc error: code = NotFound desc = could not find container \"963a5aa1c641ed6f16929af7e3016f135429161948921c103d3981e14fd8ab1d\": container with ID starting with 963a5aa1c641ed6f16929af7e3016f135429161948921c103d3981e14fd8ab1d not found: ID does not exist" Oct 31 00:18:03.173465 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:03.173309 4927 scope.go:122] "RemoveContainer" containerID="d750bd6d9be7f3a0cecd96781cede0a31a4438375a563f22bd141188270130da" Oct 31 00:18:03.173635 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:18:03.173615 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d750bd6d9be7f3a0cecd96781cede0a31a4438375a563f22bd141188270130da\": container with ID starting with d750bd6d9be7f3a0cecd96781cede0a31a4438375a563f22bd141188270130da not found: ID does not exist" containerID="d750bd6d9be7f3a0cecd96781cede0a31a4438375a563f22bd141188270130da" Oct 31 00:18:03.173678 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:03.173642 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d750bd6d9be7f3a0cecd96781cede0a31a4438375a563f22bd141188270130da"} err="failed to get container status \"d750bd6d9be7f3a0cecd96781cede0a31a4438375a563f22bd141188270130da\": rpc error: code = NotFound desc = could not find container \"d750bd6d9be7f3a0cecd96781cede0a31a4438375a563f22bd141188270130da\": container with ID starting with d750bd6d9be7f3a0cecd96781cede0a31a4438375a563f22bd141188270130da not found: ID does not exist" Oct 31 00:18:03.178920 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:03.178898 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-76b073-68467cb4b7-pf7qq" Oct 31 00:18:03.192926 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:03.192787 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b1c447d-a95f-4edb-9c9d-8987eea5eaf4-kube-api-access-j64rz\" (UniqueName: \"kubernetes.io/projected/1b1c447d-a95f-4edb-9c9d-8987eea5eaf4-kube-api-access-j64rz\") pod \"1b1c447d-a95f-4edb-9c9d-8987eea5eaf4\" (UID: \"1b1c447d-a95f-4edb-9c9d-8987eea5eaf4\") " Oct 31 00:18:03.192926 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:03.192885 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b1c447d-a95f-4edb-9c9d-8987eea5eaf4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b1c447d-a95f-4edb-9c9d-8987eea5eaf4-workload-socket\") pod \"1b1c447d-a95f-4edb-9c9d-8987eea5eaf4\" (UID: \"1b1c447d-a95f-4edb-9c9d-8987eea5eaf4\") " Oct 31 00:18:03.193094 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:03.193052 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1b1c447d-a95f-4edb-9c9d-8987eea5eaf4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b1c447d-a95f-4edb-9c9d-8987eea5eaf4-istio-podinfo\") pod \"1b1c447d-a95f-4edb-9c9d-8987eea5eaf4\" (UID: \"1b1c447d-a95f-4edb-9c9d-8987eea5eaf4\") " Oct 31 00:18:03.193094 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:03.193083 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b1c447d-a95f-4edb-9c9d-8987eea5eaf4-istio-token\" (UniqueName: \"kubernetes.io/projected/1b1c447d-a95f-4edb-9c9d-8987eea5eaf4-istio-token\") pod \"1b1c447d-a95f-4edb-9c9d-8987eea5eaf4\" (UID: \"1b1c447d-a95f-4edb-9c9d-8987eea5eaf4\") " Oct 31 00:18:03.193247 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:03.193230 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b1c447d-a95f-4edb-9c9d-8987eea5eaf4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b1c447d-a95f-4edb-9c9d-8987eea5eaf4-workload-certs\") pod \"1b1c447d-a95f-4edb-9c9d-8987eea5eaf4\" (UID: \"1b1c447d-a95f-4edb-9c9d-8987eea5eaf4\") " Oct 31 00:18:03.193312 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:03.193198 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b1c447d-a95f-4edb-9c9d-8987eea5eaf4-workload-socket" pod "1b1c447d-a95f-4edb-9c9d-8987eea5eaf4" (UID: "1b1c447d-a95f-4edb-9c9d-8987eea5eaf4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:18:03.193312 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:03.193282 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b1c447d-a95f-4edb-9c9d-8987eea5eaf4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b1c447d-a95f-4edb-9c9d-8987eea5eaf4-istio-envoy\") pod \"1b1c447d-a95f-4edb-9c9d-8987eea5eaf4\" (UID: \"1b1c447d-a95f-4edb-9c9d-8987eea5eaf4\") " Oct 31 00:18:03.193466 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:03.193309 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b1c447d-a95f-4edb-9c9d-8987eea5eaf4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b1c447d-a95f-4edb-9c9d-8987eea5eaf4-istiod-ca-cert\") pod \"1b1c447d-a95f-4edb-9c9d-8987eea5eaf4\" (UID: \"1b1c447d-a95f-4edb-9c9d-8987eea5eaf4\") " Oct 31 00:18:03.193466 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:03.193353 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b1c447d-a95f-4edb-9c9d-8987eea5eaf4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b1c447d-a95f-4edb-9c9d-8987eea5eaf4-credential-socket\") pod \"1b1c447d-a95f-4edb-9c9d-8987eea5eaf4\" (UID: \"1b1c447d-a95f-4edb-9c9d-8987eea5eaf4\") " Oct 31 00:18:03.193466 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:03.193390 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b1c447d-a95f-4edb-9c9d-8987eea5eaf4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b1c447d-a95f-4edb-9c9d-8987eea5eaf4-istio-data\") pod \"1b1c447d-a95f-4edb-9c9d-8987eea5eaf4\" (UID: \"1b1c447d-a95f-4edb-9c9d-8987eea5eaf4\") " Oct 31 00:18:03.193835 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:03.193806 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b1c447d-a95f-4edb-9c9d-8987eea5eaf4-credential-socket" pod "1b1c447d-a95f-4edb-9c9d-8987eea5eaf4" (UID: "1b1c447d-a95f-4edb-9c9d-8987eea5eaf4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:18:03.194058 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:03.194035 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b1c447d-a95f-4edb-9c9d-8987eea5eaf4-istio-data" pod "1b1c447d-a95f-4edb-9c9d-8987eea5eaf4" (UID: "1b1c447d-a95f-4edb-9c9d-8987eea5eaf4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:18:03.194246 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:03.194227 4927 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b1c447d-a95f-4edb-9c9d-8987eea5eaf4-credential-socket\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:18:03.194315 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:03.194253 4927 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b1c447d-a95f-4edb-9c9d-8987eea5eaf4-istio-data\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:18:03.194315 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:03.194271 4927 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b1c447d-a95f-4edb-9c9d-8987eea5eaf4-workload-socket\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:18:03.195400 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:03.195378 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b1c447d-a95f-4edb-9c9d-8987eea5eaf4-kube-api-access-j64rz" pod "1b1c447d-a95f-4edb-9c9d-8987eea5eaf4" (UID: "1b1c447d-a95f-4edb-9c9d-8987eea5eaf4"). InnerVolumeSpecName "kube-api-access-j64rz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:18:03.195494 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:03.195472 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b1c447d-a95f-4edb-9c9d-8987eea5eaf4-workload-certs" pod "1b1c447d-a95f-4edb-9c9d-8987eea5eaf4" (UID: "1b1c447d-a95f-4edb-9c9d-8987eea5eaf4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:18:03.195641 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:03.195500 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b1c447d-a95f-4edb-9c9d-8987eea5eaf4-istiod-ca-cert" pod "1b1c447d-a95f-4edb-9c9d-8987eea5eaf4" (UID: "1b1c447d-a95f-4edb-9c9d-8987eea5eaf4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:18:03.195846 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:03.195826 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1b1c447d-a95f-4edb-9c9d-8987eea5eaf4-istio-podinfo" pod "1b1c447d-a95f-4edb-9c9d-8987eea5eaf4" (UID: "1b1c447d-a95f-4edb-9c9d-8987eea5eaf4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:18:03.195981 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:03.195933 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b1c447d-a95f-4edb-9c9d-8987eea5eaf4-istio-token" pod "1b1c447d-a95f-4edb-9c9d-8987eea5eaf4" (UID: "1b1c447d-a95f-4edb-9c9d-8987eea5eaf4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:18:03.197071 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:03.197035 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b1c447d-a95f-4edb-9c9d-8987eea5eaf4-istio-envoy" pod "1b1c447d-a95f-4edb-9c9d-8987eea5eaf4" (UID: "1b1c447d-a95f-4edb-9c9d-8987eea5eaf4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:18:03.233402 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:03.229107 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-76b073-78787bc78-d6l68" Oct 31 00:18:03.296108 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:03.296058 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j64rz\" (UniqueName: \"kubernetes.io/projected/1b1c447d-a95f-4edb-9c9d-8987eea5eaf4-kube-api-access-j64rz\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:18:03.296108 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:03.296093 4927 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b1c447d-a95f-4edb-9c9d-8987eea5eaf4-istio-podinfo\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:18:03.296108 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:03.296113 4927 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b1c447d-a95f-4edb-9c9d-8987eea5eaf4-istio-token\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:18:03.296108 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:03.296126 4927 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b1c447d-a95f-4edb-9c9d-8987eea5eaf4-workload-certs\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:18:03.296518 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:03.296142 4927 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b1c447d-a95f-4edb-9c9d-8987eea5eaf4-istio-envoy\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:18:03.296518 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:03.296155 4927 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b1c447d-a95f-4edb-9c9d-8987eea5eaf4-istiod-ca-cert\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:18:03.566247 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:03.566162 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-4a3dba-openshift-default-dc9664475-zxb5s"] Oct 31 00:18:03.578595 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:03.578559 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-4a3dba-openshift-default-dc9664475-zxb5s"] Oct 31 00:18:03.585408 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:03.585375 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-76b073-68467cb4b7-pf7qq"] Oct 31 00:18:03.617829 ip-10-0-141-180 kubenswrapper[4927]: W1031 00:18:03.617798 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode63da6f1_8b8f_451d_84c1_9b39411a2a59.slice/crio-26b10c16016249d7950f706bd67acb85253a2c34936f3b572f15f4c78721956f WatchSource:0}: Error finding container 26b10c16016249d7950f706bd67acb85253a2c34936f3b572f15f4c78721956f: Status 404 returned error can't find the container with id 26b10c16016249d7950f706bd67acb85253a2c34936f3b572f15f4c78721956f Oct 31 00:18:03.682667 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:03.682636 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-76b073-78787bc78-d6l68"] Oct 31 00:18:03.683216 ip-10-0-141-180 kubenswrapper[4927]: W1031 00:18:03.683190 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f69c734_4a5f_427d_a682_c5bc7ebaa0b1.slice/crio-b22e4e3b9a40a5c38a9d4d453b099218e67adb1307b1c52d0d5c3f04d8686448 WatchSource:0}: Error finding container b22e4e3b9a40a5c38a9d4d453b099218e67adb1307b1c52d0d5c3f04d8686448: Status 404 returned error can't find the container with id b22e4e3b9a40a5c38a9d4d453b099218e67adb1307b1c52d0d5c3f04d8686448 Oct 31 00:18:04.061119 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:04.061080 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:18:04.149605 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:04.149561 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-76b073-78787bc78-d6l68" event={"ID":"0f69c734-4a5f-427d-a682-c5bc7ebaa0b1","Type":"ContainerStarted","Data":"25a5633a1341a3d4a982ecdd62e9411f191df7f94a0641dd2fcdd865c0a3a885"} Oct 31 00:18:04.149791 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:04.149632 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-76b073-78787bc78-d6l68" Oct 31 00:18:04.149791 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:04.149654 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-76b073-78787bc78-d6l68" event={"ID":"0f69c734-4a5f-427d-a682-c5bc7ebaa0b1","Type":"ContainerStarted","Data":"b22e4e3b9a40a5c38a9d4d453b099218e67adb1307b1c52d0d5c3f04d8686448"} Oct 31 00:18:04.152855 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:04.152818 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-76b073-68467cb4b7-pf7qq" event={"ID":"e63da6f1-8b8f-451d-84c1-9b39411a2a59","Type":"ContainerStarted","Data":"929c28a1f7a209a0d96d173f92e660232d2acf9517c8bf272072c0ff64635e2c"} Oct 31 00:18:04.153004 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:04.152881 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-76b073-68467cb4b7-pf7qq" event={"ID":"e63da6f1-8b8f-451d-84c1-9b39411a2a59","Type":"ContainerStarted","Data":"26b10c16016249d7950f706bd67acb85253a2c34936f3b572f15f4c78721956f"} Oct 31 00:18:04.153073 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:04.153029 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-76b073-68467cb4b7-pf7qq" Oct 31 00:18:04.220263 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:04.220222 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:18:04.306918 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:04.306856 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-76b073-78787bc78-d6l68" podStartSLOduration=2.306838834 podStartE2EDuration="2.306838834s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:18:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:18:04.303469204 +0000 UTC m=+2197.339800063" watchObservedRunningTime="2026-10-31 00:18:04.306838834 +0000 UTC m=+2197.343169659" Oct 31 00:18:04.446870 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:04.446826 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:18:04.457084 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:04.456568 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-76b073-68467cb4b7-pf7qq" podStartSLOduration=2.456545575 podStartE2EDuration="2.456545575s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:18:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:18:04.44750867 +0000 UTC m=+2197.483839517" watchObservedRunningTime="2026-10-31 00:18:04.456545575 +0000 UTC m=+2197.492876400" Oct 31 00:18:05.089539 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:05.089505 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b1c447d-a95f-4edb-9c9d-8987eea5eaf4" path="/var/lib/kubelet/pods/1b1c447d-a95f-4edb-9c9d-8987eea5eaf4/volumes" Oct 31 00:18:06.257430 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:06.257398 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:18:07.595370 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:07.595248 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:18:07.750992 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:07.750954 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:18:08.961530 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:08.961490 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:18:09.399677 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:09.399592 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:18:09.423022 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:09.422986 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-92e375-6d87d5b947-4249b"] Oct 31 00:18:09.429120 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:09.429086 4927 cpu_manager.go:414] "containerMap: removing container" podUID="1b1c447d-a95f-4edb-9c9d-8987eea5eaf4" containerName="istio-proxy" Oct 31 00:18:09.433079 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:09.433049 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b1c447d-a95f-4edb-9c9d-8987eea5eaf4" containerName="istio-proxy" Oct 31 00:18:09.435291 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:09.435269 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b1c447d-a95f-4edb-9c9d-8987eea5eaf4" containerName="istio-proxy" Oct 31 00:18:09.435849 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:09.435831 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b1c447d-a95f-4edb-9c9d-8987eea5eaf4" containerName="istio-proxy" Oct 31 00:18:09.449915 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:09.449875 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-92e375-6d87d5b947-4249b" Oct 31 00:18:09.481259 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:09.481227 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-92e375-6d87d5b947-4249b"] Oct 31 00:18:09.581635 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:09.581595 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-92e375\"" Oct 31 00:18:09.582021 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:09.581997 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-92e375-dockercfg-hxzjh\"" Oct 31 00:18:09.646023 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:09.645990 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-92e375-75c9fbbc84-nt42r"] Oct 31 00:18:09.646539 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:09.646510 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qt6vd\" (UniqueName: \"kubernetes.io/projected/0be20326-c5ad-48ff-8126-f4a4e067a098-kube-api-access-qt6vd\") pod \"payload-pre-processing-e2e-api-a-92e375-6d87d5b947-4249b\" (UID: \"0be20326-c5ad-48ff-8126-f4a4e067a098\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-92e375-6d87d5b947-4249b" Oct 31 00:18:09.646739 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:09.646702 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0be20326-c5ad-48ff-8126-f4a4e067a098-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-92e375-6d87d5b947-4249b\" (UID: \"0be20326-c5ad-48ff-8126-f4a4e067a098\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-92e375-6d87d5b947-4249b" Oct 31 00:18:09.647742 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:09.647717 4927 cpu_manager.go:414] "containerMap: removing container" podUID="1b1c447d-a95f-4edb-9c9d-8987eea5eaf4" containerName="istio-proxy" Oct 31 00:18:09.647867 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:09.647854 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b1c447d-a95f-4edb-9c9d-8987eea5eaf4" containerName="istio-proxy" Oct 31 00:18:09.673599 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:09.670144 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-92e375-75c9fbbc84-nt42r" Oct 31 00:18:09.682749 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:09.682716 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-92e375-75c9fbbc84-nt42r"] Oct 31 00:18:09.747611 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:09.747569 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qt6vd\" (UniqueName: \"kubernetes.io/projected/0be20326-c5ad-48ff-8126-f4a4e067a098-kube-api-access-qt6vd\") pod \"payload-pre-processing-e2e-api-a-92e375-6d87d5b947-4249b\" (UID: \"0be20326-c5ad-48ff-8126-f4a4e067a098\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-92e375-6d87d5b947-4249b" Oct 31 00:18:09.747780 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:09.747700 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0be20326-c5ad-48ff-8126-f4a4e067a098-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-92e375-6d87d5b947-4249b\" (UID: \"0be20326-c5ad-48ff-8126-f4a4e067a098\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-92e375-6d87d5b947-4249b" Oct 31 00:18:09.748391 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:09.748370 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0be20326-c5ad-48ff-8126-f4a4e067a098-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-92e375-6d87d5b947-4249b\" (UID: \"0be20326-c5ad-48ff-8126-f4a4e067a098\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-92e375-6d87d5b947-4249b" Oct 31 00:18:09.810456 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:09.810423 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qt6vd\" (UniqueName: \"kubernetes.io/projected/0be20326-c5ad-48ff-8126-f4a4e067a098-kube-api-access-qt6vd\") pod \"payload-pre-processing-e2e-api-a-92e375-6d87d5b947-4249b\" (UID: \"0be20326-c5ad-48ff-8126-f4a4e067a098\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-92e375-6d87d5b947-4249b" Oct 31 00:18:09.848806 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:09.848772 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89af18c8-54a0-4dc6-a07b-07bd814ad172-plugins-config-volume\") pod \"payload-processing-e2e-api-a-92e375-75c9fbbc84-nt42r\" (UID: \"89af18c8-54a0-4dc6-a07b-07bd814ad172\") " pod="openshift-ingress/payload-processing-e2e-api-a-92e375-75c9fbbc84-nt42r" Oct 31 00:18:09.848966 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:09.848820 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qh6td\" (UniqueName: \"kubernetes.io/projected/89af18c8-54a0-4dc6-a07b-07bd814ad172-kube-api-access-qh6td\") pod \"payload-processing-e2e-api-a-92e375-75c9fbbc84-nt42r\" (UID: \"89af18c8-54a0-4dc6-a07b-07bd814ad172\") " pod="openshift-ingress/payload-processing-e2e-api-a-92e375-75c9fbbc84-nt42r" Oct 31 00:18:09.950152 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:09.950056 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89af18c8-54a0-4dc6-a07b-07bd814ad172-plugins-config-volume\") pod \"payload-processing-e2e-api-a-92e375-75c9fbbc84-nt42r\" (UID: \"89af18c8-54a0-4dc6-a07b-07bd814ad172\") " pod="openshift-ingress/payload-processing-e2e-api-a-92e375-75c9fbbc84-nt42r" Oct 31 00:18:09.950152 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:09.950114 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qh6td\" (UniqueName: \"kubernetes.io/projected/89af18c8-54a0-4dc6-a07b-07bd814ad172-kube-api-access-qh6td\") pod \"payload-processing-e2e-api-a-92e375-75c9fbbc84-nt42r\" (UID: \"89af18c8-54a0-4dc6-a07b-07bd814ad172\") " pod="openshift-ingress/payload-processing-e2e-api-a-92e375-75c9fbbc84-nt42r" Oct 31 00:18:09.951013 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:09.950980 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89af18c8-54a0-4dc6-a07b-07bd814ad172-plugins-config-volume\") pod \"payload-processing-e2e-api-a-92e375-75c9fbbc84-nt42r\" (UID: \"89af18c8-54a0-4dc6-a07b-07bd814ad172\") " pod="openshift-ingress/payload-processing-e2e-api-a-92e375-75c9fbbc84-nt42r" Oct 31 00:18:10.013278 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:10.013231 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qh6td\" (UniqueName: \"kubernetes.io/projected/89af18c8-54a0-4dc6-a07b-07bd814ad172-kube-api-access-qh6td\") pod \"payload-processing-e2e-api-a-92e375-75c9fbbc84-nt42r\" (UID: \"89af18c8-54a0-4dc6-a07b-07bd814ad172\") " pod="openshift-ingress/payload-processing-e2e-api-a-92e375-75c9fbbc84-nt42r" Oct 31 00:18:10.080059 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:10.080024 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-92e375-6d87d5b947-4249b" Oct 31 00:18:10.290469 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:10.290386 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-92e375-75c9fbbc84-nt42r" Oct 31 00:18:10.472287 ip-10-0-141-180 kubenswrapper[4927]: W1031 00:18:10.472256 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0be20326_c5ad_48ff_8126_f4a4e067a098.slice/crio-a2d490366ed6d9bc35c78afccb5a84daa1ab0bbf79bac75446f3de0046960b15 WatchSource:0}: Error finding container a2d490366ed6d9bc35c78afccb5a84daa1ab0bbf79bac75446f3de0046960b15: Status 404 returned error can't find the container with id a2d490366ed6d9bc35c78afccb5a84daa1ab0bbf79bac75446f3de0046960b15 Oct 31 00:18:10.472438 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:10.472323 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-92e375-6d87d5b947-4249b"] Oct 31 00:18:10.681050 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:10.681015 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-92e375-75c9fbbc84-nt42r"] Oct 31 00:18:10.681345 ip-10-0-141-180 kubenswrapper[4927]: W1031 00:18:10.681304 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89af18c8_54a0_4dc6_a07b_07bd814ad172.slice/crio-4ae0fd4c8daedb66927ee99df1bbf5564d3d4fb7abaa5e9d9b831d2431837714 WatchSource:0}: Error finding container 4ae0fd4c8daedb66927ee99df1bbf5564d3d4fb7abaa5e9d9b831d2431837714: Status 404 returned error can't find the container with id 4ae0fd4c8daedb66927ee99df1bbf5564d3d4fb7abaa5e9d9b831d2431837714 Oct 31 00:18:10.694287 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:10.694262 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85672d8b-zrbx4"] Oct 31 00:18:10.700102 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:10.700084 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85672d8b-zrbx4" Oct 31 00:18:10.753927 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:10.753900 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:18:10.754158 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:10.754126 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4fdzs\"" Oct 31 00:18:10.786152 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:10.786112 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85672d8b-zrbx4"] Oct 31 00:18:10.880141 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:10.880093 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/025fab84-c395-489e-9662-3d524a5681a4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85672d8b-zrbx4\" (UID: \"025fab84-c395-489e-9662-3d524a5681a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85672d8b-zrbx4" Oct 31 00:18:10.880141 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:10.880143 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jnmvl\" (UniqueName: \"kubernetes.io/projected/025fab84-c395-489e-9662-3d524a5681a4-kube-api-access-jnmvl\") pod \"maas-api-revoke-sub-85672d8b-zrbx4\" (UID: \"025fab84-c395-489e-9662-3d524a5681a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85672d8b-zrbx4" Oct 31 00:18:10.880419 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:10.880261 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/025fab84-c395-489e-9662-3d524a5681a4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85672d8b-zrbx4\" (UID: \"025fab84-c395-489e-9662-3d524a5681a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85672d8b-zrbx4" Oct 31 00:18:10.981567 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:10.981471 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/025fab84-c395-489e-9662-3d524a5681a4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85672d8b-zrbx4\" (UID: \"025fab84-c395-489e-9662-3d524a5681a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85672d8b-zrbx4" Oct 31 00:18:10.981736 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:10.981612 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/025fab84-c395-489e-9662-3d524a5681a4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85672d8b-zrbx4\" (UID: \"025fab84-c395-489e-9662-3d524a5681a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85672d8b-zrbx4" Oct 31 00:18:10.981736 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:10.981645 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jnmvl\" (UniqueName: \"kubernetes.io/projected/025fab84-c395-489e-9662-3d524a5681a4-kube-api-access-jnmvl\") pod \"maas-api-revoke-sub-85672d8b-zrbx4\" (UID: \"025fab84-c395-489e-9662-3d524a5681a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85672d8b-zrbx4" Oct 31 00:18:10.982232 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:10.982205 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/025fab84-c395-489e-9662-3d524a5681a4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85672d8b-zrbx4\" (UID: \"025fab84-c395-489e-9662-3d524a5681a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85672d8b-zrbx4" Oct 31 00:18:10.983945 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:10.983923 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/025fab84-c395-489e-9662-3d524a5681a4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85672d8b-zrbx4\" (UID: \"025fab84-c395-489e-9662-3d524a5681a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85672d8b-zrbx4" Oct 31 00:18:11.044984 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:11.044934 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jnmvl\" (UniqueName: \"kubernetes.io/projected/025fab84-c395-489e-9662-3d524a5681a4-kube-api-access-jnmvl\") pod \"maas-api-revoke-sub-85672d8b-zrbx4\" (UID: \"025fab84-c395-489e-9662-3d524a5681a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85672d8b-zrbx4" Oct 31 00:18:11.333717 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:11.333625 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85672d8b-zrbx4" Oct 31 00:18:11.426268 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:11.426165 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-92e375-6d87d5b947-4249b" event={"ID":"0be20326-c5ad-48ff-8126-f4a4e067a098","Type":"ContainerStarted","Data":"277c89dc3a3bc7533aaccb0eca7958b43eede8f1e53e48baec2be6df32fa8e5c"} Oct 31 00:18:11.426268 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:11.426222 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-92e375-6d87d5b947-4249b" event={"ID":"0be20326-c5ad-48ff-8126-f4a4e067a098","Type":"ContainerStarted","Data":"a2d490366ed6d9bc35c78afccb5a84daa1ab0bbf79bac75446f3de0046960b15"} Oct 31 00:18:11.427012 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:11.426671 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-92e375-6d87d5b947-4249b" Oct 31 00:18:11.429147 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:11.429093 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-92e375-75c9fbbc84-nt42r" event={"ID":"89af18c8-54a0-4dc6-a07b-07bd814ad172","Type":"ContainerStarted","Data":"a7cdfc7579807ed78d04ae294b5e936bd955089beef99761085d5d97ab097598"} Oct 31 00:18:11.429251 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:11.429166 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-92e375-75c9fbbc84-nt42r" event={"ID":"89af18c8-54a0-4dc6-a07b-07bd814ad172","Type":"ContainerStarted","Data":"4ae0fd4c8daedb66927ee99df1bbf5564d3d4fb7abaa5e9d9b831d2431837714"} Oct 31 00:18:11.429251 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:11.429199 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-92e375-75c9fbbc84-nt42r" Oct 31 00:18:11.548971 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:11.548911 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-92e375-6d87d5b947-4249b" podStartSLOduration=2.548891517 podStartE2EDuration="2.548891517s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:18:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:18:11.545671766 +0000 UTC m=+2204.582002589" watchObservedRunningTime="2026-10-31 00:18:11.548891517 +0000 UTC m=+2204.585222341" Oct 31 00:18:11.590736 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:11.590658 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:18:11.710359 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:11.705827 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-92e375-75c9fbbc84-nt42r" podStartSLOduration=2.705805621 podStartE2EDuration="2.705805621s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:18:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:18:11.705560818 +0000 UTC m=+2204.741891671" watchObservedRunningTime="2026-10-31 00:18:11.705805621 +0000 UTC m=+2204.742136443" Oct 31 00:18:11.737783 ip-10-0-141-180 kubenswrapper[4927]: W1031 00:18:11.737746 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod025fab84_c395_489e_9662_3d524a5681a4.slice/crio-161f6f1b921339e4dc2afdd87129c90eff68a23e311a7fbcdd374762edd96164 WatchSource:0}: Error finding container 161f6f1b921339e4dc2afdd87129c90eff68a23e311a7fbcdd374762edd96164: Status 404 returned error can't find the container with id 161f6f1b921339e4dc2afdd87129c90eff68a23e311a7fbcdd374762edd96164 Oct 31 00:18:11.740360 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:11.740311 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85672d8b-zrbx4"] Oct 31 00:18:12.441411 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:12.441370 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85672d8b-zrbx4" event={"ID":"025fab84-c395-489e-9662-3d524a5681a4","Type":"ContainerStarted","Data":"3ea6f5675993b7a169c4f62149c867d3511b2542fc1ca288a4f707580a8296a2"} Oct 31 00:18:12.441893 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:12.441435 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85672d8b-zrbx4" event={"ID":"025fab84-c395-489e-9662-3d524a5681a4","Type":"ContainerStarted","Data":"161f6f1b921339e4dc2afdd87129c90eff68a23e311a7fbcdd374762edd96164"} Oct 31 00:18:12.565404 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:12.565312 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85672d8b-zrbx4" podStartSLOduration=2.565298748 podStartE2EDuration="2.565298748s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:18:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:18:12.5629046 +0000 UTC m=+2205.599235417" watchObservedRunningTime="2026-10-31 00:18:12.565298748 +0000 UTC m=+2205.601629567" Oct 31 00:18:13.452289 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:13.452247 4927 generic.go:356] "Generic (PLEG): container finished" podID="025fab84-c395-489e-9662-3d524a5681a4" containerID="3ea6f5675993b7a169c4f62149c867d3511b2542fc1ca288a4f707580a8296a2" exitCode=0 Oct 31 00:18:13.452760 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:13.452371 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85672d8b-zrbx4" event={"ID":"025fab84-c395-489e-9662-3d524a5681a4","Type":"ContainerDied","Data":"3ea6f5675993b7a169c4f62149c867d3511b2542fc1ca288a4f707580a8296a2"} Oct 31 00:18:14.642162 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:14.642135 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85672d8b-zrbx4" Oct 31 00:18:14.670260 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:14.670224 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/025fab84-c395-489e-9662-3d524a5681a4-kube-api-access-jnmvl\" (UniqueName: \"kubernetes.io/projected/025fab84-c395-489e-9662-3d524a5681a4-kube-api-access-jnmvl\") pod \"025fab84-c395-489e-9662-3d524a5681a4\" (UID: \"025fab84-c395-489e-9662-3d524a5681a4\") " Oct 31 00:18:14.670424 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:14.670371 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/025fab84-c395-489e-9662-3d524a5681a4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/025fab84-c395-489e-9662-3d524a5681a4-maas-api-xks-ca\") pod \"025fab84-c395-489e-9662-3d524a5681a4\" (UID: \"025fab84-c395-489e-9662-3d524a5681a4\") " Oct 31 00:18:14.670513 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:14.670439 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/025fab84-c395-489e-9662-3d524a5681a4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/025fab84-c395-489e-9662-3d524a5681a4-maas-api-openshift-ca\") pod \"025fab84-c395-489e-9662-3d524a5681a4\" (UID: \"025fab84-c395-489e-9662-3d524a5681a4\") " Oct 31 00:18:14.670974 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:14.670932 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/025fab84-c395-489e-9662-3d524a5681a4-maas-api-openshift-ca" pod "025fab84-c395-489e-9662-3d524a5681a4" (UID: "025fab84-c395-489e-9662-3d524a5681a4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:18:14.671187 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:14.671162 4927 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/025fab84-c395-489e-9662-3d524a5681a4-maas-api-openshift-ca\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:18:14.672677 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:14.672648 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/025fab84-c395-489e-9662-3d524a5681a4-maas-api-xks-ca" pod "025fab84-c395-489e-9662-3d524a5681a4" (UID: "025fab84-c395-489e-9662-3d524a5681a4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:18:14.673010 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:14.672984 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/025fab84-c395-489e-9662-3d524a5681a4-kube-api-access-jnmvl" pod "025fab84-c395-489e-9662-3d524a5681a4" (UID: "025fab84-c395-489e-9662-3d524a5681a4"). InnerVolumeSpecName "kube-api-access-jnmvl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:18:14.773224 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:14.773126 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jnmvl\" (UniqueName: \"kubernetes.io/projected/025fab84-c395-489e-9662-3d524a5681a4-kube-api-access-jnmvl\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:18:14.773224 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:14.773187 4927 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/025fab84-c395-489e-9662-3d524a5681a4-maas-api-xks-ca\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:18:15.164314 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:15.164284 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-76b073-68467cb4b7-pf7qq" Oct 31 00:18:15.164533 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:15.164383 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-76b073-78787bc78-d6l68" Oct 31 00:18:15.472509 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:15.472476 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85672d8b-zrbx4" Oct 31 00:18:15.472509 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:15.472503 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85672d8b-zrbx4" event={"ID":"025fab84-c395-489e-9662-3d524a5681a4","Type":"ContainerDied","Data":"161f6f1b921339e4dc2afdd87129c90eff68a23e311a7fbcdd374762edd96164"} Oct 31 00:18:15.472739 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:15.472552 4927 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="161f6f1b921339e4dc2afdd87129c90eff68a23e311a7fbcdd374762edd96164" Oct 31 00:18:16.401893 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:16.401859 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:18:19.681285 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:18:19.681247 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sqv69" containerName="extractor" Oct 31 00:18:21.015121 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:21.015077 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:18:22.445069 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:22.445038 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-92e375-6d87d5b947-4249b" Oct 31 00:18:22.445506 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:22.445113 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-92e375-75c9fbbc84-nt42r" Oct 31 00:18:25.154199 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:25.154164 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:18:26.285404 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:26.285323 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:18:26.665970 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:26.665583 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:18:26.908631 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:26.908597 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:18:27.582112 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:27.582075 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:18:27.870964 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:27.870871 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:18:28.077215 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:28.077164 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:18:28.245005 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:28.244902 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:18:28.853655 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:28.853619 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:18:30.559402 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:30.559373 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:18:32.539822 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:32.539788 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-a251fdfb/e2e-lifecycle-model-a251fdfb-kserve-5966998657-7wkbl"] Oct 31 00:18:32.624082 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:32.624050 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:18:33.044318 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:33.044272 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:18:34.885711 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:34.885677 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:18:35.143636 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:35.143546 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:18:35.252183 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:35.252150 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-03c703-openshift-default-78bd587bff-5m2n8"] Oct 31 00:18:35.252404 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:35.252383 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-03c703-openshift-default-78bd587bff-5m2n8" podUID="9ee78c4f-f1a0-4cc0-8441-106def585fff" containerName="istio-proxy" containerID="cri-o://3f3ceed141a812642c377a73790e461ebd99d22338f2aefc053f34bc04922e55" gracePeriod=30 Oct 31 00:18:35.306514 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:35.306458 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:18:37.717587 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:37.717548 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:18:40.541165 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:40.541131 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-03c703-openshift-default-78bd587bff-5m2n8" Oct 31 00:18:40.651556 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:40.651520 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ee78c4f-f1a0-4cc0-8441-106def585fff-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ee78c4f-f1a0-4cc0-8441-106def585fff-workload-certs\") pod \"9ee78c4f-f1a0-4cc0-8441-106def585fff\" (UID: \"9ee78c4f-f1a0-4cc0-8441-106def585fff\") " Oct 31 00:18:40.651744 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:40.651567 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9ee78c4f-f1a0-4cc0-8441-106def585fff-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ee78c4f-f1a0-4cc0-8441-106def585fff-istiod-ca-cert\") pod \"9ee78c4f-f1a0-4cc0-8441-106def585fff\" (UID: \"9ee78c4f-f1a0-4cc0-8441-106def585fff\") " Oct 31 00:18:40.651744 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:40.651592 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ee78c4f-f1a0-4cc0-8441-106def585fff-kube-api-access-fktm9\" (UniqueName: \"kubernetes.io/projected/9ee78c4f-f1a0-4cc0-8441-106def585fff-kube-api-access-fktm9\") pod \"9ee78c4f-f1a0-4cc0-8441-106def585fff\" (UID: \"9ee78c4f-f1a0-4cc0-8441-106def585fff\") " Oct 31 00:18:40.651744 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:40.651618 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ee78c4f-f1a0-4cc0-8441-106def585fff-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ee78c4f-f1a0-4cc0-8441-106def585fff-workload-socket\") pod \"9ee78c4f-f1a0-4cc0-8441-106def585fff\" (UID: \"9ee78c4f-f1a0-4cc0-8441-106def585fff\") " Oct 31 00:18:40.651744 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:40.651667 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ee78c4f-f1a0-4cc0-8441-106def585fff-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ee78c4f-f1a0-4cc0-8441-106def585fff-credential-socket\") pod \"9ee78c4f-f1a0-4cc0-8441-106def585fff\" (UID: \"9ee78c4f-f1a0-4cc0-8441-106def585fff\") " Oct 31 00:18:40.651744 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:40.651728 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9ee78c4f-f1a0-4cc0-8441-106def585fff-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ee78c4f-f1a0-4cc0-8441-106def585fff-istio-podinfo\") pod \"9ee78c4f-f1a0-4cc0-8441-106def585fff\" (UID: \"9ee78c4f-f1a0-4cc0-8441-106def585fff\") " Oct 31 00:18:40.652041 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:40.651793 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ee78c4f-f1a0-4cc0-8441-106def585fff-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ee78c4f-f1a0-4cc0-8441-106def585fff-istio-envoy\") pod \"9ee78c4f-f1a0-4cc0-8441-106def585fff\" (UID: \"9ee78c4f-f1a0-4cc0-8441-106def585fff\") " Oct 31 00:18:40.652041 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:40.651802 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ee78c4f-f1a0-4cc0-8441-106def585fff-workload-certs" pod "9ee78c4f-f1a0-4cc0-8441-106def585fff" (UID: "9ee78c4f-f1a0-4cc0-8441-106def585fff"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:18:40.652041 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:40.651833 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ee78c4f-f1a0-4cc0-8441-106def585fff-istio-token\" (UniqueName: \"kubernetes.io/projected/9ee78c4f-f1a0-4cc0-8441-106def585fff-istio-token\") pod \"9ee78c4f-f1a0-4cc0-8441-106def585fff\" (UID: \"9ee78c4f-f1a0-4cc0-8441-106def585fff\") " Oct 31 00:18:40.652041 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:40.651876 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ee78c4f-f1a0-4cc0-8441-106def585fff-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ee78c4f-f1a0-4cc0-8441-106def585fff-istio-data\") pod \"9ee78c4f-f1a0-4cc0-8441-106def585fff\" (UID: \"9ee78c4f-f1a0-4cc0-8441-106def585fff\") " Oct 31 00:18:40.652041 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:40.651953 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ee78c4f-f1a0-4cc0-8441-106def585fff-workload-socket" pod "9ee78c4f-f1a0-4cc0-8441-106def585fff" (UID: "9ee78c4f-f1a0-4cc0-8441-106def585fff"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:18:40.652041 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:40.651997 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ee78c4f-f1a0-4cc0-8441-106def585fff-credential-socket" pod "9ee78c4f-f1a0-4cc0-8441-106def585fff" (UID: "9ee78c4f-f1a0-4cc0-8441-106def585fff"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:18:40.652041 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:40.652006 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ee78c4f-f1a0-4cc0-8441-106def585fff-istiod-ca-cert" pod "9ee78c4f-f1a0-4cc0-8441-106def585fff" (UID: "9ee78c4f-f1a0-4cc0-8441-106def585fff"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:18:40.652447 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:40.652160 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ee78c4f-f1a0-4cc0-8441-106def585fff-istio-data" pod "9ee78c4f-f1a0-4cc0-8441-106def585fff" (UID: "9ee78c4f-f1a0-4cc0-8441-106def585fff"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:18:40.652648 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:40.652620 4927 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ee78c4f-f1a0-4cc0-8441-106def585fff-workload-certs\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:18:40.652781 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:40.652650 4927 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ee78c4f-f1a0-4cc0-8441-106def585fff-istiod-ca-cert\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:18:40.652781 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:40.652665 4927 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ee78c4f-f1a0-4cc0-8441-106def585fff-workload-socket\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:18:40.652781 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:40.652678 4927 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ee78c4f-f1a0-4cc0-8441-106def585fff-credential-socket\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:18:40.652781 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:40.652691 4927 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ee78c4f-f1a0-4cc0-8441-106def585fff-istio-data\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:18:40.653898 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:40.653874 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ee78c4f-f1a0-4cc0-8441-106def585fff-kube-api-access-fktm9" pod "9ee78c4f-f1a0-4cc0-8441-106def585fff" (UID: "9ee78c4f-f1a0-4cc0-8441-106def585fff"). InnerVolumeSpecName "kube-api-access-fktm9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:18:40.654202 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:40.654185 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ee78c4f-f1a0-4cc0-8441-106def585fff-istio-envoy" pod "9ee78c4f-f1a0-4cc0-8441-106def585fff" (UID: "9ee78c4f-f1a0-4cc0-8441-106def585fff"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:18:40.654441 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:40.654421 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9ee78c4f-f1a0-4cc0-8441-106def585fff-istio-podinfo" pod "9ee78c4f-f1a0-4cc0-8441-106def585fff" (UID: "9ee78c4f-f1a0-4cc0-8441-106def585fff"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:18:40.654441 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:40.654429 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ee78c4f-f1a0-4cc0-8441-106def585fff-istio-token" pod "9ee78c4f-f1a0-4cc0-8441-106def585fff" (UID: "9ee78c4f-f1a0-4cc0-8441-106def585fff"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:18:40.754315 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:40.754275 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fktm9\" (UniqueName: \"kubernetes.io/projected/9ee78c4f-f1a0-4cc0-8441-106def585fff-kube-api-access-fktm9\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:18:40.754315 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:40.754314 4927 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ee78c4f-f1a0-4cc0-8441-106def585fff-istio-podinfo\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:18:40.754605 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:40.754344 4927 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ee78c4f-f1a0-4cc0-8441-106def585fff-istio-envoy\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:18:40.754605 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:40.754357 4927 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ee78c4f-f1a0-4cc0-8441-106def585fff-istio-token\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:18:40.776373 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:40.775996 4927 generic.go:356] "Generic (PLEG): container finished" podID="9ee78c4f-f1a0-4cc0-8441-106def585fff" containerID="3f3ceed141a812642c377a73790e461ebd99d22338f2aefc053f34bc04922e55" exitCode=0 Oct 31 00:18:40.776373 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:40.776066 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-03c703-openshift-default-78bd587bff-5m2n8" event={"ID":"9ee78c4f-f1a0-4cc0-8441-106def585fff","Type":"ContainerDied","Data":"3f3ceed141a812642c377a73790e461ebd99d22338f2aefc053f34bc04922e55"} Oct 31 00:18:40.776373 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:40.776138 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-03c703-openshift-default-78bd587bff-5m2n8" event={"ID":"9ee78c4f-f1a0-4cc0-8441-106def585fff","Type":"ContainerDied","Data":"7843f7ce7891c3a4b27fd0179a97fb0d6fbe9ea064c0c1009a2a2b29e9875b42"} Oct 31 00:18:40.776373 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:40.776172 4927 scope.go:122] "RemoveContainer" containerID="3f3ceed141a812642c377a73790e461ebd99d22338f2aefc053f34bc04922e55" Oct 31 00:18:40.776373 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:40.776322 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-03c703-openshift-default-78bd587bff-5m2n8" Oct 31 00:18:40.793585 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:40.793542 4927 scope.go:122] "RemoveContainer" containerID="b203e94aec128fce45da60bb93ab9e2259f6808bbc342b332cb76a4071ddd011" Oct 31 00:18:40.804776 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:40.804751 4927 scope.go:122] "RemoveContainer" containerID="3f3ceed141a812642c377a73790e461ebd99d22338f2aefc053f34bc04922e55" Oct 31 00:18:40.805040 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:18:40.805011 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f3ceed141a812642c377a73790e461ebd99d22338f2aefc053f34bc04922e55\": container with ID starting with 3f3ceed141a812642c377a73790e461ebd99d22338f2aefc053f34bc04922e55 not found: ID does not exist" containerID="3f3ceed141a812642c377a73790e461ebd99d22338f2aefc053f34bc04922e55" Oct 31 00:18:40.805116 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:40.805041 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f3ceed141a812642c377a73790e461ebd99d22338f2aefc053f34bc04922e55"} err="failed to get container status \"3f3ceed141a812642c377a73790e461ebd99d22338f2aefc053f34bc04922e55\": rpc error: code = NotFound desc = could not find container \"3f3ceed141a812642c377a73790e461ebd99d22338f2aefc053f34bc04922e55\": container with ID starting with 3f3ceed141a812642c377a73790e461ebd99d22338f2aefc053f34bc04922e55 not found: ID does not exist" Oct 31 00:18:40.805116 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:40.805060 4927 scope.go:122] "RemoveContainer" containerID="b203e94aec128fce45da60bb93ab9e2259f6808bbc342b332cb76a4071ddd011" Oct 31 00:18:40.805603 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:18:40.805577 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b203e94aec128fce45da60bb93ab9e2259f6808bbc342b332cb76a4071ddd011\": container with ID starting with b203e94aec128fce45da60bb93ab9e2259f6808bbc342b332cb76a4071ddd011 not found: ID does not exist" containerID="b203e94aec128fce45da60bb93ab9e2259f6808bbc342b332cb76a4071ddd011" Oct 31 00:18:40.805699 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:40.805606 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b203e94aec128fce45da60bb93ab9e2259f6808bbc342b332cb76a4071ddd011"} err="failed to get container status \"b203e94aec128fce45da60bb93ab9e2259f6808bbc342b332cb76a4071ddd011\": rpc error: code = NotFound desc = could not find container \"b203e94aec128fce45da60bb93ab9e2259f6808bbc342b332cb76a4071ddd011\": container with ID starting with b203e94aec128fce45da60bb93ab9e2259f6808bbc342b332cb76a4071ddd011 not found: ID does not exist" Oct 31 00:18:40.952657 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:40.952612 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-03c703-openshift-default-78bd587bff-5m2n8"] Oct 31 00:18:40.957354 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:40.957316 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-03c703-openshift-default-78bd587bff-5m2n8"] Oct 31 00:18:41.087642 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:41.087570 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9ee78c4f-f1a0-4cc0-8441-106def585fff" path="/var/lib/kubelet/pods/9ee78c4f-f1a0-4cc0-8441-106def585fff/volumes" Oct 31 00:18:41.427842 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:41.427802 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:18:41.592091 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:41.592057 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:18:42.053470 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:42.053425 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:18:43.641894 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:43.641822 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:18:43.944926 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:43.944889 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:18:44.092307 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:44.092273 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:18:44.584249 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:44.584212 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:18:47.455081 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:47.455048 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-460d4e-6457b77684-dttcx"] Oct 31 00:18:47.562118 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:47.562080 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-mt-a251fdfb/e2e-lifecycle-model-a251fdfb-kserve-5966998657-7wkbl" podUID="e9c18450-ac2a-4e6a-a712-00c2a755535d" containerName="main" containerID="cri-o://e0443ccad024469a8e53ffc04186b0f362c0dab29e418fb70032ba59372e2435" gracePeriod=45 Oct 31 00:18:47.819023 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:47.818985 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:18:47.842590 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:47.842435 4927 generic.go:356] "Generic (PLEG): container finished" podID="e9c18450-ac2a-4e6a-a712-00c2a755535d" containerID="e0443ccad024469a8e53ffc04186b0f362c0dab29e418fb70032ba59372e2435" exitCode=0 Oct 31 00:18:47.842590 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:47.842533 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-a251fdfb/e2e-lifecycle-model-a251fdfb-kserve-5966998657-7wkbl" event={"ID":"e9c18450-ac2a-4e6a-a712-00c2a755535d","Type":"ContainerDied","Data":"e0443ccad024469a8e53ffc04186b0f362c0dab29e418fb70032ba59372e2435"} Oct 31 00:18:47.917778 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:47.917749 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-a251fdfb/e2e-lifecycle-model-a251fdfb-kserve-5966998657-7wkbl" Oct 31 00:18:47.928563 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:47.928536 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9c18450-ac2a-4e6a-a712-00c2a755535d-home\" (UniqueName: \"kubernetes.io/empty-dir/e9c18450-ac2a-4e6a-a712-00c2a755535d-home\") pod \"e9c18450-ac2a-4e6a-a712-00c2a755535d\" (UID: \"e9c18450-ac2a-4e6a-a712-00c2a755535d\") " Oct 31 00:18:47.928702 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:47.928655 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9c18450-ac2a-4e6a-a712-00c2a755535d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9c18450-ac2a-4e6a-a712-00c2a755535d-tmp-dir\") pod \"e9c18450-ac2a-4e6a-a712-00c2a755535d\" (UID: \"e9c18450-ac2a-4e6a-a712-00c2a755535d\") " Oct 31 00:18:47.928747 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:47.928720 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9c18450-ac2a-4e6a-a712-00c2a755535d-kube-api-access-2p984\" (UniqueName: \"kubernetes.io/projected/e9c18450-ac2a-4e6a-a712-00c2a755535d-kube-api-access-2p984\") pod \"e9c18450-ac2a-4e6a-a712-00c2a755535d\" (UID: \"e9c18450-ac2a-4e6a-a712-00c2a755535d\") " Oct 31 00:18:47.928839 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:47.928817 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9c18450-ac2a-4e6a-a712-00c2a755535d-home" pod "e9c18450-ac2a-4e6a-a712-00c2a755535d" (UID: "e9c18450-ac2a-4e6a-a712-00c2a755535d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:18:47.928883 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:47.928835 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9c18450-ac2a-4e6a-a712-00c2a755535d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9c18450-ac2a-4e6a-a712-00c2a755535d-model-cache\") pod \"e9c18450-ac2a-4e6a-a712-00c2a755535d\" (UID: \"e9c18450-ac2a-4e6a-a712-00c2a755535d\") " Oct 31 00:18:47.928883 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:47.928864 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9c18450-ac2a-4e6a-a712-00c2a755535d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9c18450-ac2a-4e6a-a712-00c2a755535d-dshm\") pod \"e9c18450-ac2a-4e6a-a712-00c2a755535d\" (UID: \"e9c18450-ac2a-4e6a-a712-00c2a755535d\") " Oct 31 00:18:47.928991 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:47.928892 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e9c18450-ac2a-4e6a-a712-00c2a755535d-tls-certs\" (UniqueName: \"kubernetes.io/secret/e9c18450-ac2a-4e6a-a712-00c2a755535d-tls-certs\") pod \"e9c18450-ac2a-4e6a-a712-00c2a755535d\" (UID: \"e9c18450-ac2a-4e6a-a712-00c2a755535d\") " Oct 31 00:18:47.928991 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:47.928935 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9c18450-ac2a-4e6a-a712-00c2a755535d-tmp-dir" pod "e9c18450-ac2a-4e6a-a712-00c2a755535d" (UID: "e9c18450-ac2a-4e6a-a712-00c2a755535d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:18:47.929156 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:47.929122 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9c18450-ac2a-4e6a-a712-00c2a755535d-model-cache" pod "e9c18450-ac2a-4e6a-a712-00c2a755535d" (UID: "e9c18450-ac2a-4e6a-a712-00c2a755535d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:18:47.929609 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:47.929588 4927 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9c18450-ac2a-4e6a-a712-00c2a755535d-tmp-dir\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:18:47.929609 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:47.929609 4927 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9c18450-ac2a-4e6a-a712-00c2a755535d-model-cache\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:18:47.929793 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:47.929619 4927 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e9c18450-ac2a-4e6a-a712-00c2a755535d-home\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:18:47.931524 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:47.931496 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9c18450-ac2a-4e6a-a712-00c2a755535d-kube-api-access-2p984" pod "e9c18450-ac2a-4e6a-a712-00c2a755535d" (UID: "e9c18450-ac2a-4e6a-a712-00c2a755535d"). InnerVolumeSpecName "kube-api-access-2p984". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:18:47.931524 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:47.931506 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e9c18450-ac2a-4e6a-a712-00c2a755535d-tls-certs" pod "e9c18450-ac2a-4e6a-a712-00c2a755535d" (UID: "e9c18450-ac2a-4e6a-a712-00c2a755535d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:18:47.931668 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:47.931521 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9c18450-ac2a-4e6a-a712-00c2a755535d-dshm" pod "e9c18450-ac2a-4e6a-a712-00c2a755535d" (UID: "e9c18450-ac2a-4e6a-a712-00c2a755535d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:18:48.016384 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:48.016182 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:18:48.030662 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:48.030626 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2p984\" (UniqueName: \"kubernetes.io/projected/e9c18450-ac2a-4e6a-a712-00c2a755535d-kube-api-access-2p984\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:18:48.030662 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:48.030658 4927 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9c18450-ac2a-4e6a-a712-00c2a755535d-dshm\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:18:48.030662 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:48.030668 4927 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9c18450-ac2a-4e6a-a712-00c2a755535d-tls-certs\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:18:48.566235 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:48.566201 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:18:48.764114 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:48.764081 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:18:48.910658 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:48.908515 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-a251fdfb/e2e-lifecycle-model-a251fdfb-kserve-5966998657-7wkbl" event={"ID":"e9c18450-ac2a-4e6a-a712-00c2a755535d","Type":"ContainerDied","Data":"a20c82547afb2207318e618562bdf8715907b9104bf786d494d5f1e7dc78a8b7"} Oct 31 00:18:48.910658 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:48.908571 4927 scope.go:122] "RemoveContainer" containerID="e0443ccad024469a8e53ffc04186b0f362c0dab29e418fb70032ba59372e2435" Oct 31 00:18:48.910658 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:48.908714 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-a251fdfb/e2e-lifecycle-model-a251fdfb-kserve-5966998657-7wkbl" Oct 31 00:18:48.953483 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:48.946831 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:18:49.163438 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:49.123264 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-a251fdfb/e2e-lifecycle-model-a251fdfb-kserve-5966998657-7wkbl"] Oct 31 00:18:49.171367 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:49.170188 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-a251fdfb/e2e-lifecycle-model-a251fdfb-kserve-5966998657-7wkbl"] Oct 31 00:18:49.678900 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:18:49.678853 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sqv69" containerName="extractor" Oct 31 00:18:51.134032 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:51.133993 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e9c18450-ac2a-4e6a-a712-00c2a755535d" path="/var/lib/kubelet/pods/e9c18450-ac2a-4e6a-a712-00c2a755535d/volumes" Oct 31 00:18:51.789425 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:51.789385 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:18:51.988877 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:51.988843 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:18:52.178572 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:52.178539 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:18:52.455447 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:52.455326 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-460d4e-6457b77684-dttcx" podUID="bbe18c5f-bf04-448b-8385-e99804d6056d" containerName="payload-processing" containerID="cri-o://afb90c351598ccb99abd88d34a8edf2a35aac81d8551d9cdd51003840808454b" gracePeriod=25 Oct 31 00:18:52.641933 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:52.641797 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:18:52.759240 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:52.759215 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-460d4e-6457b77684-dttcx" Oct 31 00:18:52.795758 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:52.795726 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bbe18c5f-bf04-448b-8385-e99804d6056d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbe18c5f-bf04-448b-8385-e99804d6056d-plugins-config-volume\") pod \"bbe18c5f-bf04-448b-8385-e99804d6056d\" (UID: \"bbe18c5f-bf04-448b-8385-e99804d6056d\") " Oct 31 00:18:52.795938 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:52.795765 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bbe18c5f-bf04-448b-8385-e99804d6056d-kube-api-access-wnx6x\" (UniqueName: \"kubernetes.io/projected/bbe18c5f-bf04-448b-8385-e99804d6056d-kube-api-access-wnx6x\") pod \"bbe18c5f-bf04-448b-8385-e99804d6056d\" (UID: \"bbe18c5f-bf04-448b-8385-e99804d6056d\") " Oct 31 00:18:52.796162 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:52.796140 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bbe18c5f-bf04-448b-8385-e99804d6056d-plugins-config-volume" pod "bbe18c5f-bf04-448b-8385-e99804d6056d" (UID: "bbe18c5f-bf04-448b-8385-e99804d6056d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:18:52.796315 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:52.796301 4927 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbe18c5f-bf04-448b-8385-e99804d6056d-plugins-config-volume\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:18:52.797947 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:52.797927 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbe18c5f-bf04-448b-8385-e99804d6056d-kube-api-access-wnx6x" pod "bbe18c5f-bf04-448b-8385-e99804d6056d" (UID: "bbe18c5f-bf04-448b-8385-e99804d6056d"). InnerVolumeSpecName "kube-api-access-wnx6x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:18:52.806438 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:52.806408 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:18:52.898220 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:52.898178 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wnx6x\" (UniqueName: \"kubernetes.io/projected/bbe18c5f-bf04-448b-8385-e99804d6056d-kube-api-access-wnx6x\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:18:52.964343 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:52.964291 4927 generic.go:356] "Generic (PLEG): container finished" podID="bbe18c5f-bf04-448b-8385-e99804d6056d" containerID="afb90c351598ccb99abd88d34a8edf2a35aac81d8551d9cdd51003840808454b" exitCode=0 Oct 31 00:18:52.964548 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:52.964396 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-460d4e-6457b77684-dttcx" Oct 31 00:18:52.964548 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:52.964395 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-460d4e-6457b77684-dttcx" event={"ID":"bbe18c5f-bf04-448b-8385-e99804d6056d","Type":"ContainerDied","Data":"afb90c351598ccb99abd88d34a8edf2a35aac81d8551d9cdd51003840808454b"} Oct 31 00:18:52.964653 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:52.964581 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-460d4e-6457b77684-dttcx" event={"ID":"bbe18c5f-bf04-448b-8385-e99804d6056d","Type":"ContainerDied","Data":"48c8ff945fac5dd851c8966db2edcdc777eeabab14bf7dfb09e3fae28f271d9d"} Oct 31 00:18:52.964653 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:52.964618 4927 scope.go:122] "RemoveContainer" containerID="afb90c351598ccb99abd88d34a8edf2a35aac81d8551d9cdd51003840808454b" Oct 31 00:18:52.978505 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:52.978485 4927 scope.go:122] "RemoveContainer" containerID="afb90c351598ccb99abd88d34a8edf2a35aac81d8551d9cdd51003840808454b" Oct 31 00:18:52.978983 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:18:52.978961 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"afb90c351598ccb99abd88d34a8edf2a35aac81d8551d9cdd51003840808454b\": container with ID starting with afb90c351598ccb99abd88d34a8edf2a35aac81d8551d9cdd51003840808454b not found: ID does not exist" containerID="afb90c351598ccb99abd88d34a8edf2a35aac81d8551d9cdd51003840808454b" Oct 31 00:18:52.979076 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:52.978993 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"afb90c351598ccb99abd88d34a8edf2a35aac81d8551d9cdd51003840808454b"} err="failed to get container status \"afb90c351598ccb99abd88d34a8edf2a35aac81d8551d9cdd51003840808454b\": rpc error: code = NotFound desc = could not find container \"afb90c351598ccb99abd88d34a8edf2a35aac81d8551d9cdd51003840808454b\": container with ID starting with afb90c351598ccb99abd88d34a8edf2a35aac81d8551d9cdd51003840808454b not found: ID does not exist" Oct 31 00:18:53.061380 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:53.059726 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:18:53.137543 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:53.137497 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-460d4e-6457b77684-dttcx"] Oct 31 00:18:53.141372 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:53.141321 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-460d4e-6457b77684-dttcx"] Oct 31 00:18:54.043525 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:54.043486 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:18:54.673425 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:54.673395 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-ae3cd0-openshift-default-5dbcd797d5-hvcv4"] Oct 31 00:18:54.674075 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:54.674060 4927 cpu_manager.go:414] "containerMap: removing container" podUID="9ee78c4f-f1a0-4cc0-8441-106def585fff" containerName="istio-proxy" Oct 31 00:18:54.674130 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:54.674077 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ee78c4f-f1a0-4cc0-8441-106def585fff" containerName="istio-proxy" Oct 31 00:18:54.674130 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:54.674098 4927 cpu_manager.go:414] "containerMap: removing container" podUID="bbe18c5f-bf04-448b-8385-e99804d6056d" containerName="payload-processing" Oct 31 00:18:54.674130 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:54.674104 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bbe18c5f-bf04-448b-8385-e99804d6056d" containerName="payload-processing" Oct 31 00:18:54.674130 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:54.674130 4927 cpu_manager.go:414] "containerMap: removing container" podUID="025fab84-c395-489e-9662-3d524a5681a4" containerName="revoke-keys" Oct 31 00:18:54.674263 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:54.674135 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="025fab84-c395-489e-9662-3d524a5681a4" containerName="revoke-keys" Oct 31 00:18:54.674263 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:54.674153 4927 cpu_manager.go:414] "containerMap: removing container" podUID="e9c18450-ac2a-4e6a-a712-00c2a755535d" containerName="main" Oct 31 00:18:54.674263 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:54.674158 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9c18450-ac2a-4e6a-a712-00c2a755535d" containerName="main" Oct 31 00:18:54.674263 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:54.674251 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="025fab84-c395-489e-9662-3d524a5681a4" containerName="revoke-keys" Oct 31 00:18:54.674263 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:54.674262 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9c18450-ac2a-4e6a-a712-00c2a755535d" containerName="main" Oct 31 00:18:54.674497 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:54.674270 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ee78c4f-f1a0-4cc0-8441-106def585fff" containerName="istio-proxy" Oct 31 00:18:54.674497 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:54.674279 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ee78c4f-f1a0-4cc0-8441-106def585fff" containerName="istio-proxy" Oct 31 00:18:54.674497 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:54.674287 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="bbe18c5f-bf04-448b-8385-e99804d6056d" containerName="payload-processing" Oct 31 00:18:54.679597 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:54.679574 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-ae3cd0-openshift-default-5dbcd797d5-hvcv4" Oct 31 00:18:54.733869 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:54.733832 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-ae3cd0-openshift-default-dockercfg-q8jgf\"" Oct 31 00:18:54.734461 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:54.734429 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31803f91-4f1e-407f-b82e-386fd8a80be8-istiod-ca-cert\") pod \"e2e-api-a-ae3cd0-openshift-default-5dbcd797d5-hvcv4\" (UID: \"31803f91-4f1e-407f-b82e-386fd8a80be8\") " pod="openshift-ingress/e2e-api-a-ae3cd0-openshift-default-5dbcd797d5-hvcv4" Oct 31 00:18:54.734691 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:54.734662 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2f45\" (UniqueName: \"kubernetes.io/projected/31803f91-4f1e-407f-b82e-386fd8a80be8-kube-api-access-l2f45\") pod \"e2e-api-a-ae3cd0-openshift-default-5dbcd797d5-hvcv4\" (UID: \"31803f91-4f1e-407f-b82e-386fd8a80be8\") " pod="openshift-ingress/e2e-api-a-ae3cd0-openshift-default-5dbcd797d5-hvcv4" Oct 31 00:18:54.734827 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:54.734802 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/31803f91-4f1e-407f-b82e-386fd8a80be8-istio-token\") pod \"e2e-api-a-ae3cd0-openshift-default-5dbcd797d5-hvcv4\" (UID: \"31803f91-4f1e-407f-b82e-386fd8a80be8\") " pod="openshift-ingress/e2e-api-a-ae3cd0-openshift-default-5dbcd797d5-hvcv4" Oct 31 00:18:54.734913 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:54.734884 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31803f91-4f1e-407f-b82e-386fd8a80be8-istio-data\") pod \"e2e-api-a-ae3cd0-openshift-default-5dbcd797d5-hvcv4\" (UID: \"31803f91-4f1e-407f-b82e-386fd8a80be8\") " pod="openshift-ingress/e2e-api-a-ae3cd0-openshift-default-5dbcd797d5-hvcv4" Oct 31 00:18:54.734979 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:54.734949 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31803f91-4f1e-407f-b82e-386fd8a80be8-workload-socket\") pod \"e2e-api-a-ae3cd0-openshift-default-5dbcd797d5-hvcv4\" (UID: \"31803f91-4f1e-407f-b82e-386fd8a80be8\") " pod="openshift-ingress/e2e-api-a-ae3cd0-openshift-default-5dbcd797d5-hvcv4" Oct 31 00:18:54.735066 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:54.735050 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31803f91-4f1e-407f-b82e-386fd8a80be8-istio-envoy\") pod \"e2e-api-a-ae3cd0-openshift-default-5dbcd797d5-hvcv4\" (UID: \"31803f91-4f1e-407f-b82e-386fd8a80be8\") " pod="openshift-ingress/e2e-api-a-ae3cd0-openshift-default-5dbcd797d5-hvcv4" Oct 31 00:18:54.735133 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:54.735117 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31803f91-4f1e-407f-b82e-386fd8a80be8-istio-podinfo\") pod \"e2e-api-a-ae3cd0-openshift-default-5dbcd797d5-hvcv4\" (UID: \"31803f91-4f1e-407f-b82e-386fd8a80be8\") " pod="openshift-ingress/e2e-api-a-ae3cd0-openshift-default-5dbcd797d5-hvcv4" Oct 31 00:18:54.735201 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:54.735178 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31803f91-4f1e-407f-b82e-386fd8a80be8-credential-socket\") pod \"e2e-api-a-ae3cd0-openshift-default-5dbcd797d5-hvcv4\" (UID: \"31803f91-4f1e-407f-b82e-386fd8a80be8\") " pod="openshift-ingress/e2e-api-a-ae3cd0-openshift-default-5dbcd797d5-hvcv4" Oct 31 00:18:54.735273 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:54.735257 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31803f91-4f1e-407f-b82e-386fd8a80be8-workload-certs\") pod \"e2e-api-a-ae3cd0-openshift-default-5dbcd797d5-hvcv4\" (UID: \"31803f91-4f1e-407f-b82e-386fd8a80be8\") " pod="openshift-ingress/e2e-api-a-ae3cd0-openshift-default-5dbcd797d5-hvcv4" Oct 31 00:18:54.770579 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:54.770546 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:18:54.773924 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:54.773893 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-ae3cd0-openshift-default-5dbcd797d5-hvcv4"] Oct 31 00:18:54.895157 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:54.876513 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31803f91-4f1e-407f-b82e-386fd8a80be8-istio-envoy\") pod \"e2e-api-a-ae3cd0-openshift-default-5dbcd797d5-hvcv4\" (UID: \"31803f91-4f1e-407f-b82e-386fd8a80be8\") " pod="openshift-ingress/e2e-api-a-ae3cd0-openshift-default-5dbcd797d5-hvcv4" Oct 31 00:18:54.895617 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:54.895570 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31803f91-4f1e-407f-b82e-386fd8a80be8-istio-podinfo\") pod \"e2e-api-a-ae3cd0-openshift-default-5dbcd797d5-hvcv4\" (UID: \"31803f91-4f1e-407f-b82e-386fd8a80be8\") " pod="openshift-ingress/e2e-api-a-ae3cd0-openshift-default-5dbcd797d5-hvcv4" Oct 31 00:18:54.895897 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:54.895876 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31803f91-4f1e-407f-b82e-386fd8a80be8-credential-socket\") pod \"e2e-api-a-ae3cd0-openshift-default-5dbcd797d5-hvcv4\" (UID: \"31803f91-4f1e-407f-b82e-386fd8a80be8\") " pod="openshift-ingress/e2e-api-a-ae3cd0-openshift-default-5dbcd797d5-hvcv4" Oct 31 00:18:54.896168 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:54.896133 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31803f91-4f1e-407f-b82e-386fd8a80be8-workload-certs\") pod \"e2e-api-a-ae3cd0-openshift-default-5dbcd797d5-hvcv4\" (UID: \"31803f91-4f1e-407f-b82e-386fd8a80be8\") " pod="openshift-ingress/e2e-api-a-ae3cd0-openshift-default-5dbcd797d5-hvcv4" Oct 31 00:18:54.896594 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:54.896571 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31803f91-4f1e-407f-b82e-386fd8a80be8-istiod-ca-cert\") pod \"e2e-api-a-ae3cd0-openshift-default-5dbcd797d5-hvcv4\" (UID: \"31803f91-4f1e-407f-b82e-386fd8a80be8\") " pod="openshift-ingress/e2e-api-a-ae3cd0-openshift-default-5dbcd797d5-hvcv4" Oct 31 00:18:54.896923 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:54.896900 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2f45\" (UniqueName: \"kubernetes.io/projected/31803f91-4f1e-407f-b82e-386fd8a80be8-kube-api-access-l2f45\") pod \"e2e-api-a-ae3cd0-openshift-default-5dbcd797d5-hvcv4\" (UID: \"31803f91-4f1e-407f-b82e-386fd8a80be8\") " pod="openshift-ingress/e2e-api-a-ae3cd0-openshift-default-5dbcd797d5-hvcv4" Oct 31 00:18:54.897348 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:54.897310 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/31803f91-4f1e-407f-b82e-386fd8a80be8-istio-token\") pod \"e2e-api-a-ae3cd0-openshift-default-5dbcd797d5-hvcv4\" (UID: \"31803f91-4f1e-407f-b82e-386fd8a80be8\") " pod="openshift-ingress/e2e-api-a-ae3cd0-openshift-default-5dbcd797d5-hvcv4" Oct 31 00:18:54.897608 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:54.897525 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31803f91-4f1e-407f-b82e-386fd8a80be8-istio-data\") pod \"e2e-api-a-ae3cd0-openshift-default-5dbcd797d5-hvcv4\" (UID: \"31803f91-4f1e-407f-b82e-386fd8a80be8\") " pod="openshift-ingress/e2e-api-a-ae3cd0-openshift-default-5dbcd797d5-hvcv4" Oct 31 00:18:54.897839 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:54.897803 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31803f91-4f1e-407f-b82e-386fd8a80be8-workload-socket\") pod \"e2e-api-a-ae3cd0-openshift-default-5dbcd797d5-hvcv4\" (UID: \"31803f91-4f1e-407f-b82e-386fd8a80be8\") " pod="openshift-ingress/e2e-api-a-ae3cd0-openshift-default-5dbcd797d5-hvcv4" Oct 31 00:18:54.899132 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:54.898269 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31803f91-4f1e-407f-b82e-386fd8a80be8-istiod-ca-cert\") pod \"e2e-api-a-ae3cd0-openshift-default-5dbcd797d5-hvcv4\" (UID: \"31803f91-4f1e-407f-b82e-386fd8a80be8\") " pod="openshift-ingress/e2e-api-a-ae3cd0-openshift-default-5dbcd797d5-hvcv4" Oct 31 00:18:54.900605 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:54.899536 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31803f91-4f1e-407f-b82e-386fd8a80be8-istio-envoy\") pod \"e2e-api-a-ae3cd0-openshift-default-5dbcd797d5-hvcv4\" (UID: \"31803f91-4f1e-407f-b82e-386fd8a80be8\") " pod="openshift-ingress/e2e-api-a-ae3cd0-openshift-default-5dbcd797d5-hvcv4" Oct 31 00:18:54.901919 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:54.900975 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31803f91-4f1e-407f-b82e-386fd8a80be8-istio-podinfo\") pod \"e2e-api-a-ae3cd0-openshift-default-5dbcd797d5-hvcv4\" (UID: \"31803f91-4f1e-407f-b82e-386fd8a80be8\") " pod="openshift-ingress/e2e-api-a-ae3cd0-openshift-default-5dbcd797d5-hvcv4" Oct 31 00:18:54.901919 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:54.901039 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31803f91-4f1e-407f-b82e-386fd8a80be8-credential-socket\") pod \"e2e-api-a-ae3cd0-openshift-default-5dbcd797d5-hvcv4\" (UID: \"31803f91-4f1e-407f-b82e-386fd8a80be8\") " pod="openshift-ingress/e2e-api-a-ae3cd0-openshift-default-5dbcd797d5-hvcv4" Oct 31 00:18:54.901919 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:54.901264 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31803f91-4f1e-407f-b82e-386fd8a80be8-istio-data\") pod \"e2e-api-a-ae3cd0-openshift-default-5dbcd797d5-hvcv4\" (UID: \"31803f91-4f1e-407f-b82e-386fd8a80be8\") " pod="openshift-ingress/e2e-api-a-ae3cd0-openshift-default-5dbcd797d5-hvcv4" Oct 31 00:18:54.901919 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:54.901602 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31803f91-4f1e-407f-b82e-386fd8a80be8-workload-socket\") pod \"e2e-api-a-ae3cd0-openshift-default-5dbcd797d5-hvcv4\" (UID: \"31803f91-4f1e-407f-b82e-386fd8a80be8\") " pod="openshift-ingress/e2e-api-a-ae3cd0-openshift-default-5dbcd797d5-hvcv4" Oct 31 00:18:54.901919 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:54.901838 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31803f91-4f1e-407f-b82e-386fd8a80be8-workload-certs\") pod \"e2e-api-a-ae3cd0-openshift-default-5dbcd797d5-hvcv4\" (UID: \"31803f91-4f1e-407f-b82e-386fd8a80be8\") " pod="openshift-ingress/e2e-api-a-ae3cd0-openshift-default-5dbcd797d5-hvcv4" Oct 31 00:18:54.983367 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:54.983265 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:18:54.988767 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:54.988739 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/31803f91-4f1e-407f-b82e-386fd8a80be8-istio-token\") pod \"e2e-api-a-ae3cd0-openshift-default-5dbcd797d5-hvcv4\" (UID: \"31803f91-4f1e-407f-b82e-386fd8a80be8\") " pod="openshift-ingress/e2e-api-a-ae3cd0-openshift-default-5dbcd797d5-hvcv4" Oct 31 00:18:54.989597 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:54.989527 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2f45\" (UniqueName: \"kubernetes.io/projected/31803f91-4f1e-407f-b82e-386fd8a80be8-kube-api-access-l2f45\") pod \"e2e-api-a-ae3cd0-openshift-default-5dbcd797d5-hvcv4\" (UID: \"31803f91-4f1e-407f-b82e-386fd8a80be8\") " pod="openshift-ingress/e2e-api-a-ae3cd0-openshift-default-5dbcd797d5-hvcv4" Oct 31 00:18:54.994052 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:54.994030 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-ae3cd0-openshift-default-5dbcd797d5-hvcv4" Oct 31 00:18:55.101186 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:55.101148 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bbe18c5f-bf04-448b-8385-e99804d6056d" path="/var/lib/kubelet/pods/bbe18c5f-bf04-448b-8385-e99804d6056d/volumes" Oct 31 00:18:55.446076 ip-10-0-141-180 kubenswrapper[4927]: W1031 00:18:55.446035 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31803f91_4f1e_407f_b82e_386fd8a80be8.slice/crio-681fe38e39b6fcce80cc7eaca86251337c0aaf6e3b408355fa84b3af0285124b WatchSource:0}: Error finding container 681fe38e39b6fcce80cc7eaca86251337c0aaf6e3b408355fa84b3af0285124b: Status 404 returned error can't find the container with id 681fe38e39b6fcce80cc7eaca86251337c0aaf6e3b408355fa84b3af0285124b Oct 31 00:18:55.448425 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:55.448397 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-ae3cd0-openshift-default-5dbcd797d5-hvcv4"] Oct 31 00:18:55.448899 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:55.448867 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:18:55.448961 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:55.448948 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:18:55.449003 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:55.448979 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:18:56.002729 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:56.002691 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ae3cd0-openshift-default-5dbcd797d5-hvcv4" event={"ID":"31803f91-4f1e-407f-b82e-386fd8a80be8","Type":"ContainerStarted","Data":"dac22421d14078433923c10beb17d05dee975d3befd8509fa2341aa6c6bfe2ae"} Oct 31 00:18:56.002908 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:56.002749 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ae3cd0-openshift-default-5dbcd797d5-hvcv4" event={"ID":"31803f91-4f1e-407f-b82e-386fd8a80be8","Type":"ContainerStarted","Data":"681fe38e39b6fcce80cc7eaca86251337c0aaf6e3b408355fa84b3af0285124b"} Oct 31 00:18:56.141296 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:56.141238 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-ae3cd0-openshift-default-5dbcd797d5-hvcv4" podStartSLOduration=2.141220688 podStartE2EDuration="2.141220688s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:18:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:18:56.13939583 +0000 UTC m=+2249.175726654" watchObservedRunningTime="2026-10-31 00:18:56.141220688 +0000 UTC m=+2249.177551510" Oct 31 00:18:56.994160 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:56.994127 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-ae3cd0-openshift-default-5dbcd797d5-hvcv4" Oct 31 00:18:57.000136 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:57.000103 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-ae3cd0-openshift-default-5dbcd797d5-hvcv4" Oct 31 00:18:57.011380 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:57.011345 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-ae3cd0-openshift-default-5dbcd797d5-hvcv4" Oct 31 00:18:57.012284 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:57.012263 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-ae3cd0-openshift-default-5dbcd797d5-hvcv4" Oct 31 00:18:58.924949 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:58.924912 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:18:59.138207 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:59.138044 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:18:59.295943 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:18:59.295853 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:01.325181 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:01.325133 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:02.126428 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:02.126382 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:02.335556 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:02.335412 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:02.548354 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:02.544446 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-ae3cd0-7ffbd9dc87-l54tr"] Oct 31 00:19:02.550119 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:02.550080 4927 cpu_manager.go:414] "containerMap: removing container" podUID="9ee78c4f-f1a0-4cc0-8441-106def585fff" containerName="istio-proxy" Oct 31 00:19:02.550354 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:02.550316 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ee78c4f-f1a0-4cc0-8441-106def585fff" containerName="istio-proxy" Oct 31 00:19:02.555975 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:02.555948 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ae3cd0-7ffbd9dc87-l54tr" Oct 31 00:19:02.619541 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:02.619504 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-ae3cd0-7ffbd9dc87-l54tr"] Oct 31 00:19:02.650293 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:02.650169 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-ae3cd0-dockercfg-jmpgn\"" Oct 31 00:19:02.650293 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:02.650261 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-ae3cd0\"" Oct 31 00:19:02.830132 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:02.778314 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e77c7f6d-5bda-4b89-b7f1-2f647f774d03-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-ae3cd0-7ffbd9dc87-l54tr\" (UID: \"e77c7f6d-5bda-4b89-b7f1-2f647f774d03\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-ae3cd0-7ffbd9dc87-l54tr" Oct 31 00:19:02.830132 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:02.822159 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9dh27\" (UniqueName: \"kubernetes.io/projected/e77c7f6d-5bda-4b89-b7f1-2f647f774d03-kube-api-access-9dh27\") pod \"payload-pre-processing-e2e-api-a-ae3cd0-7ffbd9dc87-l54tr\" (UID: \"e77c7f6d-5bda-4b89-b7f1-2f647f774d03\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-ae3cd0-7ffbd9dc87-l54tr" Oct 31 00:19:02.830132 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:02.822837 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-ae3cd0-9bb5478db-92qxb"] Oct 31 00:19:02.856689 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:02.856627 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-ae3cd0-9bb5478db-92qxb"] Oct 31 00:19:02.856859 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:02.856771 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-ae3cd0-9bb5478db-92qxb" Oct 31 00:19:02.923353 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:02.923305 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l5rz7\" (UniqueName: \"kubernetes.io/projected/1f3db095-44dc-4645-bc72-32bee635facf-kube-api-access-l5rz7\") pod \"payload-processing-e2e-api-a-ae3cd0-9bb5478db-92qxb\" (UID: \"1f3db095-44dc-4645-bc72-32bee635facf\") " pod="openshift-ingress/payload-processing-e2e-api-a-ae3cd0-9bb5478db-92qxb" Oct 31 00:19:02.923546 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:02.923476 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e77c7f6d-5bda-4b89-b7f1-2f647f774d03-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-ae3cd0-7ffbd9dc87-l54tr\" (UID: \"e77c7f6d-5bda-4b89-b7f1-2f647f774d03\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-ae3cd0-7ffbd9dc87-l54tr" Oct 31 00:19:02.923546 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:02.923514 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f3db095-44dc-4645-bc72-32bee635facf-plugins-config-volume\") pod \"payload-processing-e2e-api-a-ae3cd0-9bb5478db-92qxb\" (UID: \"1f3db095-44dc-4645-bc72-32bee635facf\") " pod="openshift-ingress/payload-processing-e2e-api-a-ae3cd0-9bb5478db-92qxb" Oct 31 00:19:02.923668 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:02.923599 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9dh27\" (UniqueName: \"kubernetes.io/projected/e77c7f6d-5bda-4b89-b7f1-2f647f774d03-kube-api-access-9dh27\") pod \"payload-pre-processing-e2e-api-a-ae3cd0-7ffbd9dc87-l54tr\" (UID: \"e77c7f6d-5bda-4b89-b7f1-2f647f774d03\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-ae3cd0-7ffbd9dc87-l54tr" Oct 31 00:19:02.924255 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:02.924230 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e77c7f6d-5bda-4b89-b7f1-2f647f774d03-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-ae3cd0-7ffbd9dc87-l54tr\" (UID: \"e77c7f6d-5bda-4b89-b7f1-2f647f774d03\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-ae3cd0-7ffbd9dc87-l54tr" Oct 31 00:19:02.986323 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:02.986292 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9dh27\" (UniqueName: \"kubernetes.io/projected/e77c7f6d-5bda-4b89-b7f1-2f647f774d03-kube-api-access-9dh27\") pod \"payload-pre-processing-e2e-api-a-ae3cd0-7ffbd9dc87-l54tr\" (UID: \"e77c7f6d-5bda-4b89-b7f1-2f647f774d03\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-ae3cd0-7ffbd9dc87-l54tr" Oct 31 00:19:03.025305 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:03.025272 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5rz7\" (UniqueName: \"kubernetes.io/projected/1f3db095-44dc-4645-bc72-32bee635facf-kube-api-access-l5rz7\") pod \"payload-processing-e2e-api-a-ae3cd0-9bb5478db-92qxb\" (UID: \"1f3db095-44dc-4645-bc72-32bee635facf\") " pod="openshift-ingress/payload-processing-e2e-api-a-ae3cd0-9bb5478db-92qxb" Oct 31 00:19:03.025525 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:03.025391 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f3db095-44dc-4645-bc72-32bee635facf-plugins-config-volume\") pod \"payload-processing-e2e-api-a-ae3cd0-9bb5478db-92qxb\" (UID: \"1f3db095-44dc-4645-bc72-32bee635facf\") " pod="openshift-ingress/payload-processing-e2e-api-a-ae3cd0-9bb5478db-92qxb" Oct 31 00:19:03.026400 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:03.026368 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f3db095-44dc-4645-bc72-32bee635facf-plugins-config-volume\") pod \"payload-processing-e2e-api-a-ae3cd0-9bb5478db-92qxb\" (UID: \"1f3db095-44dc-4645-bc72-32bee635facf\") " pod="openshift-ingress/payload-processing-e2e-api-a-ae3cd0-9bb5478db-92qxb" Oct 31 00:19:03.087930 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:03.087864 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5rz7\" (UniqueName: \"kubernetes.io/projected/1f3db095-44dc-4645-bc72-32bee635facf-kube-api-access-l5rz7\") pod \"payload-processing-e2e-api-a-ae3cd0-9bb5478db-92qxb\" (UID: \"1f3db095-44dc-4645-bc72-32bee635facf\") " pod="openshift-ingress/payload-processing-e2e-api-a-ae3cd0-9bb5478db-92qxb" Oct 31 00:19:03.175636 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:03.175599 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ae3cd0-7ffbd9dc87-l54tr" Oct 31 00:19:03.178384 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:03.178318 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-ae3cd0-9bb5478db-92qxb" Oct 31 00:19:03.565683 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:03.565647 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-ae3cd0-7ffbd9dc87-l54tr"] Oct 31 00:19:03.603116 ip-10-0-141-180 kubenswrapper[4927]: W1031 00:19:03.603076 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode77c7f6d_5bda_4b89_b7f1_2f647f774d03.slice/crio-9f5424e54450425c93cc07ff06416bf11b0d6abe8ea3d763de715311208797e0 WatchSource:0}: Error finding container 9f5424e54450425c93cc07ff06416bf11b0d6abe8ea3d763de715311208797e0: Status 404 returned error can't find the container with id 9f5424e54450425c93cc07ff06416bf11b0d6abe8ea3d763de715311208797e0 Oct 31 00:19:03.618522 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:03.618489 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-ae3cd0-9bb5478db-92qxb"] Oct 31 00:19:03.673437 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:03.673409 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-76b073-6d97458769-pplqh"] Oct 31 00:19:03.678627 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:03.678605 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-76b073-6d97458769-pplqh" Oct 31 00:19:03.733631 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:03.733602 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-76b073-serving-cert\"" Oct 31 00:19:03.735114 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:03.735084 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d4c3ca5-2a40-4959-b4df-8a24cf25b043-metrics-certs\") pod \"maas-api-e2e-inf-a-76b073-6d97458769-pplqh\" (UID: \"7d4c3ca5-2a40-4959-b4df-8a24cf25b043\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-76b073-6d97458769-pplqh" Oct 31 00:19:03.735248 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:03.735166 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cpqqp\" (UniqueName: \"kubernetes.io/projected/7d4c3ca5-2a40-4959-b4df-8a24cf25b043-kube-api-access-cpqqp\") pod \"maas-api-e2e-inf-a-76b073-6d97458769-pplqh\" (UID: \"7d4c3ca5-2a40-4959-b4df-8a24cf25b043\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-76b073-6d97458769-pplqh" Oct 31 00:19:03.735305 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:03.735280 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d4c3ca5-2a40-4959-b4df-8a24cf25b043-openshift-service-ca\") pod \"maas-api-e2e-inf-a-76b073-6d97458769-pplqh\" (UID: \"7d4c3ca5-2a40-4959-b4df-8a24cf25b043\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-76b073-6d97458769-pplqh" Oct 31 00:19:03.735450 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:03.735429 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7d4c3ca5-2a40-4959-b4df-8a24cf25b043-maas-api-tls\") pod \"maas-api-e2e-inf-a-76b073-6d97458769-pplqh\" (UID: \"7d4c3ca5-2a40-4959-b4df-8a24cf25b043\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-76b073-6d97458769-pplqh" Oct 31 00:19:03.737296 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:03.737246 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-76b073-6d97458769-pplqh"] Oct 31 00:19:03.836469 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:03.836390 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d4c3ca5-2a40-4959-b4df-8a24cf25b043-openshift-service-ca\") pod \"maas-api-e2e-inf-a-76b073-6d97458769-pplqh\" (UID: \"7d4c3ca5-2a40-4959-b4df-8a24cf25b043\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-76b073-6d97458769-pplqh" Oct 31 00:19:03.836469 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:03.836467 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7d4c3ca5-2a40-4959-b4df-8a24cf25b043-maas-api-tls\") pod \"maas-api-e2e-inf-a-76b073-6d97458769-pplqh\" (UID: \"7d4c3ca5-2a40-4959-b4df-8a24cf25b043\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-76b073-6d97458769-pplqh" Oct 31 00:19:03.836712 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:19:03.836640 4927 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-76b073-serving-cert: secret "maas-api-e2e-inf-a-76b073-serving-cert" not found Oct 31 00:19:03.836712 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:19:03.836706 4927 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7d4c3ca5-2a40-4959-b4df-8a24cf25b043-maas-api-tls podName:7d4c3ca5-2a40-4959-b4df-8a24cf25b043 nodeName:}" failed. No retries permitted until 2026-10-31 00:19:04.336684942 +0000 UTC m=+2257.373015756 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7d4c3ca5-2a40-4959-b4df-8a24cf25b043-maas-api-tls") pod "maas-api-e2e-inf-a-76b073-6d97458769-pplqh" (UID: "7d4c3ca5-2a40-4959-b4df-8a24cf25b043") : secret "maas-api-e2e-inf-a-76b073-serving-cert" not found Oct 31 00:19:03.836850 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:03.836636 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d4c3ca5-2a40-4959-b4df-8a24cf25b043-metrics-certs\") pod \"maas-api-e2e-inf-a-76b073-6d97458769-pplqh\" (UID: \"7d4c3ca5-2a40-4959-b4df-8a24cf25b043\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-76b073-6d97458769-pplqh" Oct 31 00:19:03.836850 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:03.836789 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cpqqp\" (UniqueName: \"kubernetes.io/projected/7d4c3ca5-2a40-4959-b4df-8a24cf25b043-kube-api-access-cpqqp\") pod \"maas-api-e2e-inf-a-76b073-6d97458769-pplqh\" (UID: \"7d4c3ca5-2a40-4959-b4df-8a24cf25b043\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-76b073-6d97458769-pplqh" Oct 31 00:19:03.837213 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:03.837188 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d4c3ca5-2a40-4959-b4df-8a24cf25b043-openshift-service-ca\") pod \"maas-api-e2e-inf-a-76b073-6d97458769-pplqh\" (UID: \"7d4c3ca5-2a40-4959-b4df-8a24cf25b043\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-76b073-6d97458769-pplqh" Oct 31 00:19:03.839033 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:03.839000 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d4c3ca5-2a40-4959-b4df-8a24cf25b043-metrics-certs\") pod \"maas-api-e2e-inf-a-76b073-6d97458769-pplqh\" (UID: \"7d4c3ca5-2a40-4959-b4df-8a24cf25b043\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-76b073-6d97458769-pplqh" Oct 31 00:19:03.866474 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:03.866434 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:03.900948 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:03.900915 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cpqqp\" (UniqueName: \"kubernetes.io/projected/7d4c3ca5-2a40-4959-b4df-8a24cf25b043-kube-api-access-cpqqp\") pod \"maas-api-e2e-inf-a-76b073-6d97458769-pplqh\" (UID: \"7d4c3ca5-2a40-4959-b4df-8a24cf25b043\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-76b073-6d97458769-pplqh" Oct 31 00:19:04.059409 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:04.059322 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:04.090573 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:04.090487 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ae3cd0-7ffbd9dc87-l54tr" event={"ID":"e77c7f6d-5bda-4b89-b7f1-2f647f774d03","Type":"ContainerStarted","Data":"c33fccba96501aec0f1a94c5ae1806295128eca56f9db13f60276151d97f2b2c"} Oct 31 00:19:04.090855 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:04.090816 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ae3cd0-7ffbd9dc87-l54tr" event={"ID":"e77c7f6d-5bda-4b89-b7f1-2f647f774d03","Type":"ContainerStarted","Data":"9f5424e54450425c93cc07ff06416bf11b0d6abe8ea3d763de715311208797e0"} Oct 31 00:19:04.091050 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:04.091031 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ae3cd0-7ffbd9dc87-l54tr" Oct 31 00:19:04.094021 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:04.093995 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-ae3cd0-9bb5478db-92qxb" event={"ID":"1f3db095-44dc-4645-bc72-32bee635facf","Type":"ContainerStarted","Data":"4b78c94d4f323ee60628d81df4364bf5440cd6daa2f3acea24537d781ac02e72"} Oct 31 00:19:04.094192 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:04.094175 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-ae3cd0-9bb5478db-92qxb" event={"ID":"1f3db095-44dc-4645-bc72-32bee635facf","Type":"ContainerStarted","Data":"25c9f6a96ccd1204299fba0ee938e9ff0a28ce19806f6408cd61c03a2411dd4d"} Oct 31 00:19:04.095123 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:04.095087 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-ae3cd0-9bb5478db-92qxb" Oct 31 00:19:04.215288 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:04.215229 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ae3cd0-7ffbd9dc87-l54tr" podStartSLOduration=2.215214789 podStartE2EDuration="2.215214789s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:19:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:19:04.214570429 +0000 UTC m=+2257.250901276" watchObservedRunningTime="2026-10-31 00:19:04.215214789 +0000 UTC m=+2257.251545611" Oct 31 00:19:04.229296 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:04.229257 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:04.360800 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:04.360672 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7d4c3ca5-2a40-4959-b4df-8a24cf25b043-maas-api-tls\") pod \"maas-api-e2e-inf-a-76b073-6d97458769-pplqh\" (UID: \"7d4c3ca5-2a40-4959-b4df-8a24cf25b043\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-76b073-6d97458769-pplqh" Oct 31 00:19:04.391882 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:04.381095 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-ae3cd0-9bb5478db-92qxb" podStartSLOduration=2.381073753 podStartE2EDuration="2.381073753s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:19:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:19:04.374827713 +0000 UTC m=+2257.411158556" watchObservedRunningTime="2026-10-31 00:19:04.381073753 +0000 UTC m=+2257.417404576" Oct 31 00:19:04.393273 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:04.392913 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7d4c3ca5-2a40-4959-b4df-8a24cf25b043-maas-api-tls\") pod \"maas-api-e2e-inf-a-76b073-6d97458769-pplqh\" (UID: \"7d4c3ca5-2a40-4959-b4df-8a24cf25b043\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-76b073-6d97458769-pplqh" Oct 31 00:19:04.593660 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:04.593629 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-76b073-6d97458769-pplqh" Oct 31 00:19:04.993284 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:04.993251 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-76b073-6d97458769-pplqh"] Oct 31 00:19:04.994099 ip-10-0-141-180 kubenswrapper[4927]: W1031 00:19:04.994073 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d4c3ca5_2a40_4959_b4df_8a24cf25b043.slice/crio-768d2be4d7dea41fdade7a3b3bc7cc50d6cba35ca3c7fb3944b80e3fc62ee5ef WatchSource:0}: Error finding container 768d2be4d7dea41fdade7a3b3bc7cc50d6cba35ca3c7fb3944b80e3fc62ee5ef: Status 404 returned error can't find the container with id 768d2be4d7dea41fdade7a3b3bc7cc50d6cba35ca3c7fb3944b80e3fc62ee5ef Oct 31 00:19:05.106060 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:05.106020 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-76b073-6d97458769-pplqh" event={"ID":"7d4c3ca5-2a40-4959-b4df-8a24cf25b043","Type":"ContainerStarted","Data":"768d2be4d7dea41fdade7a3b3bc7cc50d6cba35ca3c7fb3944b80e3fc62ee5ef"} Oct 31 00:19:06.120347 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:06.120290 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-76b073-6d97458769-pplqh" event={"ID":"7d4c3ca5-2a40-4959-b4df-8a24cf25b043","Type":"ContainerStarted","Data":"e9771c90fb3593a57dcc470c6f32dd42cdc8b1515a3292854cb03fb9ba13a3b7"} Oct 31 00:19:06.120708 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:06.120629 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-76b073-6d97458769-pplqh" Oct 31 00:19:06.240883 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:06.240835 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-76b073-6d97458769-pplqh" podStartSLOduration=2.232093858 podStartE2EDuration="3.240821119s" totalImagesPullingTime="1.008727261s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:19:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:19:06.240383516 +0000 UTC m=+2259.276714355" watchObservedRunningTime="2026-10-31 00:19:06.240821119 +0000 UTC m=+2259.277151942" Oct 31 00:19:09.027678 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:09.027619 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:09.199718 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:09.199679 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:10.555715 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:10.555682 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:12.134469 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:12.134436 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-76b073-6d97458769-pplqh" Oct 31 00:19:13.558817 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:13.558787 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-92e375-6c7f5568fd-s99n8"] Oct 31 00:19:13.563726 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:13.563708 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-92e375-6c7f5568fd-s99n8" Oct 31 00:19:13.622945 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:13.622912 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-92e375-6c7f5568fd-s99n8"] Oct 31 00:19:13.666594 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:13.666568 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-92e375-serving-cert\"" Oct 31 00:19:13.721955 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:13.721924 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlpvs\" (UniqueName: \"kubernetes.io/projected/2c32e6d3-ca5b-459f-b393-9ba509a1247c-kube-api-access-dlpvs\") pod \"maas-api-e2e-api-a-92e375-6c7f5568fd-s99n8\" (UID: \"2c32e6d3-ca5b-459f-b393-9ba509a1247c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-92e375-6c7f5568fd-s99n8" Oct 31 00:19:13.722116 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:13.722093 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2c32e6d3-ca5b-459f-b393-9ba509a1247c-maas-api-tls\") pod \"maas-api-e2e-api-a-92e375-6c7f5568fd-s99n8\" (UID: \"2c32e6d3-ca5b-459f-b393-9ba509a1247c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-92e375-6c7f5568fd-s99n8" Oct 31 00:19:13.722191 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:13.722177 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2c32e6d3-ca5b-459f-b393-9ba509a1247c-metrics-certs\") pod \"maas-api-e2e-api-a-92e375-6c7f5568fd-s99n8\" (UID: \"2c32e6d3-ca5b-459f-b393-9ba509a1247c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-92e375-6c7f5568fd-s99n8" Oct 31 00:19:13.722244 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:13.722233 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c32e6d3-ca5b-459f-b393-9ba509a1247c-openshift-service-ca\") pod \"maas-api-e2e-api-a-92e375-6c7f5568fd-s99n8\" (UID: \"2c32e6d3-ca5b-459f-b393-9ba509a1247c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-92e375-6c7f5568fd-s99n8" Oct 31 00:19:13.745794 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:13.745746 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:13.833558 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:13.833474 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2c32e6d3-ca5b-459f-b393-9ba509a1247c-maas-api-tls\") pod \"maas-api-e2e-api-a-92e375-6c7f5568fd-s99n8\" (UID: \"2c32e6d3-ca5b-459f-b393-9ba509a1247c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-92e375-6c7f5568fd-s99n8" Oct 31 00:19:13.833895 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:13.833868 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2c32e6d3-ca5b-459f-b393-9ba509a1247c-metrics-certs\") pod \"maas-api-e2e-api-a-92e375-6c7f5568fd-s99n8\" (UID: \"2c32e6d3-ca5b-459f-b393-9ba509a1247c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-92e375-6c7f5568fd-s99n8" Oct 31 00:19:13.834094 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:13.834074 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c32e6d3-ca5b-459f-b393-9ba509a1247c-openshift-service-ca\") pod \"maas-api-e2e-api-a-92e375-6c7f5568fd-s99n8\" (UID: \"2c32e6d3-ca5b-459f-b393-9ba509a1247c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-92e375-6c7f5568fd-s99n8" Oct 31 00:19:13.845875 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:13.845837 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlpvs\" (UniqueName: \"kubernetes.io/projected/2c32e6d3-ca5b-459f-b393-9ba509a1247c-kube-api-access-dlpvs\") pod \"maas-api-e2e-api-a-92e375-6c7f5568fd-s99n8\" (UID: \"2c32e6d3-ca5b-459f-b393-9ba509a1247c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-92e375-6c7f5568fd-s99n8" Oct 31 00:19:13.847925 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:19:13.847897 4927 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-92e375-serving-cert: secret "maas-api-e2e-api-a-92e375-serving-cert" not found Oct 31 00:19:13.848160 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:19:13.848143 4927 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2c32e6d3-ca5b-459f-b393-9ba509a1247c-maas-api-tls podName:2c32e6d3-ca5b-459f-b393-9ba509a1247c nodeName:}" failed. No retries permitted until 2026-10-31 00:19:14.348119757 +0000 UTC m=+2267.384450580 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/2c32e6d3-ca5b-459f-b393-9ba509a1247c-maas-api-tls") pod "maas-api-e2e-api-a-92e375-6c7f5568fd-s99n8" (UID: "2c32e6d3-ca5b-459f-b393-9ba509a1247c") : secret "maas-api-e2e-api-a-92e375-serving-cert" not found Oct 31 00:19:13.849760 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:13.849723 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c32e6d3-ca5b-459f-b393-9ba509a1247c-openshift-service-ca\") pod \"maas-api-e2e-api-a-92e375-6c7f5568fd-s99n8\" (UID: \"2c32e6d3-ca5b-459f-b393-9ba509a1247c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-92e375-6c7f5568fd-s99n8" Oct 31 00:19:13.858630 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:13.858573 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2c32e6d3-ca5b-459f-b393-9ba509a1247c-metrics-certs\") pod \"maas-api-e2e-api-a-92e375-6c7f5568fd-s99n8\" (UID: \"2c32e6d3-ca5b-459f-b393-9ba509a1247c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-92e375-6c7f5568fd-s99n8" Oct 31 00:19:13.909443 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:13.909414 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlpvs\" (UniqueName: \"kubernetes.io/projected/2c32e6d3-ca5b-459f-b393-9ba509a1247c-kube-api-access-dlpvs\") pod \"maas-api-e2e-api-a-92e375-6c7f5568fd-s99n8\" (UID: \"2c32e6d3-ca5b-459f-b393-9ba509a1247c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-92e375-6c7f5568fd-s99n8" Oct 31 00:19:13.944977 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:13.944941 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:14.108894 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:14.108804 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:14.355019 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:14.354980 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2c32e6d3-ca5b-459f-b393-9ba509a1247c-maas-api-tls\") pod \"maas-api-e2e-api-a-92e375-6c7f5568fd-s99n8\" (UID: \"2c32e6d3-ca5b-459f-b393-9ba509a1247c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-92e375-6c7f5568fd-s99n8" Oct 31 00:19:14.357577 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:14.357549 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2c32e6d3-ca5b-459f-b393-9ba509a1247c-maas-api-tls\") pod \"maas-api-e2e-api-a-92e375-6c7f5568fd-s99n8\" (UID: \"2c32e6d3-ca5b-459f-b393-9ba509a1247c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-92e375-6c7f5568fd-s99n8" Oct 31 00:19:14.478364 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:14.478308 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-92e375-6c7f5568fd-s99n8" Oct 31 00:19:14.872919 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:14.872885 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-92e375-6c7f5568fd-s99n8"] Oct 31 00:19:14.872919 ip-10-0-141-180 kubenswrapper[4927]: W1031 00:19:14.872901 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c32e6d3_ca5b_459f_b393_9ba509a1247c.slice/crio-b18accd9383843f2fcd9bc7e22f174a25fe4f2a2bec9e21219f3d4b30086e83d WatchSource:0}: Error finding container b18accd9383843f2fcd9bc7e22f174a25fe4f2a2bec9e21219f3d4b30086e83d: Status 404 returned error can't find the container with id b18accd9383843f2fcd9bc7e22f174a25fe4f2a2bec9e21219f3d4b30086e83d Oct 31 00:19:15.253680 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:15.253634 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-92e375-6c7f5568fd-s99n8" event={"ID":"2c32e6d3-ca5b-459f-b393-9ba509a1247c","Type":"ContainerStarted","Data":"b18accd9383843f2fcd9bc7e22f174a25fe4f2a2bec9e21219f3d4b30086e83d"} Oct 31 00:19:16.123709 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:16.123628 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ae3cd0-7ffbd9dc87-l54tr" Oct 31 00:19:16.124093 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:16.123733 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-ae3cd0-9bb5478db-92qxb" Oct 31 00:19:16.265037 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:16.264995 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-92e375-6c7f5568fd-s99n8" event={"ID":"2c32e6d3-ca5b-459f-b393-9ba509a1247c","Type":"ContainerStarted","Data":"bec463672838581119115800a07feec7fe3e73824910eaf1a9ad41a58eeaa70b"} Oct 31 00:19:16.265219 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:16.265070 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-92e375-6c7f5568fd-s99n8" Oct 31 00:19:17.131656 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:17.131616 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:17.385180 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:17.385096 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:17.455289 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:17.455216 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-92e375-6c7f5568fd-s99n8" podStartSLOduration=3.5829504009999997 podStartE2EDuration="4.455193708s" totalImagesPullingTime="872.243307ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:19:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:19:16.530653403 +0000 UTC m=+2269.566984240" watchObservedRunningTime="2026-10-31 00:19:17.455193708 +0000 UTC m=+2270.491524533" Oct 31 00:19:17.462226 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:17.462190 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-fxvfs"] Oct 31 00:19:17.462489 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:17.462460 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-fxvfs" podUID="671d81ce-d6d7-4973-a112-ad795eaeac84" containerName="istio-proxy" containerID="cri-o://dfb7b227af3f459480d783b2124743080f0c67f8b6813ababc0060db977bd188" gracePeriod=30 Oct 31 00:19:17.475405 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:17.475380 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-qmsfx"] Oct 31 00:19:17.475649 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:17.475626 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-qmsfx" podUID="4b7dab75-7510-49a6-ace3-f72da2adef72" containerName="istio-proxy" containerID="cri-o://cd1513b92d6b20f19598f3496630c1fbbd4d2c3e11b905dacb8d6cf3212592c3" gracePeriod=30 Oct 31 00:19:17.480995 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:17.480971 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-c55xx"] Oct 31 00:19:17.481158 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:17.481133 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-c55xx" podUID="40dd0190-5043-4730-8830-2e8aad25cade" containerName="istio-proxy" containerID="cri-o://6946f32d9ffb28294eff3233bc6fbd3298655fff11993cb9a3bfcea3307ea78e" gracePeriod=30 Oct 31 00:19:17.505670 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:17.505646 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-q6whh"] Oct 31 00:19:17.505851 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:17.505832 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-q6whh" podUID="f1a9527e-1f11-49b9-a3bf-dae62908d3a9" containerName="istio-proxy" containerID="cri-o://fb92d5c2411c710b2e44b4bb5e1228f647f9ded4358b64e748c2cb740913cca1" gracePeriod=30 Oct 31 00:19:17.712352 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:17.708990 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:18.588689 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:18.588647 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:19.679080 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:19:19.679038 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sqv69" containerName="extractor" Oct 31 00:19:22.279262 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:22.279231 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-92e375-6c7f5568fd-s99n8" Oct 31 00:19:22.654051 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:22.653974 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:22.976063 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:22.976032 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-fxvfs" Oct 31 00:19:23.043794 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:23.037421 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/671d81ce-d6d7-4973-a112-ad795eaeac84-istio-token\" (UniqueName: \"kubernetes.io/projected/671d81ce-d6d7-4973-a112-ad795eaeac84-istio-token\") pod \"671d81ce-d6d7-4973-a112-ad795eaeac84\" (UID: \"671d81ce-d6d7-4973-a112-ad795eaeac84\") " Oct 31 00:19:23.043794 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:23.037535 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/671d81ce-d6d7-4973-a112-ad795eaeac84-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/671d81ce-d6d7-4973-a112-ad795eaeac84-istiod-ca-cert\") pod \"671d81ce-d6d7-4973-a112-ad795eaeac84\" (UID: \"671d81ce-d6d7-4973-a112-ad795eaeac84\") " Oct 31 00:19:23.043794 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:23.037580 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/671d81ce-d6d7-4973-a112-ad795eaeac84-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/671d81ce-d6d7-4973-a112-ad795eaeac84-credential-socket\") pod \"671d81ce-d6d7-4973-a112-ad795eaeac84\" (UID: \"671d81ce-d6d7-4973-a112-ad795eaeac84\") " Oct 31 00:19:23.043794 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:23.037611 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/671d81ce-d6d7-4973-a112-ad795eaeac84-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/671d81ce-d6d7-4973-a112-ad795eaeac84-workload-certs\") pod \"671d81ce-d6d7-4973-a112-ad795eaeac84\" (UID: \"671d81ce-d6d7-4973-a112-ad795eaeac84\") " Oct 31 00:19:23.043794 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:23.037645 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/671d81ce-d6d7-4973-a112-ad795eaeac84-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/671d81ce-d6d7-4973-a112-ad795eaeac84-istio-podinfo\") pod \"671d81ce-d6d7-4973-a112-ad795eaeac84\" (UID: \"671d81ce-d6d7-4973-a112-ad795eaeac84\") " Oct 31 00:19:23.043794 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:23.037704 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/671d81ce-d6d7-4973-a112-ad795eaeac84-kube-api-access-krnpc\" (UniqueName: \"kubernetes.io/projected/671d81ce-d6d7-4973-a112-ad795eaeac84-kube-api-access-krnpc\") pod \"671d81ce-d6d7-4973-a112-ad795eaeac84\" (UID: \"671d81ce-d6d7-4973-a112-ad795eaeac84\") " Oct 31 00:19:23.043794 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:23.037744 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/671d81ce-d6d7-4973-a112-ad795eaeac84-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/671d81ce-d6d7-4973-a112-ad795eaeac84-workload-socket\") pod \"671d81ce-d6d7-4973-a112-ad795eaeac84\" (UID: \"671d81ce-d6d7-4973-a112-ad795eaeac84\") " Oct 31 00:19:23.043794 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:23.037854 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/671d81ce-d6d7-4973-a112-ad795eaeac84-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/671d81ce-d6d7-4973-a112-ad795eaeac84-istio-envoy\") pod \"671d81ce-d6d7-4973-a112-ad795eaeac84\" (UID: \"671d81ce-d6d7-4973-a112-ad795eaeac84\") " Oct 31 00:19:23.043794 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:23.037925 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/671d81ce-d6d7-4973-a112-ad795eaeac84-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/671d81ce-d6d7-4973-a112-ad795eaeac84-istio-data\") pod \"671d81ce-d6d7-4973-a112-ad795eaeac84\" (UID: \"671d81ce-d6d7-4973-a112-ad795eaeac84\") " Oct 31 00:19:23.056485 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:23.056440 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/671d81ce-d6d7-4973-a112-ad795eaeac84-istio-data" pod "671d81ce-d6d7-4973-a112-ad795eaeac84" (UID: "671d81ce-d6d7-4973-a112-ad795eaeac84"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:19:23.078355 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:23.076688 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/671d81ce-d6d7-4973-a112-ad795eaeac84-credential-socket" pod "671d81ce-d6d7-4973-a112-ad795eaeac84" (UID: "671d81ce-d6d7-4973-a112-ad795eaeac84"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:19:23.078355 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:23.076978 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/671d81ce-d6d7-4973-a112-ad795eaeac84-istiod-ca-cert" pod "671d81ce-d6d7-4973-a112-ad795eaeac84" (UID: "671d81ce-d6d7-4973-a112-ad795eaeac84"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:19:23.078355 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:23.077120 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/671d81ce-d6d7-4973-a112-ad795eaeac84-workload-socket" pod "671d81ce-d6d7-4973-a112-ad795eaeac84" (UID: "671d81ce-d6d7-4973-a112-ad795eaeac84"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:19:23.078355 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:23.077932 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/671d81ce-d6d7-4973-a112-ad795eaeac84-workload-certs" pod "671d81ce-d6d7-4973-a112-ad795eaeac84" (UID: "671d81ce-d6d7-4973-a112-ad795eaeac84"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:19:23.150644 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:23.142879 4927 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/671d81ce-d6d7-4973-a112-ad795eaeac84-istio-data\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:19:23.150644 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:23.142913 4927 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/671d81ce-d6d7-4973-a112-ad795eaeac84-istiod-ca-cert\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:19:23.150644 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:23.142932 4927 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/671d81ce-d6d7-4973-a112-ad795eaeac84-credential-socket\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:19:23.150644 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:23.142947 4927 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/671d81ce-d6d7-4973-a112-ad795eaeac84-workload-certs\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:19:23.150644 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:23.142962 4927 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/671d81ce-d6d7-4973-a112-ad795eaeac84-workload-socket\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:19:23.154852 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:23.154796 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/671d81ce-d6d7-4973-a112-ad795eaeac84-kube-api-access-krnpc" pod "671d81ce-d6d7-4973-a112-ad795eaeac84" (UID: "671d81ce-d6d7-4973-a112-ad795eaeac84"). InnerVolumeSpecName "kube-api-access-krnpc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:19:23.158557 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:23.158502 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/671d81ce-d6d7-4973-a112-ad795eaeac84-istio-envoy" pod "671d81ce-d6d7-4973-a112-ad795eaeac84" (UID: "671d81ce-d6d7-4973-a112-ad795eaeac84"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:19:23.163181 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:23.162714 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/671d81ce-d6d7-4973-a112-ad795eaeac84-istio-podinfo" pod "671d81ce-d6d7-4973-a112-ad795eaeac84" (UID: "671d81ce-d6d7-4973-a112-ad795eaeac84"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:19:23.179294 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:23.178901 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-qmsfx" Oct 31 00:19:23.188245 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:23.186544 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/671d81ce-d6d7-4973-a112-ad795eaeac84-istio-token" pod "671d81ce-d6d7-4973-a112-ad795eaeac84" (UID: "671d81ce-d6d7-4973-a112-ad795eaeac84"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:19:23.194181 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:23.194150 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-c55xx" Oct 31 00:19:23.247364 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:23.247067 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b7dab75-7510-49a6-ace3-f72da2adef72-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b7dab75-7510-49a6-ace3-f72da2adef72-istio-data\") pod \"4b7dab75-7510-49a6-ace3-f72da2adef72\" (UID: \"4b7dab75-7510-49a6-ace3-f72da2adef72\") " Oct 31 00:19:23.247364 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:23.247138 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b7dab75-7510-49a6-ace3-f72da2adef72-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b7dab75-7510-49a6-ace3-f72da2adef72-workload-certs\") pod \"4b7dab75-7510-49a6-ace3-f72da2adef72\" (UID: \"4b7dab75-7510-49a6-ace3-f72da2adef72\") " Oct 31 00:19:23.247364 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:23.247219 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40dd0190-5043-4730-8830-2e8aad25cade-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40dd0190-5043-4730-8830-2e8aad25cade-istio-envoy\") pod \"40dd0190-5043-4730-8830-2e8aad25cade\" (UID: \"40dd0190-5043-4730-8830-2e8aad25cade\") " Oct 31 00:19:23.247364 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:23.247304 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b7dab75-7510-49a6-ace3-f72da2adef72-istio-data" pod "4b7dab75-7510-49a6-ace3-f72da2adef72" (UID: "4b7dab75-7510-49a6-ace3-f72da2adef72"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:19:23.247983 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:23.247955 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b7dab75-7510-49a6-ace3-f72da2adef72-workload-certs" pod "4b7dab75-7510-49a6-ace3-f72da2adef72" (UID: "4b7dab75-7510-49a6-ace3-f72da2adef72"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:19:23.248070 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:23.248035 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b7dab75-7510-49a6-ace3-f72da2adef72-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b7dab75-7510-49a6-ace3-f72da2adef72-workload-socket\") pod \"4b7dab75-7510-49a6-ace3-f72da2adef72\" (UID: \"4b7dab75-7510-49a6-ace3-f72da2adef72\") " Oct 31 00:19:23.248134 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:23.248107 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/40dd0190-5043-4730-8830-2e8aad25cade-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40dd0190-5043-4730-8830-2e8aad25cade-istiod-ca-cert\") pod \"40dd0190-5043-4730-8830-2e8aad25cade\" (UID: \"40dd0190-5043-4730-8830-2e8aad25cade\") " Oct 31 00:19:23.248424 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:23.248401 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b7dab75-7510-49a6-ace3-f72da2adef72-workload-socket" pod "4b7dab75-7510-49a6-ace3-f72da2adef72" (UID: "4b7dab75-7510-49a6-ace3-f72da2adef72"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:19:23.248543 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:23.248507 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b7dab75-7510-49a6-ace3-f72da2adef72-istio-token\" (UniqueName: \"kubernetes.io/projected/4b7dab75-7510-49a6-ace3-f72da2adef72-istio-token\") pod \"4b7dab75-7510-49a6-ace3-f72da2adef72\" (UID: \"4b7dab75-7510-49a6-ace3-f72da2adef72\") " Oct 31 00:19:23.248651 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:23.248628 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40dd0190-5043-4730-8830-2e8aad25cade-istiod-ca-cert" pod "40dd0190-5043-4730-8830-2e8aad25cade" (UID: "40dd0190-5043-4730-8830-2e8aad25cade"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:19:23.248708 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:23.248691 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40dd0190-5043-4730-8830-2e8aad25cade-kube-api-access-ttk8x\" (UniqueName: \"kubernetes.io/projected/40dd0190-5043-4730-8830-2e8aad25cade-kube-api-access-ttk8x\") pod \"40dd0190-5043-4730-8830-2e8aad25cade\" (UID: \"40dd0190-5043-4730-8830-2e8aad25cade\") " Oct 31 00:19:23.248844 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:23.248826 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40dd0190-5043-4730-8830-2e8aad25cade-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40dd0190-5043-4730-8830-2e8aad25cade-credential-socket\") pod \"40dd0190-5043-4730-8830-2e8aad25cade\" (UID: \"40dd0190-5043-4730-8830-2e8aad25cade\") " Oct 31 00:19:23.249061 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:23.249028 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40dd0190-5043-4730-8830-2e8aad25cade-credential-socket" pod "40dd0190-5043-4730-8830-2e8aad25cade" (UID: "40dd0190-5043-4730-8830-2e8aad25cade"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:19:23.249144 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:23.249075 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b7dab75-7510-49a6-ace3-f72da2adef72-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b7dab75-7510-49a6-ace3-f72da2adef72-istio-envoy\") pod \"4b7dab75-7510-49a6-ace3-f72da2adef72\" (UID: \"4b7dab75-7510-49a6-ace3-f72da2adef72\") " Oct 31 00:19:23.249144 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:23.249110 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b7dab75-7510-49a6-ace3-f72da2adef72-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b7dab75-7510-49a6-ace3-f72da2adef72-credential-socket\") pod \"4b7dab75-7510-49a6-ace3-f72da2adef72\" (UID: \"4b7dab75-7510-49a6-ace3-f72da2adef72\") " Oct 31 00:19:23.249276 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:23.249180 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/40dd0190-5043-4730-8830-2e8aad25cade-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40dd0190-5043-4730-8830-2e8aad25cade-istio-podinfo\") pod \"40dd0190-5043-4730-8830-2e8aad25cade\" (UID: \"40dd0190-5043-4730-8830-2e8aad25cade\") " Oct 31 00:19:23.249276 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:23.249243 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4b7dab75-7510-49a6-ace3-f72da2adef72-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b7dab75-7510-49a6-ace3-f72da2adef72-istiod-ca-cert\") pod \"4b7dab75-7510-49a6-ace3-f72da2adef72\" (UID: \"4b7dab75-7510-49a6-ace3-f72da2adef72\") " Oct 31 00:19:23.249414 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:23.249276 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40dd0190-5043-4730-8830-2e8aad25cade-istio-token\" (UniqueName: \"kubernetes.io/projected/40dd0190-5043-4730-8830-2e8aad25cade-istio-token\") pod \"40dd0190-5043-4730-8830-2e8aad25cade\" (UID: \"40dd0190-5043-4730-8830-2e8aad25cade\") " Oct 31 00:19:23.249544 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:23.249523 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b7dab75-7510-49a6-ace3-f72da2adef72-credential-socket" pod "4b7dab75-7510-49a6-ace3-f72da2adef72" (UID: "4b7dab75-7510-49a6-ace3-f72da2adef72"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:19:23.249615 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:23.249583 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40dd0190-5043-4730-8830-2e8aad25cade-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40dd0190-5043-4730-8830-2e8aad25cade-workload-certs\") pod \"40dd0190-5043-4730-8830-2e8aad25cade\" (UID: \"40dd0190-5043-4730-8830-2e8aad25cade\") " Oct 31 00:19:23.249615 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:23.249596 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b7dab75-7510-49a6-ace3-f72da2adef72-istiod-ca-cert" pod "4b7dab75-7510-49a6-ace3-f72da2adef72" (UID: "4b7dab75-7510-49a6-ace3-f72da2adef72"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:19:23.249962 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:23.249925 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40dd0190-5043-4730-8830-2e8aad25cade-workload-certs" pod "40dd0190-5043-4730-8830-2e8aad25cade" (UID: "40dd0190-5043-4730-8830-2e8aad25cade"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:19:23.250077 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:23.249973 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40dd0190-5043-4730-8830-2e8aad25cade-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40dd0190-5043-4730-8830-2e8aad25cade-workload-socket\") pod \"40dd0190-5043-4730-8830-2e8aad25cade\" (UID: \"40dd0190-5043-4730-8830-2e8aad25cade\") " Oct 31 00:19:23.250409 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:23.250266 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-q6whh" Oct 31 00:19:23.250409 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:23.250206 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40dd0190-5043-4730-8830-2e8aad25cade-workload-socket" pod "40dd0190-5043-4730-8830-2e8aad25cade" (UID: "40dd0190-5043-4730-8830-2e8aad25cade"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:19:23.250409 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:23.250381 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40dd0190-5043-4730-8830-2e8aad25cade-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40dd0190-5043-4730-8830-2e8aad25cade-istio-data\") pod \"40dd0190-5043-4730-8830-2e8aad25cade\" (UID: \"40dd0190-5043-4730-8830-2e8aad25cade\") " Oct 31 00:19:23.250709 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:23.250600 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40dd0190-5043-4730-8830-2e8aad25cade-istio-data" pod "40dd0190-5043-4730-8830-2e8aad25cade" (UID: "40dd0190-5043-4730-8830-2e8aad25cade"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:19:23.251033 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:23.251010 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b7dab75-7510-49a6-ace3-f72da2adef72-kube-api-access-kg5lj\" (UniqueName: \"kubernetes.io/projected/4b7dab75-7510-49a6-ace3-f72da2adef72-kube-api-access-kg5lj\") pod \"4b7dab75-7510-49a6-ace3-f72da2adef72\" (UID: \"4b7dab75-7510-49a6-ace3-f72da2adef72\") " Oct 31 00:19:23.251138 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:23.251122 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4b7dab75-7510-49a6-ace3-f72da2adef72-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b7dab75-7510-49a6-ace3-f72da2adef72-istio-podinfo\") pod \"4b7dab75-7510-49a6-ace3-f72da2adef72\" (UID: \"4b7dab75-7510-49a6-ace3-f72da2adef72\") " Oct 31 00:19:23.251376 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:23.251023 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40dd0190-5043-4730-8830-2e8aad25cade-istio-envoy" pod "40dd0190-5043-4730-8830-2e8aad25cade" (UID: "40dd0190-5043-4730-8830-2e8aad25cade"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:19:23.252035 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:23.252011 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40dd0190-5043-4730-8830-2e8aad25cade-istio-token" pod "40dd0190-5043-4730-8830-2e8aad25cade" (UID: "40dd0190-5043-4730-8830-2e8aad25cade"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:19:23.252389 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:23.252321 4927 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b7dab75-7510-49a6-ace3-f72da2adef72-credential-socket\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:19:23.252389 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:23.252367 4927 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b7dab75-7510-49a6-ace3-f72da2adef72-istiod-ca-cert\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:19:23.252389 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:23.252382 4927 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40dd0190-5043-4730-8830-2e8aad25cade-istio-token\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:19:23.252580 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:23.252396 4927 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40dd0190-5043-4730-8830-2e8aad25cade-workload-certs\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:19:23.252580 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:23.252409 4927 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40dd0190-5043-4730-8830-2e8aad25cade-workload-socket\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:19:23.252580 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:23.252437 4927 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40dd0190-5043-4730-8830-2e8aad25cade-istio-data\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:19:23.252580 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:23.252450 4927 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/671d81ce-d6d7-4973-a112-ad795eaeac84-istio-token\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:19:23.252580 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:23.252463 4927 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/671d81ce-d6d7-4973-a112-ad795eaeac84-istio-podinfo\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:19:23.252580 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:23.252477 4927 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b7dab75-7510-49a6-ace3-f72da2adef72-istio-data\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:19:23.252580 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:23.252502 4927 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b7dab75-7510-49a6-ace3-f72da2adef72-workload-certs\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:19:23.252580 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:23.252517 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-krnpc\" (UniqueName: \"kubernetes.io/projected/671d81ce-d6d7-4973-a112-ad795eaeac84-kube-api-access-krnpc\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:19:23.252580 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:23.252532 4927 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40dd0190-5043-4730-8830-2e8aad25cade-istio-envoy\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:19:23.252580 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:23.252545 4927 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b7dab75-7510-49a6-ace3-f72da2adef72-workload-socket\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:19:23.252580 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:23.252559 4927 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40dd0190-5043-4730-8830-2e8aad25cade-istiod-ca-cert\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:19:23.252580 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:23.252585 4927 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40dd0190-5043-4730-8830-2e8aad25cade-credential-socket\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:19:23.253090 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:23.252599 4927 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/671d81ce-d6d7-4973-a112-ad795eaeac84-istio-envoy\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:19:23.253259 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:23.253220 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b7dab75-7510-49a6-ace3-f72da2adef72-kube-api-access-kg5lj" pod "4b7dab75-7510-49a6-ace3-f72da2adef72" (UID: "4b7dab75-7510-49a6-ace3-f72da2adef72"). InnerVolumeSpecName "kube-api-access-kg5lj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:19:23.253325 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:23.253257 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40dd0190-5043-4730-8830-2e8aad25cade-kube-api-access-ttk8x" pod "40dd0190-5043-4730-8830-2e8aad25cade" (UID: "40dd0190-5043-4730-8830-2e8aad25cade"). InnerVolumeSpecName "kube-api-access-ttk8x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:19:23.253510 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:23.253487 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b7dab75-7510-49a6-ace3-f72da2adef72-istio-token" pod "4b7dab75-7510-49a6-ace3-f72da2adef72" (UID: "4b7dab75-7510-49a6-ace3-f72da2adef72"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:19:23.253918 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:23.253894 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/40dd0190-5043-4730-8830-2e8aad25cade-istio-podinfo" pod "40dd0190-5043-4730-8830-2e8aad25cade" (UID: "40dd0190-5043-4730-8830-2e8aad25cade"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:19:23.254992 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:23.254967 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b7dab75-7510-49a6-ace3-f72da2adef72-istio-envoy" pod "4b7dab75-7510-49a6-ace3-f72da2adef72" (UID: "4b7dab75-7510-49a6-ace3-f72da2adef72"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:19:23.255144 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:23.255121 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4b7dab75-7510-49a6-ace3-f72da2adef72-istio-podinfo" pod "4b7dab75-7510-49a6-ace3-f72da2adef72" (UID: "4b7dab75-7510-49a6-ace3-f72da2adef72"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:19:23.337505 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:23.337466 4927 generic.go:356] "Generic (PLEG): container finished" podID="671d81ce-d6d7-4973-a112-ad795eaeac84" containerID="dfb7b227af3f459480d783b2124743080f0c67f8b6813ababc0060db977bd188" exitCode=0 Oct 31 00:19:23.337983 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:23.337654 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-fxvfs" event={"ID":"671d81ce-d6d7-4973-a112-ad795eaeac84","Type":"ContainerDied","Data":"dfb7b227af3f459480d783b2124743080f0c67f8b6813ababc0060db977bd188"} Oct 31 00:19:23.337983 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:23.337707 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-fxvfs" event={"ID":"671d81ce-d6d7-4973-a112-ad795eaeac84","Type":"ContainerDied","Data":"77ffd3b888e61b41da625582632dc12049d0d4beff6eede577ba8d0d82f246ef"} Oct 31 00:19:23.337983 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:23.337740 4927 scope.go:122] "RemoveContainer" containerID="dfb7b227af3f459480d783b2124743080f0c67f8b6813ababc0060db977bd188" Oct 31 00:19:23.337983 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:23.337905 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-fxvfs" Oct 31 00:19:23.341796 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:23.341772 4927 generic.go:356] "Generic (PLEG): container finished" podID="f1a9527e-1f11-49b9-a3bf-dae62908d3a9" containerID="fb92d5c2411c710b2e44b4bb5e1228f647f9ded4358b64e748c2cb740913cca1" exitCode=0 Oct 31 00:19:23.344071 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:23.341803 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-q6whh" Oct 31 00:19:23.344071 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:23.341818 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-q6whh" event={"ID":"f1a9527e-1f11-49b9-a3bf-dae62908d3a9","Type":"ContainerDied","Data":"fb92d5c2411c710b2e44b4bb5e1228f647f9ded4358b64e748c2cb740913cca1"} Oct 31 00:19:23.344071 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:23.342226 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-q6whh" event={"ID":"f1a9527e-1f11-49b9-a3bf-dae62908d3a9","Type":"ContainerDied","Data":"e066abb39050adca65f03b6be6f5d86baf3dd2998461838bc3cbd2c46c16df5c"} Oct 31 00:19:23.358375 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:23.345552 4927 generic.go:356] "Generic (PLEG): container finished" podID="4b7dab75-7510-49a6-ace3-f72da2adef72" containerID="cd1513b92d6b20f19598f3496630c1fbbd4d2c3e11b905dacb8d6cf3212592c3" exitCode=0 Oct 31 00:19:23.358375 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:23.345628 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-qmsfx" event={"ID":"4b7dab75-7510-49a6-ace3-f72da2adef72","Type":"ContainerDied","Data":"cd1513b92d6b20f19598f3496630c1fbbd4d2c3e11b905dacb8d6cf3212592c3"} Oct 31 00:19:23.358375 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:23.345714 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-qmsfx" event={"ID":"4b7dab75-7510-49a6-ace3-f72da2adef72","Type":"ContainerDied","Data":"17180e51f67db863588cc9349c62454864303f7dbee8ad9eb78c987e1d2c7d4a"} Oct 31 00:19:23.358375 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:23.346222 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-qmsfx" Oct 31 00:19:23.358375 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:23.352979 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1a9527e-1f11-49b9-a3bf-dae62908d3a9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1a9527e-1f11-49b9-a3bf-dae62908d3a9-workload-socket\") pod \"f1a9527e-1f11-49b9-a3bf-dae62908d3a9\" (UID: \"f1a9527e-1f11-49b9-a3bf-dae62908d3a9\") " Oct 31 00:19:23.358375 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:23.353021 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f1a9527e-1f11-49b9-a3bf-dae62908d3a9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1a9527e-1f11-49b9-a3bf-dae62908d3a9-istiod-ca-cert\") pod \"f1a9527e-1f11-49b9-a3bf-dae62908d3a9\" (UID: \"f1a9527e-1f11-49b9-a3bf-dae62908d3a9\") " Oct 31 00:19:23.358375 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:23.353064 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1a9527e-1f11-49b9-a3bf-dae62908d3a9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1a9527e-1f11-49b9-a3bf-dae62908d3a9-credential-socket\") pod \"f1a9527e-1f11-49b9-a3bf-dae62908d3a9\" (UID: \"f1a9527e-1f11-49b9-a3bf-dae62908d3a9\") " Oct 31 00:19:23.358375 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:23.353183 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1a9527e-1f11-49b9-a3bf-dae62908d3a9-istio-token\" (UniqueName: \"kubernetes.io/projected/f1a9527e-1f11-49b9-a3bf-dae62908d3a9-istio-token\") pod \"f1a9527e-1f11-49b9-a3bf-dae62908d3a9\" (UID: \"f1a9527e-1f11-49b9-a3bf-dae62908d3a9\") " Oct 31 00:19:23.358375 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:23.353266 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f1a9527e-1f11-49b9-a3bf-dae62908d3a9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1a9527e-1f11-49b9-a3bf-dae62908d3a9-istio-podinfo\") pod \"f1a9527e-1f11-49b9-a3bf-dae62908d3a9\" (UID: \"f1a9527e-1f11-49b9-a3bf-dae62908d3a9\") " Oct 31 00:19:23.358375 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:23.353692 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1a9527e-1f11-49b9-a3bf-dae62908d3a9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1a9527e-1f11-49b9-a3bf-dae62908d3a9-istio-data\") pod \"f1a9527e-1f11-49b9-a3bf-dae62908d3a9\" (UID: \"f1a9527e-1f11-49b9-a3bf-dae62908d3a9\") " Oct 31 00:19:23.358375 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:23.353740 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1a9527e-1f11-49b9-a3bf-dae62908d3a9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1a9527e-1f11-49b9-a3bf-dae62908d3a9-istio-envoy\") pod \"f1a9527e-1f11-49b9-a3bf-dae62908d3a9\" (UID: \"f1a9527e-1f11-49b9-a3bf-dae62908d3a9\") " Oct 31 00:19:23.358375 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:23.353824 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1a9527e-1f11-49b9-a3bf-dae62908d3a9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1a9527e-1f11-49b9-a3bf-dae62908d3a9-workload-certs\") pod \"f1a9527e-1f11-49b9-a3bf-dae62908d3a9\" (UID: \"f1a9527e-1f11-49b9-a3bf-dae62908d3a9\") " Oct 31 00:19:23.358375 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:23.353947 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1a9527e-1f11-49b9-a3bf-dae62908d3a9-kube-api-access-97dl5\" (UniqueName: \"kubernetes.io/projected/f1a9527e-1f11-49b9-a3bf-dae62908d3a9-kube-api-access-97dl5\") pod \"f1a9527e-1f11-49b9-a3bf-dae62908d3a9\" (UID: \"f1a9527e-1f11-49b9-a3bf-dae62908d3a9\") " Oct 31 00:19:23.358375 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:23.354480 4927 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b7dab75-7510-49a6-ace3-f72da2adef72-istio-token\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:19:23.358375 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:23.354501 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ttk8x\" (UniqueName: \"kubernetes.io/projected/40dd0190-5043-4730-8830-2e8aad25cade-kube-api-access-ttk8x\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:19:23.358375 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:23.354516 4927 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b7dab75-7510-49a6-ace3-f72da2adef72-istio-envoy\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:19:23.358375 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:23.354531 4927 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40dd0190-5043-4730-8830-2e8aad25cade-istio-podinfo\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:19:23.358375 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:23.354546 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kg5lj\" (UniqueName: \"kubernetes.io/projected/4b7dab75-7510-49a6-ace3-f72da2adef72-kube-api-access-kg5lj\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:19:23.358375 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:23.354560 4927 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b7dab75-7510-49a6-ace3-f72da2adef72-istio-podinfo\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:19:23.358375 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:23.357368 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1a9527e-1f11-49b9-a3bf-dae62908d3a9-kube-api-access-97dl5" pod "f1a9527e-1f11-49b9-a3bf-dae62908d3a9" (UID: "f1a9527e-1f11-49b9-a3bf-dae62908d3a9"). InnerVolumeSpecName "kube-api-access-97dl5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:19:23.358375 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:23.357633 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1a9527e-1f11-49b9-a3bf-dae62908d3a9-workload-socket" pod "f1a9527e-1f11-49b9-a3bf-dae62908d3a9" (UID: "f1a9527e-1f11-49b9-a3bf-dae62908d3a9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:19:23.358375 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:23.357905 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1a9527e-1f11-49b9-a3bf-dae62908d3a9-istiod-ca-cert" pod "f1a9527e-1f11-49b9-a3bf-dae62908d3a9" (UID: "f1a9527e-1f11-49b9-a3bf-dae62908d3a9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:19:23.358375 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:23.358116 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1a9527e-1f11-49b9-a3bf-dae62908d3a9-credential-socket" pod "f1a9527e-1f11-49b9-a3bf-dae62908d3a9" (UID: "f1a9527e-1f11-49b9-a3bf-dae62908d3a9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:19:23.362927 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:23.360516 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1a9527e-1f11-49b9-a3bf-dae62908d3a9-istio-token" pod "f1a9527e-1f11-49b9-a3bf-dae62908d3a9" (UID: "f1a9527e-1f11-49b9-a3bf-dae62908d3a9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:19:23.365372 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:23.363138 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f1a9527e-1f11-49b9-a3bf-dae62908d3a9-istio-podinfo" pod "f1a9527e-1f11-49b9-a3bf-dae62908d3a9" (UID: "f1a9527e-1f11-49b9-a3bf-dae62908d3a9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:19:23.365372 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:23.363432 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1a9527e-1f11-49b9-a3bf-dae62908d3a9-istio-data" pod "f1a9527e-1f11-49b9-a3bf-dae62908d3a9" (UID: "f1a9527e-1f11-49b9-a3bf-dae62908d3a9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:19:23.365549 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:23.365451 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1a9527e-1f11-49b9-a3bf-dae62908d3a9-istio-envoy" pod "f1a9527e-1f11-49b9-a3bf-dae62908d3a9" (UID: "f1a9527e-1f11-49b9-a3bf-dae62908d3a9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:19:23.365657 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:23.365630 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1a9527e-1f11-49b9-a3bf-dae62908d3a9-workload-certs" pod "f1a9527e-1f11-49b9-a3bf-dae62908d3a9" (UID: "f1a9527e-1f11-49b9-a3bf-dae62908d3a9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:19:23.366618 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:23.366594 4927 generic.go:356] "Generic (PLEG): container finished" podID="40dd0190-5043-4730-8830-2e8aad25cade" containerID="6946f32d9ffb28294eff3233bc6fbd3298655fff11993cb9a3bfcea3307ea78e" exitCode=0 Oct 31 00:19:23.366696 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:23.366682 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-c55xx" event={"ID":"40dd0190-5043-4730-8830-2e8aad25cade","Type":"ContainerDied","Data":"6946f32d9ffb28294eff3233bc6fbd3298655fff11993cb9a3bfcea3307ea78e"} Oct 31 00:19:23.366738 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:23.366727 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-c55xx" event={"ID":"40dd0190-5043-4730-8830-2e8aad25cade","Type":"ContainerDied","Data":"6735e23360b5646866d6fce27b567c6108dbe9eff234fffa3d72f50a50bf8a58"} Oct 31 00:19:23.366813 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:23.366801 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-c55xx" Oct 31 00:19:23.367454 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:23.367397 4927 scope.go:122] "RemoveContainer" containerID="dfb7b227af3f459480d783b2124743080f0c67f8b6813ababc0060db977bd188" Oct 31 00:19:23.367798 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:19:23.367778 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dfb7b227af3f459480d783b2124743080f0c67f8b6813ababc0060db977bd188\": container with ID starting with dfb7b227af3f459480d783b2124743080f0c67f8b6813ababc0060db977bd188 not found: ID does not exist" containerID="dfb7b227af3f459480d783b2124743080f0c67f8b6813ababc0060db977bd188" Oct 31 00:19:23.367898 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:23.367811 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dfb7b227af3f459480d783b2124743080f0c67f8b6813ababc0060db977bd188"} err="failed to get container status \"dfb7b227af3f459480d783b2124743080f0c67f8b6813ababc0060db977bd188\": rpc error: code = NotFound desc = could not find container \"dfb7b227af3f459480d783b2124743080f0c67f8b6813ababc0060db977bd188\": container with ID starting with dfb7b227af3f459480d783b2124743080f0c67f8b6813ababc0060db977bd188 not found: ID does not exist" Oct 31 00:19:23.367898 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:23.367846 4927 scope.go:122] "RemoveContainer" containerID="fb92d5c2411c710b2e44b4bb5e1228f647f9ded4358b64e748c2cb740913cca1" Oct 31 00:19:23.379540 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:23.379524 4927 scope.go:122] "RemoveContainer" containerID="fb92d5c2411c710b2e44b4bb5e1228f647f9ded4358b64e748c2cb740913cca1" Oct 31 00:19:23.379811 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:19:23.379787 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb92d5c2411c710b2e44b4bb5e1228f647f9ded4358b64e748c2cb740913cca1\": container with ID starting with fb92d5c2411c710b2e44b4bb5e1228f647f9ded4358b64e748c2cb740913cca1 not found: ID does not exist" containerID="fb92d5c2411c710b2e44b4bb5e1228f647f9ded4358b64e748c2cb740913cca1" Oct 31 00:19:23.379866 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:23.379818 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb92d5c2411c710b2e44b4bb5e1228f647f9ded4358b64e748c2cb740913cca1"} err="failed to get container status \"fb92d5c2411c710b2e44b4bb5e1228f647f9ded4358b64e748c2cb740913cca1\": rpc error: code = NotFound desc = could not find container \"fb92d5c2411c710b2e44b4bb5e1228f647f9ded4358b64e748c2cb740913cca1\": container with ID starting with fb92d5c2411c710b2e44b4bb5e1228f647f9ded4358b64e748c2cb740913cca1 not found: ID does not exist" Oct 31 00:19:23.379866 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:23.379835 4927 scope.go:122] "RemoveContainer" containerID="cd1513b92d6b20f19598f3496630c1fbbd4d2c3e11b905dacb8d6cf3212592c3" Oct 31 00:19:23.390102 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:23.390085 4927 scope.go:122] "RemoveContainer" containerID="cd1513b92d6b20f19598f3496630c1fbbd4d2c3e11b905dacb8d6cf3212592c3" Oct 31 00:19:23.390431 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:19:23.390410 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd1513b92d6b20f19598f3496630c1fbbd4d2c3e11b905dacb8d6cf3212592c3\": container with ID starting with cd1513b92d6b20f19598f3496630c1fbbd4d2c3e11b905dacb8d6cf3212592c3 not found: ID does not exist" containerID="cd1513b92d6b20f19598f3496630c1fbbd4d2c3e11b905dacb8d6cf3212592c3" Oct 31 00:19:23.390516 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:23.390438 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd1513b92d6b20f19598f3496630c1fbbd4d2c3e11b905dacb8d6cf3212592c3"} err="failed to get container status \"cd1513b92d6b20f19598f3496630c1fbbd4d2c3e11b905dacb8d6cf3212592c3\": rpc error: code = NotFound desc = could not find container \"cd1513b92d6b20f19598f3496630c1fbbd4d2c3e11b905dacb8d6cf3212592c3\": container with ID starting with cd1513b92d6b20f19598f3496630c1fbbd4d2c3e11b905dacb8d6cf3212592c3 not found: ID does not exist" Oct 31 00:19:23.390516 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:23.390454 4927 scope.go:122] "RemoveContainer" containerID="6946f32d9ffb28294eff3233bc6fbd3298655fff11993cb9a3bfcea3307ea78e" Oct 31 00:19:23.405260 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:23.404253 4927 scope.go:122] "RemoveContainer" containerID="6946f32d9ffb28294eff3233bc6fbd3298655fff11993cb9a3bfcea3307ea78e" Oct 31 00:19:23.405260 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:19:23.404720 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6946f32d9ffb28294eff3233bc6fbd3298655fff11993cb9a3bfcea3307ea78e\": container with ID starting with 6946f32d9ffb28294eff3233bc6fbd3298655fff11993cb9a3bfcea3307ea78e not found: ID does not exist" containerID="6946f32d9ffb28294eff3233bc6fbd3298655fff11993cb9a3bfcea3307ea78e" Oct 31 00:19:23.405260 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:23.404759 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6946f32d9ffb28294eff3233bc6fbd3298655fff11993cb9a3bfcea3307ea78e"} err="failed to get container status \"6946f32d9ffb28294eff3233bc6fbd3298655fff11993cb9a3bfcea3307ea78e\": rpc error: code = NotFound desc = could not find container \"6946f32d9ffb28294eff3233bc6fbd3298655fff11993cb9a3bfcea3307ea78e\": container with ID starting with 6946f32d9ffb28294eff3233bc6fbd3298655fff11993cb9a3bfcea3307ea78e not found: ID does not exist" Oct 31 00:19:23.525920 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:23.519149 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-97dl5\" (UniqueName: \"kubernetes.io/projected/f1a9527e-1f11-49b9-a3bf-dae62908d3a9-kube-api-access-97dl5\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:19:23.525920 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:23.519183 4927 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1a9527e-1f11-49b9-a3bf-dae62908d3a9-workload-socket\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:19:23.525920 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:23.519201 4927 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1a9527e-1f11-49b9-a3bf-dae62908d3a9-istiod-ca-cert\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:19:23.525920 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:23.519215 4927 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1a9527e-1f11-49b9-a3bf-dae62908d3a9-credential-socket\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:19:23.525920 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:23.519228 4927 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1a9527e-1f11-49b9-a3bf-dae62908d3a9-istio-token\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:19:23.525920 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:23.519242 4927 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1a9527e-1f11-49b9-a3bf-dae62908d3a9-istio-podinfo\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:19:23.525920 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:23.519258 4927 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1a9527e-1f11-49b9-a3bf-dae62908d3a9-istio-data\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:19:23.525920 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:23.519272 4927 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1a9527e-1f11-49b9-a3bf-dae62908d3a9-istio-envoy\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:19:23.525920 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:23.519286 4927 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1a9527e-1f11-49b9-a3bf-dae62908d3a9-workload-certs\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:19:23.661418 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:23.661373 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-39e3a8-d9cc779c5-zxtvc"] Oct 31 00:19:23.661648 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:23.661618 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-39e3a8-d9cc779c5-zxtvc" podUID="334b65b1-c1d7-48a8-8837-6f0268fbef2a" containerName="maas-api" containerID="cri-o://184cfba3d3e5360b6dd2bf23e9e2050a30009de770a4de69c2464924731ca392" gracePeriod=30 Oct 31 00:19:23.725949 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:23.724878 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:23.775378 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:23.775344 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-fxvfs"] Oct 31 00:19:23.779565 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:23.779536 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-fxvfs"] Oct 31 00:19:23.957097 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:23.954002 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-qmsfx"] Oct 31 00:19:23.957097 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:23.956975 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:23.960817 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:23.960789 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-qmsfx"] Oct 31 00:19:24.119356 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:24.112031 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:24.153705 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:24.143110 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-c55xx"] Oct 31 00:19:24.153705 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:24.144308 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-39e3a8-d9cc779c5-zxtvc" Oct 31 00:19:24.168380 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:24.168321 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-c55xx"] Oct 31 00:19:24.268184 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:24.268106 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/334b65b1-c1d7-48a8-8837-6f0268fbef2a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/334b65b1-c1d7-48a8-8837-6f0268fbef2a-openshift-service-ca\") pod \"334b65b1-c1d7-48a8-8837-6f0268fbef2a\" (UID: \"334b65b1-c1d7-48a8-8837-6f0268fbef2a\") " Oct 31 00:19:24.268184 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:24.268147 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/334b65b1-c1d7-48a8-8837-6f0268fbef2a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/334b65b1-c1d7-48a8-8837-6f0268fbef2a-metrics-certs\") pod \"334b65b1-c1d7-48a8-8837-6f0268fbef2a\" (UID: \"334b65b1-c1d7-48a8-8837-6f0268fbef2a\") " Oct 31 00:19:24.268435 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:24.268304 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/334b65b1-c1d7-48a8-8837-6f0268fbef2a-kube-api-access-7kpjb\" (UniqueName: \"kubernetes.io/projected/334b65b1-c1d7-48a8-8837-6f0268fbef2a-kube-api-access-7kpjb\") pod \"334b65b1-c1d7-48a8-8837-6f0268fbef2a\" (UID: \"334b65b1-c1d7-48a8-8837-6f0268fbef2a\") " Oct 31 00:19:24.268435 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:24.268363 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/334b65b1-c1d7-48a8-8837-6f0268fbef2a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/334b65b1-c1d7-48a8-8837-6f0268fbef2a-maas-api-tls\") pod \"334b65b1-c1d7-48a8-8837-6f0268fbef2a\" (UID: \"334b65b1-c1d7-48a8-8837-6f0268fbef2a\") " Oct 31 00:19:24.268578 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:24.268481 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/334b65b1-c1d7-48a8-8837-6f0268fbef2a-openshift-service-ca" pod "334b65b1-c1d7-48a8-8837-6f0268fbef2a" (UID: "334b65b1-c1d7-48a8-8837-6f0268fbef2a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:19:24.268886 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:24.268864 4927 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/334b65b1-c1d7-48a8-8837-6f0268fbef2a-openshift-service-ca\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:19:24.270505 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:24.270481 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/334b65b1-c1d7-48a8-8837-6f0268fbef2a-kube-api-access-7kpjb" pod "334b65b1-c1d7-48a8-8837-6f0268fbef2a" (UID: "334b65b1-c1d7-48a8-8837-6f0268fbef2a"). InnerVolumeSpecName "kube-api-access-7kpjb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:19:24.270762 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:24.270742 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/334b65b1-c1d7-48a8-8837-6f0268fbef2a-metrics-certs" pod "334b65b1-c1d7-48a8-8837-6f0268fbef2a" (UID: "334b65b1-c1d7-48a8-8837-6f0268fbef2a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:19:24.270826 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:24.270756 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/334b65b1-c1d7-48a8-8837-6f0268fbef2a-maas-api-tls" pod "334b65b1-c1d7-48a8-8837-6f0268fbef2a" (UID: "334b65b1-c1d7-48a8-8837-6f0268fbef2a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:19:24.315523 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:24.315485 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-q6whh"] Oct 31 00:19:24.319588 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:24.319558 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-q6whh"] Oct 31 00:19:24.353230 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:24.353180 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:24.370571 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:24.370538 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7kpjb\" (UniqueName: \"kubernetes.io/projected/334b65b1-c1d7-48a8-8837-6f0268fbef2a-kube-api-access-7kpjb\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:19:24.370571 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:24.370571 4927 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/334b65b1-c1d7-48a8-8837-6f0268fbef2a-maas-api-tls\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:19:24.370762 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:24.370589 4927 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/334b65b1-c1d7-48a8-8837-6f0268fbef2a-metrics-certs\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:19:24.377934 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:24.377907 4927 generic.go:356] "Generic (PLEG): container finished" podID="334b65b1-c1d7-48a8-8837-6f0268fbef2a" containerID="184cfba3d3e5360b6dd2bf23e9e2050a30009de770a4de69c2464924731ca392" exitCode=0 Oct 31 00:19:24.378074 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:24.377958 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-39e3a8-d9cc779c5-zxtvc" Oct 31 00:19:24.378136 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:24.377960 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-39e3a8-d9cc779c5-zxtvc" event={"ID":"334b65b1-c1d7-48a8-8837-6f0268fbef2a","Type":"ContainerDied","Data":"184cfba3d3e5360b6dd2bf23e9e2050a30009de770a4de69c2464924731ca392"} Oct 31 00:19:24.378136 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:24.378123 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-39e3a8-d9cc779c5-zxtvc" event={"ID":"334b65b1-c1d7-48a8-8837-6f0268fbef2a","Type":"ContainerDied","Data":"1d3b6365ee107d8c4cd88624c629dd879b0e7e7cf901fc4b813a789106cfa4c1"} Oct 31 00:19:24.378239 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:24.378150 4927 scope.go:122] "RemoveContainer" containerID="184cfba3d3e5360b6dd2bf23e9e2050a30009de770a4de69c2464924731ca392" Oct 31 00:19:24.392081 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:24.391858 4927 scope.go:122] "RemoveContainer" containerID="184cfba3d3e5360b6dd2bf23e9e2050a30009de770a4de69c2464924731ca392" Oct 31 00:19:24.392739 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:19:24.392658 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"184cfba3d3e5360b6dd2bf23e9e2050a30009de770a4de69c2464924731ca392\": container with ID starting with 184cfba3d3e5360b6dd2bf23e9e2050a30009de770a4de69c2464924731ca392 not found: ID does not exist" containerID="184cfba3d3e5360b6dd2bf23e9e2050a30009de770a4de69c2464924731ca392" Oct 31 00:19:24.392739 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:24.392707 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"184cfba3d3e5360b6dd2bf23e9e2050a30009de770a4de69c2464924731ca392"} err="failed to get container status \"184cfba3d3e5360b6dd2bf23e9e2050a30009de770a4de69c2464924731ca392\": rpc error: code = NotFound desc = could not find container \"184cfba3d3e5360b6dd2bf23e9e2050a30009de770a4de69c2464924731ca392\": container with ID starting with 184cfba3d3e5360b6dd2bf23e9e2050a30009de770a4de69c2464924731ca392 not found: ID does not exist" Oct 31 00:19:24.616412 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:24.607553 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-39e3a8-d9cc779c5-zxtvc"] Oct 31 00:19:24.643562 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:24.643524 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-39e3a8-d9cc779c5-zxtvc"] Oct 31 00:19:25.011359 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:25.011306 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:25.087817 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:25.087784 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="334b65b1-c1d7-48a8-8837-6f0268fbef2a" path="/var/lib/kubelet/pods/334b65b1-c1d7-48a8-8837-6f0268fbef2a/volumes" Oct 31 00:19:25.088264 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:25.088249 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="40dd0190-5043-4730-8830-2e8aad25cade" path="/var/lib/kubelet/pods/40dd0190-5043-4730-8830-2e8aad25cade/volumes" Oct 31 00:19:25.088744 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:25.088728 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b7dab75-7510-49a6-ace3-f72da2adef72" path="/var/lib/kubelet/pods/4b7dab75-7510-49a6-ace3-f72da2adef72/volumes" Oct 31 00:19:25.089543 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:25.089528 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="671d81ce-d6d7-4973-a112-ad795eaeac84" path="/var/lib/kubelet/pods/671d81ce-d6d7-4973-a112-ad795eaeac84/volumes" Oct 31 00:19:25.089964 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:25.089951 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1a9527e-1f11-49b9-a3bf-dae62908d3a9" path="/var/lib/kubelet/pods/f1a9527e-1f11-49b9-a3bf-dae62908d3a9/volumes" Oct 31 00:19:25.188897 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:25.188861 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:25.399255 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:25.399160 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:25.663956 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:25.663919 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:26.176989 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:26.176956 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:30.489731 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:30.489693 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:30.918939 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:30.918902 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:31.225024 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:31.224934 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:31.401106 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:31.401015 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:31.779354 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:31.778755 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:33.660039 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:33.660001 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:33.856429 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:33.856390 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:34.066597 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:34.066493 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:34.072436 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:34.072399 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-39e3a8-openshift-default-6b6658797-ghqtx"] Oct 31 00:19:34.072642 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:34.072621 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-39e3a8-openshift-default-6b6658797-ghqtx" podUID="7a19c11a-45ef-434e-9667-e75e17898834" containerName="istio-proxy" containerID="cri-o://95c0c344e1207a0e409bc745275f504fef3764d367fd47a27a853fe7cb5c598e" gracePeriod=30 Oct 31 00:19:34.926505 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:34.926472 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:35.457359 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:35.452688 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-ddf61a-openshift-default-f57f54bc-4gvfk"] Oct 31 00:19:35.461261 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:35.461224 4927 cpu_manager.go:414] "containerMap: removing container" podUID="f1a9527e-1f11-49b9-a3bf-dae62908d3a9" containerName="istio-proxy" Oct 31 00:19:35.461261 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:35.461258 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1a9527e-1f11-49b9-a3bf-dae62908d3a9" containerName="istio-proxy" Oct 31 00:19:35.461546 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:35.461278 4927 cpu_manager.go:414] "containerMap: removing container" podUID="4b7dab75-7510-49a6-ace3-f72da2adef72" containerName="istio-proxy" Oct 31 00:19:35.461546 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:35.461287 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b7dab75-7510-49a6-ace3-f72da2adef72" containerName="istio-proxy" Oct 31 00:19:35.461546 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:35.461305 4927 cpu_manager.go:414] "containerMap: removing container" podUID="671d81ce-d6d7-4973-a112-ad795eaeac84" containerName="istio-proxy" Oct 31 00:19:35.461546 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:35.461313 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="671d81ce-d6d7-4973-a112-ad795eaeac84" containerName="istio-proxy" Oct 31 00:19:35.461546 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:35.461360 4927 cpu_manager.go:414] "containerMap: removing container" podUID="40dd0190-5043-4730-8830-2e8aad25cade" containerName="istio-proxy" Oct 31 00:19:35.461546 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:35.461370 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40dd0190-5043-4730-8830-2e8aad25cade" containerName="istio-proxy" Oct 31 00:19:35.461546 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:35.461388 4927 cpu_manager.go:414] "containerMap: removing container" podUID="334b65b1-c1d7-48a8-8837-6f0268fbef2a" containerName="maas-api" Oct 31 00:19:35.461546 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:35.461396 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="334b65b1-c1d7-48a8-8837-6f0268fbef2a" containerName="maas-api" Oct 31 00:19:35.461976 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:35.461590 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1a9527e-1f11-49b9-a3bf-dae62908d3a9" containerName="istio-proxy" Oct 31 00:19:35.461976 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:35.461607 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b7dab75-7510-49a6-ace3-f72da2adef72" containerName="istio-proxy" Oct 31 00:19:35.461976 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:35.461618 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="671d81ce-d6d7-4973-a112-ad795eaeac84" containerName="istio-proxy" Oct 31 00:19:35.461976 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:35.461632 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="40dd0190-5043-4730-8830-2e8aad25cade" containerName="istio-proxy" Oct 31 00:19:35.461976 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:35.461644 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="334b65b1-c1d7-48a8-8837-6f0268fbef2a" containerName="maas-api" Oct 31 00:19:35.470369 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:35.470311 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-ddf61a-openshift-default-f57f54bc-4gvfk" Oct 31 00:19:35.559357 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:35.536487 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-ddf61a-openshift-default-dockercfg-7rq8m\"" Oct 31 00:19:35.570844 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:35.570634 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:35.765794 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:35.765697 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a91e846-1ef4-4063-860e-46cdbf225ced-istiod-ca-cert\") pod \"e2e-inf-a-ddf61a-openshift-default-f57f54bc-4gvfk\" (UID: \"8a91e846-1ef4-4063-860e-46cdbf225ced\") " pod="openshift-ingress/e2e-inf-a-ddf61a-openshift-default-f57f54bc-4gvfk" Oct 31 00:19:35.766104 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:35.766078 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a91e846-1ef4-4063-860e-46cdbf225ced-istio-podinfo\") pod \"e2e-inf-a-ddf61a-openshift-default-f57f54bc-4gvfk\" (UID: \"8a91e846-1ef4-4063-860e-46cdbf225ced\") " pod="openshift-ingress/e2e-inf-a-ddf61a-openshift-default-f57f54bc-4gvfk" Oct 31 00:19:35.766272 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:35.766255 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a91e846-1ef4-4063-860e-46cdbf225ced-istio-data\") pod \"e2e-inf-a-ddf61a-openshift-default-f57f54bc-4gvfk\" (UID: \"8a91e846-1ef4-4063-860e-46cdbf225ced\") " pod="openshift-ingress/e2e-inf-a-ddf61a-openshift-default-f57f54bc-4gvfk" Oct 31 00:19:35.771352 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:35.766440 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a91e846-1ef4-4063-860e-46cdbf225ced-workload-socket\") pod \"e2e-inf-a-ddf61a-openshift-default-f57f54bc-4gvfk\" (UID: \"8a91e846-1ef4-4063-860e-46cdbf225ced\") " pod="openshift-ingress/e2e-inf-a-ddf61a-openshift-default-f57f54bc-4gvfk" Oct 31 00:19:35.785533 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:35.771614 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a91e846-1ef4-4063-860e-46cdbf225ced-credential-socket\") pod \"e2e-inf-a-ddf61a-openshift-default-f57f54bc-4gvfk\" (UID: \"8a91e846-1ef4-4063-860e-46cdbf225ced\") " pod="openshift-ingress/e2e-inf-a-ddf61a-openshift-default-f57f54bc-4gvfk" Oct 31 00:19:35.785833 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:35.785799 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a91e846-1ef4-4063-860e-46cdbf225ced-workload-certs\") pod \"e2e-inf-a-ddf61a-openshift-default-f57f54bc-4gvfk\" (UID: \"8a91e846-1ef4-4063-860e-46cdbf225ced\") " pod="openshift-ingress/e2e-inf-a-ddf61a-openshift-default-f57f54bc-4gvfk" Oct 31 00:19:35.789409 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:35.786091 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2g5r\" (UniqueName: \"kubernetes.io/projected/8a91e846-1ef4-4063-860e-46cdbf225ced-kube-api-access-s2g5r\") pod \"e2e-inf-a-ddf61a-openshift-default-f57f54bc-4gvfk\" (UID: \"8a91e846-1ef4-4063-860e-46cdbf225ced\") " pod="openshift-ingress/e2e-inf-a-ddf61a-openshift-default-f57f54bc-4gvfk" Oct 31 00:19:35.789700 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:35.789658 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a91e846-1ef4-4063-860e-46cdbf225ced-istio-envoy\") pod \"e2e-inf-a-ddf61a-openshift-default-f57f54bc-4gvfk\" (UID: \"8a91e846-1ef4-4063-860e-46cdbf225ced\") " pod="openshift-ingress/e2e-inf-a-ddf61a-openshift-default-f57f54bc-4gvfk" Oct 31 00:19:35.789966 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:35.789945 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a91e846-1ef4-4063-860e-46cdbf225ced-istio-token\") pod \"e2e-inf-a-ddf61a-openshift-default-f57f54bc-4gvfk\" (UID: \"8a91e846-1ef4-4063-860e-46cdbf225ced\") " pod="openshift-ingress/e2e-inf-a-ddf61a-openshift-default-f57f54bc-4gvfk" Oct 31 00:19:35.876809 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:35.876771 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:35.914496 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:35.905104 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a91e846-1ef4-4063-860e-46cdbf225ced-istiod-ca-cert\") pod \"e2e-inf-a-ddf61a-openshift-default-f57f54bc-4gvfk\" (UID: \"8a91e846-1ef4-4063-860e-46cdbf225ced\") " pod="openshift-ingress/e2e-inf-a-ddf61a-openshift-default-f57f54bc-4gvfk" Oct 31 00:19:35.914840 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:35.914820 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a91e846-1ef4-4063-860e-46cdbf225ced-istio-podinfo\") pod \"e2e-inf-a-ddf61a-openshift-default-f57f54bc-4gvfk\" (UID: \"8a91e846-1ef4-4063-860e-46cdbf225ced\") " pod="openshift-ingress/e2e-inf-a-ddf61a-openshift-default-f57f54bc-4gvfk" Oct 31 00:19:35.914978 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:35.914960 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a91e846-1ef4-4063-860e-46cdbf225ced-istio-data\") pod \"e2e-inf-a-ddf61a-openshift-default-f57f54bc-4gvfk\" (UID: \"8a91e846-1ef4-4063-860e-46cdbf225ced\") " pod="openshift-ingress/e2e-inf-a-ddf61a-openshift-default-f57f54bc-4gvfk" Oct 31 00:19:35.915135 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:35.915117 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a91e846-1ef4-4063-860e-46cdbf225ced-workload-socket\") pod \"e2e-inf-a-ddf61a-openshift-default-f57f54bc-4gvfk\" (UID: \"8a91e846-1ef4-4063-860e-46cdbf225ced\") " pod="openshift-ingress/e2e-inf-a-ddf61a-openshift-default-f57f54bc-4gvfk" Oct 31 00:19:35.915826 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:35.915802 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a91e846-1ef4-4063-860e-46cdbf225ced-credential-socket\") pod \"e2e-inf-a-ddf61a-openshift-default-f57f54bc-4gvfk\" (UID: \"8a91e846-1ef4-4063-860e-46cdbf225ced\") " pod="openshift-ingress/e2e-inf-a-ddf61a-openshift-default-f57f54bc-4gvfk" Oct 31 00:19:35.939491 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:35.939453 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a91e846-1ef4-4063-860e-46cdbf225ced-workload-certs\") pod \"e2e-inf-a-ddf61a-openshift-default-f57f54bc-4gvfk\" (UID: \"8a91e846-1ef4-4063-860e-46cdbf225ced\") " pod="openshift-ingress/e2e-inf-a-ddf61a-openshift-default-f57f54bc-4gvfk" Oct 31 00:19:35.940554 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:35.940530 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2g5r\" (UniqueName: \"kubernetes.io/projected/8a91e846-1ef4-4063-860e-46cdbf225ced-kube-api-access-s2g5r\") pod \"e2e-inf-a-ddf61a-openshift-default-f57f54bc-4gvfk\" (UID: \"8a91e846-1ef4-4063-860e-46cdbf225ced\") " pod="openshift-ingress/e2e-inf-a-ddf61a-openshift-default-f57f54bc-4gvfk" Oct 31 00:19:35.941101 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:35.941076 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a91e846-1ef4-4063-860e-46cdbf225ced-istio-envoy\") pod \"e2e-inf-a-ddf61a-openshift-default-f57f54bc-4gvfk\" (UID: \"8a91e846-1ef4-4063-860e-46cdbf225ced\") " pod="openshift-ingress/e2e-inf-a-ddf61a-openshift-default-f57f54bc-4gvfk" Oct 31 00:19:35.946491 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:35.946466 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a91e846-1ef4-4063-860e-46cdbf225ced-istio-token\") pod \"e2e-inf-a-ddf61a-openshift-default-f57f54bc-4gvfk\" (UID: \"8a91e846-1ef4-4063-860e-46cdbf225ced\") " pod="openshift-ingress/e2e-inf-a-ddf61a-openshift-default-f57f54bc-4gvfk" Oct 31 00:19:35.946825 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:35.940326 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a91e846-1ef4-4063-860e-46cdbf225ced-workload-certs\") pod \"e2e-inf-a-ddf61a-openshift-default-f57f54bc-4gvfk\" (UID: \"8a91e846-1ef4-4063-860e-46cdbf225ced\") " pod="openshift-ingress/e2e-inf-a-ddf61a-openshift-default-f57f54bc-4gvfk" Oct 31 00:19:35.963196 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:35.916262 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a91e846-1ef4-4063-860e-46cdbf225ced-istio-data\") pod \"e2e-inf-a-ddf61a-openshift-default-f57f54bc-4gvfk\" (UID: \"8a91e846-1ef4-4063-860e-46cdbf225ced\") " pod="openshift-ingress/e2e-inf-a-ddf61a-openshift-default-f57f54bc-4gvfk" Oct 31 00:19:35.963426 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:35.905885 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a91e846-1ef4-4063-860e-46cdbf225ced-istiod-ca-cert\") pod \"e2e-inf-a-ddf61a-openshift-default-f57f54bc-4gvfk\" (UID: \"8a91e846-1ef4-4063-860e-46cdbf225ced\") " pod="openshift-ingress/e2e-inf-a-ddf61a-openshift-default-f57f54bc-4gvfk" Oct 31 00:19:35.963584 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:35.916534 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a91e846-1ef4-4063-860e-46cdbf225ced-credential-socket\") pod \"e2e-inf-a-ddf61a-openshift-default-f57f54bc-4gvfk\" (UID: \"8a91e846-1ef4-4063-860e-46cdbf225ced\") " pod="openshift-ingress/e2e-inf-a-ddf61a-openshift-default-f57f54bc-4gvfk" Oct 31 00:19:35.963725 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:35.916791 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a91e846-1ef4-4063-860e-46cdbf225ced-workload-socket\") pod \"e2e-inf-a-ddf61a-openshift-default-f57f54bc-4gvfk\" (UID: \"8a91e846-1ef4-4063-860e-46cdbf225ced\") " pod="openshift-ingress/e2e-inf-a-ddf61a-openshift-default-f57f54bc-4gvfk" Oct 31 00:19:35.963853 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:35.963476 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a91e846-1ef4-4063-860e-46cdbf225ced-istio-podinfo\") pod \"e2e-inf-a-ddf61a-openshift-default-f57f54bc-4gvfk\" (UID: \"8a91e846-1ef4-4063-860e-46cdbf225ced\") " pod="openshift-ingress/e2e-inf-a-ddf61a-openshift-default-f57f54bc-4gvfk" Oct 31 00:19:35.978359 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:35.971840 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a91e846-1ef4-4063-860e-46cdbf225ced-istio-envoy\") pod \"e2e-inf-a-ddf61a-openshift-default-f57f54bc-4gvfk\" (UID: \"8a91e846-1ef4-4063-860e-46cdbf225ced\") " pod="openshift-ingress/e2e-inf-a-ddf61a-openshift-default-f57f54bc-4gvfk" Oct 31 00:19:35.978359 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:35.977438 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-ddf61a-openshift-default-f57f54bc-4gvfk"] Oct 31 00:19:36.005073 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:36.005039 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2g5r\" (UniqueName: \"kubernetes.io/projected/8a91e846-1ef4-4063-860e-46cdbf225ced-kube-api-access-s2g5r\") pod \"e2e-inf-a-ddf61a-openshift-default-f57f54bc-4gvfk\" (UID: \"8a91e846-1ef4-4063-860e-46cdbf225ced\") " pod="openshift-ingress/e2e-inf-a-ddf61a-openshift-default-f57f54bc-4gvfk" Oct 31 00:19:36.070402 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:36.067923 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a91e846-1ef4-4063-860e-46cdbf225ced-istio-token\") pod \"e2e-inf-a-ddf61a-openshift-default-f57f54bc-4gvfk\" (UID: \"8a91e846-1ef4-4063-860e-46cdbf225ced\") " pod="openshift-ingress/e2e-inf-a-ddf61a-openshift-default-f57f54bc-4gvfk" Oct 31 00:19:36.089412 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:36.089381 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-ddf61a-openshift-default-f57f54bc-4gvfk" Oct 31 00:19:36.665535 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:36.664877 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-ddf61a-openshift-default-f57f54bc-4gvfk"] Oct 31 00:19:36.670585 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:36.670543 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:19:36.670723 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:36.670615 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:19:36.670723 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:36.670645 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:19:37.501801 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:37.501762 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ddf61a-openshift-default-f57f54bc-4gvfk" event={"ID":"8a91e846-1ef4-4063-860e-46cdbf225ced","Type":"ContainerStarted","Data":"75c18b5577f51dca639c1901a93d0bd173d4a1772389f80ba0fe87b53cab3d52"} Oct 31 00:19:37.502183 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:37.501820 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ddf61a-openshift-default-f57f54bc-4gvfk" event={"ID":"8a91e846-1ef4-4063-860e-46cdbf225ced","Type":"ContainerStarted","Data":"5bcb39b4f13615bb376a993efcf7899a29402c386aded5bb45dd3316b316a884"} Oct 31 00:19:37.689566 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:37.689512 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-ddf61a-openshift-default-f57f54bc-4gvfk" podStartSLOduration=2.689499336 podStartE2EDuration="2.689499336s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:19:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:19:37.635130579 +0000 UTC m=+2290.671461428" watchObservedRunningTime="2026-10-31 00:19:37.689499336 +0000 UTC m=+2290.725830158" Oct 31 00:19:38.090273 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:38.090236 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-ddf61a-openshift-default-f57f54bc-4gvfk" Oct 31 00:19:38.099137 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:38.099108 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-ddf61a-openshift-default-f57f54bc-4gvfk" Oct 31 00:19:38.561355 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:38.556443 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-ddf61a-openshift-default-f57f54bc-4gvfk" Oct 31 00:19:38.568308 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:38.564761 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-ddf61a-openshift-default-f57f54bc-4gvfk" Oct 31 00:19:39.404918 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:39.404888 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-39e3a8-openshift-default-6b6658797-ghqtx" Oct 31 00:19:39.475845 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:39.475808 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a19c11a-45ef-434e-9667-e75e17898834-kube-api-access-qphrp\" (UniqueName: \"kubernetes.io/projected/7a19c11a-45ef-434e-9667-e75e17898834-kube-api-access-qphrp\") pod \"7a19c11a-45ef-434e-9667-e75e17898834\" (UID: \"7a19c11a-45ef-434e-9667-e75e17898834\") " Oct 31 00:19:39.476042 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:39.475863 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a19c11a-45ef-434e-9667-e75e17898834-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a19c11a-45ef-434e-9667-e75e17898834-workload-certs\") pod \"7a19c11a-45ef-434e-9667-e75e17898834\" (UID: \"7a19c11a-45ef-434e-9667-e75e17898834\") " Oct 31 00:19:39.476042 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:39.475928 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a19c11a-45ef-434e-9667-e75e17898834-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a19c11a-45ef-434e-9667-e75e17898834-istio-data\") pod \"7a19c11a-45ef-434e-9667-e75e17898834\" (UID: \"7a19c11a-45ef-434e-9667-e75e17898834\") " Oct 31 00:19:39.476042 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:39.475983 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a19c11a-45ef-434e-9667-e75e17898834-istio-token\" (UniqueName: \"kubernetes.io/projected/7a19c11a-45ef-434e-9667-e75e17898834-istio-token\") pod \"7a19c11a-45ef-434e-9667-e75e17898834\" (UID: \"7a19c11a-45ef-434e-9667-e75e17898834\") " Oct 31 00:19:39.476214 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:39.476060 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a19c11a-45ef-434e-9667-e75e17898834-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a19c11a-45ef-434e-9667-e75e17898834-credential-socket\") pod \"7a19c11a-45ef-434e-9667-e75e17898834\" (UID: \"7a19c11a-45ef-434e-9667-e75e17898834\") " Oct 31 00:19:39.476214 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:39.476091 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a19c11a-45ef-434e-9667-e75e17898834-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a19c11a-45ef-434e-9667-e75e17898834-istio-envoy\") pod \"7a19c11a-45ef-434e-9667-e75e17898834\" (UID: \"7a19c11a-45ef-434e-9667-e75e17898834\") " Oct 31 00:19:39.476214 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:39.476117 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7a19c11a-45ef-434e-9667-e75e17898834-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a19c11a-45ef-434e-9667-e75e17898834-istiod-ca-cert\") pod \"7a19c11a-45ef-434e-9667-e75e17898834\" (UID: \"7a19c11a-45ef-434e-9667-e75e17898834\") " Oct 31 00:19:39.476416 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:39.476239 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a19c11a-45ef-434e-9667-e75e17898834-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a19c11a-45ef-434e-9667-e75e17898834-workload-socket\") pod \"7a19c11a-45ef-434e-9667-e75e17898834\" (UID: \"7a19c11a-45ef-434e-9667-e75e17898834\") " Oct 31 00:19:39.476416 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:39.476257 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a19c11a-45ef-434e-9667-e75e17898834-workload-certs" pod "7a19c11a-45ef-434e-9667-e75e17898834" (UID: "7a19c11a-45ef-434e-9667-e75e17898834"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:19:39.476416 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:39.476270 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7a19c11a-45ef-434e-9667-e75e17898834-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a19c11a-45ef-434e-9667-e75e17898834-istio-podinfo\") pod \"7a19c11a-45ef-434e-9667-e75e17898834\" (UID: \"7a19c11a-45ef-434e-9667-e75e17898834\") " Oct 31 00:19:39.476416 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:39.476354 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a19c11a-45ef-434e-9667-e75e17898834-credential-socket" pod "7a19c11a-45ef-434e-9667-e75e17898834" (UID: "7a19c11a-45ef-434e-9667-e75e17898834"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:19:39.476796 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:39.476770 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a19c11a-45ef-434e-9667-e75e17898834-istio-data" pod "7a19c11a-45ef-434e-9667-e75e17898834" (UID: "7a19c11a-45ef-434e-9667-e75e17898834"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:19:39.477039 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:39.477014 4927 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a19c11a-45ef-434e-9667-e75e17898834-workload-certs\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:19:39.477119 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:39.477047 4927 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a19c11a-45ef-434e-9667-e75e17898834-credential-socket\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:19:39.477465 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:39.477442 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a19c11a-45ef-434e-9667-e75e17898834-istiod-ca-cert" pod "7a19c11a-45ef-434e-9667-e75e17898834" (UID: "7a19c11a-45ef-434e-9667-e75e17898834"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:19:39.477685 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:39.477628 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a19c11a-45ef-434e-9667-e75e17898834-workload-socket" pod "7a19c11a-45ef-434e-9667-e75e17898834" (UID: "7a19c11a-45ef-434e-9667-e75e17898834"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:19:39.478794 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:39.478766 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a19c11a-45ef-434e-9667-e75e17898834-kube-api-access-qphrp" pod "7a19c11a-45ef-434e-9667-e75e17898834" (UID: "7a19c11a-45ef-434e-9667-e75e17898834"). InnerVolumeSpecName "kube-api-access-qphrp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:19:39.479800 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:39.479772 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7a19c11a-45ef-434e-9667-e75e17898834-istio-podinfo" pod "7a19c11a-45ef-434e-9667-e75e17898834" (UID: "7a19c11a-45ef-434e-9667-e75e17898834"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:19:39.479908 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:39.479822 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a19c11a-45ef-434e-9667-e75e17898834-istio-envoy" pod "7a19c11a-45ef-434e-9667-e75e17898834" (UID: "7a19c11a-45ef-434e-9667-e75e17898834"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:19:39.479963 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:39.479917 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a19c11a-45ef-434e-9667-e75e17898834-istio-token" pod "7a19c11a-45ef-434e-9667-e75e17898834" (UID: "7a19c11a-45ef-434e-9667-e75e17898834"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:19:39.572849 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:39.572801 4927 generic.go:356] "Generic (PLEG): container finished" podID="7a19c11a-45ef-434e-9667-e75e17898834" containerID="95c0c344e1207a0e409bc745275f504fef3764d367fd47a27a853fe7cb5c598e" exitCode=0 Oct 31 00:19:39.573361 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:39.573242 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-39e3a8-openshift-default-6b6658797-ghqtx" Oct 31 00:19:39.574315 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:39.573795 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-39e3a8-openshift-default-6b6658797-ghqtx" event={"ID":"7a19c11a-45ef-434e-9667-e75e17898834","Type":"ContainerDied","Data":"95c0c344e1207a0e409bc745275f504fef3764d367fd47a27a853fe7cb5c598e"} Oct 31 00:19:39.574315 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:39.573854 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-39e3a8-openshift-default-6b6658797-ghqtx" event={"ID":"7a19c11a-45ef-434e-9667-e75e17898834","Type":"ContainerDied","Data":"079aa604033378b84f3b0900109d737a9c0665e94c9b4800f81f4b464e670138"} Oct 31 00:19:39.574315 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:39.573887 4927 scope.go:122] "RemoveContainer" containerID="95c0c344e1207a0e409bc745275f504fef3764d367fd47a27a853fe7cb5c598e" Oct 31 00:19:39.578612 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:39.578404 4927 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a19c11a-45ef-434e-9667-e75e17898834-istio-token\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:19:39.578612 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:39.578433 4927 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a19c11a-45ef-434e-9667-e75e17898834-istio-envoy\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:19:39.578612 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:39.578449 4927 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a19c11a-45ef-434e-9667-e75e17898834-istiod-ca-cert\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:19:39.578612 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:39.578466 4927 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a19c11a-45ef-434e-9667-e75e17898834-workload-socket\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:19:39.578612 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:39.578481 4927 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a19c11a-45ef-434e-9667-e75e17898834-istio-podinfo\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:19:39.578612 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:39.578497 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qphrp\" (UniqueName: \"kubernetes.io/projected/7a19c11a-45ef-434e-9667-e75e17898834-kube-api-access-qphrp\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:19:39.578612 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:39.578513 4927 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a19c11a-45ef-434e-9667-e75e17898834-istio-data\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:19:39.607320 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:39.606900 4927 scope.go:122] "RemoveContainer" containerID="77edaee9d31170d4678826952d723309b37e54f7f38e8893aa8fc44583812e4e" Oct 31 00:19:39.640106 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:39.639789 4927 scope.go:122] "RemoveContainer" containerID="95c0c344e1207a0e409bc745275f504fef3764d367fd47a27a853fe7cb5c598e" Oct 31 00:19:39.640918 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:19:39.640772 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95c0c344e1207a0e409bc745275f504fef3764d367fd47a27a853fe7cb5c598e\": container with ID starting with 95c0c344e1207a0e409bc745275f504fef3764d367fd47a27a853fe7cb5c598e not found: ID does not exist" containerID="95c0c344e1207a0e409bc745275f504fef3764d367fd47a27a853fe7cb5c598e" Oct 31 00:19:39.640918 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:39.640812 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95c0c344e1207a0e409bc745275f504fef3764d367fd47a27a853fe7cb5c598e"} err="failed to get container status \"95c0c344e1207a0e409bc745275f504fef3764d367fd47a27a853fe7cb5c598e\": rpc error: code = NotFound desc = could not find container \"95c0c344e1207a0e409bc745275f504fef3764d367fd47a27a853fe7cb5c598e\": container with ID starting with 95c0c344e1207a0e409bc745275f504fef3764d367fd47a27a853fe7cb5c598e not found: ID does not exist" Oct 31 00:19:39.640918 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:39.640835 4927 scope.go:122] "RemoveContainer" containerID="77edaee9d31170d4678826952d723309b37e54f7f38e8893aa8fc44583812e4e" Oct 31 00:19:39.641666 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:19:39.641642 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"77edaee9d31170d4678826952d723309b37e54f7f38e8893aa8fc44583812e4e\": container with ID starting with 77edaee9d31170d4678826952d723309b37e54f7f38e8893aa8fc44583812e4e not found: ID does not exist" containerID="77edaee9d31170d4678826952d723309b37e54f7f38e8893aa8fc44583812e4e" Oct 31 00:19:39.641767 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:39.641673 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77edaee9d31170d4678826952d723309b37e54f7f38e8893aa8fc44583812e4e"} err="failed to get container status \"77edaee9d31170d4678826952d723309b37e54f7f38e8893aa8fc44583812e4e\": rpc error: code = NotFound desc = could not find container \"77edaee9d31170d4678826952d723309b37e54f7f38e8893aa8fc44583812e4e\": container with ID starting with 77edaee9d31170d4678826952d723309b37e54f7f38e8893aa8fc44583812e4e not found: ID does not exist" Oct 31 00:19:39.686717 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:39.686625 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:39.780116 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:39.780061 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-39e3a8-openshift-default-6b6658797-ghqtx"] Oct 31 00:19:39.837768 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:39.837700 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-39e3a8-openshift-default-6b6658797-ghqtx"] Oct 31 00:19:40.505231 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:40.505146 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:40.679384 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:40.673957 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:40.850684 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:40.850591 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:41.090850 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:41.090816 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a19c11a-45ef-434e-9667-e75e17898834" path="/var/lib/kubelet/pods/7a19c11a-45ef-434e-9667-e75e17898834/volumes" Oct 31 00:19:41.335354 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:41.331828 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:41.541195 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:41.541152 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:41.716385 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:41.713753 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:42.257475 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:42.257430 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:43.743702 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:43.743668 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:44.533149 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:43.920480 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:44.533149 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:44.091557 4927 patch_prober.go:29] interesting pod/e2e-inf-a-76b073-openshift-default-697d759dd5-ql7sd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.230:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:19:44.533149 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:44.091617 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-76b073-openshift-default-697d759dd5-ql7sd" podUID="78b89047-2bad-472e-a80d-0ab8f1755877" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.230:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:19:44.640193 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:44.640160 4927 generic.go:356] "Generic (PLEG): container finished" podID="0802e5e7-e2ef-4a19-8663-6ba9b29ec0c2" containerID="678ab560f58ffa4cbda83d6e146dea152a309dabcc7fbecda4d40d0b9cf12dc7" exitCode=0 Oct 31 00:19:44.640437 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:44.640236 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-92e375-openshift-default-6987df58bc-jzxns" event={"ID":"0802e5e7-e2ef-4a19-8663-6ba9b29ec0c2","Type":"ContainerDied","Data":"678ab560f58ffa4cbda83d6e146dea152a309dabcc7fbecda4d40d0b9cf12dc7"} Oct 31 00:19:44.640796 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:44.640778 4927 scope.go:122] "RemoveContainer" containerID="678ab560f58ffa4cbda83d6e146dea152a309dabcc7fbecda4d40d0b9cf12dc7" Oct 31 00:19:44.654645 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:44.642857 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:19:44.654645 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:44.642959 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:19:44.654645 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:44.643013 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:19:45.650920 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:45.650882 4927 generic.go:356] "Generic (PLEG): container finished" podID="78b89047-2bad-472e-a80d-0ab8f1755877" containerID="cf6775d7b3f36612239f5a7641c6558beb5c7e657831a0ecd62d672653c6541e" exitCode=0 Oct 31 00:19:45.651399 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:45.650945 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-76b073-openshift-default-697d759dd5-ql7sd" event={"ID":"78b89047-2bad-472e-a80d-0ab8f1755877","Type":"ContainerDied","Data":"cf6775d7b3f36612239f5a7641c6558beb5c7e657831a0ecd62d672653c6541e"} Oct 31 00:19:45.651511 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:45.651491 4927 scope.go:122] "RemoveContainer" containerID="cf6775d7b3f36612239f5a7641c6558beb5c7e657831a0ecd62d672653c6541e" Oct 31 00:19:45.679896 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:45.671349 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:19:45.679896 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:45.671480 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:19:45.679896 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:45.671550 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:19:45.702572 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:45.702536 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-92e375-openshift-default-6987df58bc-jzxns" event={"ID":"0802e5e7-e2ef-4a19-8663-6ba9b29ec0c2","Type":"ContainerStarted","Data":"fffc5fcd5f608f9b8fffa309ac1f3d53a4c358b183655f1b894e9f796a69deb8"} Oct 31 00:19:45.937714 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:45.937202 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-92e375-openshift-default-6987df58bc-jzxns" Oct 31 00:19:45.939382 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:45.939045 4927 patch_prober.go:29] interesting pod/e2e-api-a-92e375-openshift-default-6987df58bc-jzxns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.231:15021/healthz/ready\": dial tcp 10.132.0.231:15021: connect: connection refused" start-of-body= Oct 31 00:19:45.939382 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:45.939111 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-92e375-openshift-default-6987df58bc-jzxns" podUID="0802e5e7-e2ef-4a19-8663-6ba9b29ec0c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.231:15021/healthz/ready\": dial tcp 10.132.0.231:15021: connect: connection refused" Oct 31 00:19:46.738431 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:46.737065 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-76b073-openshift-default-697d759dd5-ql7sd" event={"ID":"78b89047-2bad-472e-a80d-0ab8f1755877","Type":"ContainerStarted","Data":"00f406a06fb7820ebcbd0fc66650dde9c2b11ce43d216b26ac4b9206655d1205"} Oct 31 00:19:46.754736 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:46.754689 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:46.936354 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:46.936231 4927 patch_prober.go:29] interesting pod/e2e-api-a-92e375-openshift-default-6987df58bc-jzxns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.231:15021/healthz/ready\": dial tcp 10.132.0.231:15021: connect: connection refused" start-of-body= Oct 31 00:19:46.936354 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:46.936296 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-92e375-openshift-default-6987df58bc-jzxns" podUID="0802e5e7-e2ef-4a19-8663-6ba9b29ec0c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.231:15021/healthz/ready\": dial tcp 10.132.0.231:15021: connect: connection refused" Oct 31 00:19:47.130613 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:47.130446 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-92e375-openshift-default-6987df58bc-jzxns" Oct 31 00:19:47.141278 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:47.140706 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-76b073-openshift-default-697d759dd5-ql7sd" Oct 31 00:19:47.141278 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:47.140835 4927 patch_prober.go:29] interesting pod/e2e-inf-a-76b073-openshift-default-697d759dd5-ql7sd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.230:15021/healthz/ready\": dial tcp 10.132.0.230:15021: connect: connection refused" start-of-body= Oct 31 00:19:47.141278 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:47.140882 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-76b073-openshift-default-697d759dd5-ql7sd" podUID="78b89047-2bad-472e-a80d-0ab8f1755877" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.230:15021/healthz/ready\": dial tcp 10.132.0.230:15021: connect: connection refused" Oct 31 00:19:47.938802 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:47.938759 4927 patch_prober.go:29] interesting pod/e2e-api-a-92e375-openshift-default-6987df58bc-jzxns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.231:15021/healthz/ready\": dial tcp 10.132.0.231:15021: connect: connection refused" start-of-body= Oct 31 00:19:47.940163 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:47.940130 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-92e375-openshift-default-6987df58bc-jzxns" podUID="0802e5e7-e2ef-4a19-8663-6ba9b29ec0c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.231:15021/healthz/ready\": dial tcp 10.132.0.231:15021: connect: connection refused" Oct 31 00:19:48.145288 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:48.145251 4927 patch_prober.go:29] interesting pod/e2e-inf-a-76b073-openshift-default-697d759dd5-ql7sd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.230:15021/healthz/ready\": dial tcp 10.132.0.230:15021: connect: connection refused" start-of-body= Oct 31 00:19:48.145604 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:48.145568 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-76b073-openshift-default-697d759dd5-ql7sd" podUID="78b89047-2bad-472e-a80d-0ab8f1755877" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.230:15021/healthz/ready\": dial tcp 10.132.0.230:15021: connect: connection refused" Oct 31 00:19:48.810357 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:48.810309 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:48.935785 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:48.935745 4927 patch_prober.go:29] interesting pod/e2e-api-a-92e375-openshift-default-6987df58bc-jzxns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.231:15021/healthz/ready\": dial tcp 10.132.0.231:15021: connect: connection refused" start-of-body= Oct 31 00:19:48.936068 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:48.936040 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-92e375-openshift-default-6987df58bc-jzxns" podUID="0802e5e7-e2ef-4a19-8663-6ba9b29ec0c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.231:15021/healthz/ready\": dial tcp 10.132.0.231:15021: connect: connection refused" Oct 31 00:19:48.970274 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:48.970239 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:49.196352 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:49.195796 4927 patch_prober.go:29] interesting pod/e2e-inf-a-76b073-openshift-default-697d759dd5-ql7sd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.230:15021/healthz/ready\": dial tcp 10.132.0.230:15021: connect: connection refused" start-of-body= Oct 31 00:19:49.196352 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:49.195878 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-76b073-openshift-default-697d759dd5-ql7sd" podUID="78b89047-2bad-472e-a80d-0ab8f1755877" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.230:15021/healthz/ready\": dial tcp 10.132.0.230:15021: connect: connection refused" Oct 31 00:19:49.196352 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:49.195922 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-460d4e-openshift-default-575d5fbfb7-ftjgz"] Oct 31 00:19:49.196352 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:49.196150 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-460d4e-openshift-default-575d5fbfb7-ftjgz" podUID="d058674e-2706-41c4-8495-d7005ac1f562" containerName="istio-proxy" containerID="cri-o://b467bcb4497c1dc2f83567a7020a33b6ac5a1e9219577fb1c368f8866aa3ff14" gracePeriod=30 Oct 31 00:19:49.201906 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:49.199310 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:49.698930 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:19:49.698893 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sqv69" containerName="extractor" Oct 31 00:19:49.936849 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:49.936810 4927 patch_prober.go:29] interesting pod/e2e-api-a-92e375-openshift-default-6987df58bc-jzxns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.231:15021/healthz/ready\": dial tcp 10.132.0.231:15021: connect: connection refused" start-of-body= Oct 31 00:19:49.937595 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:49.937115 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-92e375-openshift-default-6987df58bc-jzxns" podUID="0802e5e7-e2ef-4a19-8663-6ba9b29ec0c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.231:15021/healthz/ready\": dial tcp 10.132.0.231:15021: connect: connection refused" Oct 31 00:19:50.144850 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:50.144758 4927 patch_prober.go:29] interesting pod/e2e-inf-a-76b073-openshift-default-697d759dd5-ql7sd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.230:15021/healthz/ready\": dial tcp 10.132.0.230:15021: connect: connection refused" start-of-body= Oct 31 00:19:50.145475 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:50.145437 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-76b073-openshift-default-697d759dd5-ql7sd" podUID="78b89047-2bad-472e-a80d-0ab8f1755877" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.230:15021/healthz/ready\": dial tcp 10.132.0.230:15021: connect: connection refused" Oct 31 00:19:50.945771 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:50.945725 4927 patch_prober.go:29] interesting pod/e2e-api-a-92e375-openshift-default-6987df58bc-jzxns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.231:15021/healthz/ready\": dial tcp 10.132.0.231:15021: connect: connection refused" start-of-body= Oct 31 00:19:50.947223 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:50.947186 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-92e375-openshift-default-6987df58bc-jzxns" podUID="0802e5e7-e2ef-4a19-8663-6ba9b29ec0c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.231:15021/healthz/ready\": dial tcp 10.132.0.231:15021: connect: connection refused" Oct 31 00:19:51.154160 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:51.154119 4927 patch_prober.go:29] interesting pod/e2e-inf-a-76b073-openshift-default-697d759dd5-ql7sd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.230:15021/healthz/ready\": dial tcp 10.132.0.230:15021: connect: connection refused" start-of-body= Oct 31 00:19:51.154760 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:51.154728 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-76b073-openshift-default-697d759dd5-ql7sd" podUID="78b89047-2bad-472e-a80d-0ab8f1755877" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.230:15021/healthz/ready\": dial tcp 10.132.0.230:15021: connect: connection refused" Oct 31 00:19:51.681960 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:51.681921 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:51.966412 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:51.966316 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-92e375-openshift-default-6987df58bc-jzxns" Oct 31 00:19:51.967878 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:51.967853 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-92e375-openshift-default-6987df58bc-jzxns" Oct 31 00:19:52.142522 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:52.142483 4927 patch_prober.go:29] interesting pod/e2e-inf-a-76b073-openshift-default-697d759dd5-ql7sd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.230:15021/healthz/ready\": dial tcp 10.132.0.230:15021: connect: connection refused" start-of-body= Oct 31 00:19:52.142802 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:52.142773 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-76b073-openshift-default-697d759dd5-ql7sd" podUID="78b89047-2bad-472e-a80d-0ab8f1755877" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.230:15021/healthz/ready\": dial tcp 10.132.0.230:15021: connect: connection refused" Oct 31 00:19:53.145432 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:53.145394 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-76b073-openshift-default-697d759dd5-ql7sd" Oct 31 00:19:53.145909 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:53.145886 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-76b073-openshift-default-697d759dd5-ql7sd" Oct 31 00:19:53.146780 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:53.146756 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-76b073-openshift-default-697d759dd5-ql7sd" Oct 31 00:19:53.672033 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:53.671989 4927 patch_prober.go:29] interesting pod/e2e-inf-b-460d4e-openshift-default-575d5fbfb7-ftjgz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 00:19:53.672207 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:53.672077 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-460d4e-openshift-default-575d5fbfb7-ftjgz" podUID="d058674e-2706-41c4-8495-d7005ac1f562" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:19:54.489951 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:54.489925 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-460d4e-openshift-default-575d5fbfb7-ftjgz" Oct 31 00:19:54.543672 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:54.543639 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d058674e-2706-41c4-8495-d7005ac1f562-istio-token\" (UniqueName: \"kubernetes.io/projected/d058674e-2706-41c4-8495-d7005ac1f562-istio-token\") pod \"d058674e-2706-41c4-8495-d7005ac1f562\" (UID: \"d058674e-2706-41c4-8495-d7005ac1f562\") " Oct 31 00:19:54.543880 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:54.543720 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d058674e-2706-41c4-8495-d7005ac1f562-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d058674e-2706-41c4-8495-d7005ac1f562-istio-podinfo\") pod \"d058674e-2706-41c4-8495-d7005ac1f562\" (UID: \"d058674e-2706-41c4-8495-d7005ac1f562\") " Oct 31 00:19:54.543880 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:54.543755 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d058674e-2706-41c4-8495-d7005ac1f562-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d058674e-2706-41c4-8495-d7005ac1f562-workload-certs\") pod \"d058674e-2706-41c4-8495-d7005ac1f562\" (UID: \"d058674e-2706-41c4-8495-d7005ac1f562\") " Oct 31 00:19:54.543880 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:54.543809 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d058674e-2706-41c4-8495-d7005ac1f562-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d058674e-2706-41c4-8495-d7005ac1f562-workload-socket\") pod \"d058674e-2706-41c4-8495-d7005ac1f562\" (UID: \"d058674e-2706-41c4-8495-d7005ac1f562\") " Oct 31 00:19:54.544058 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:54.543913 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d058674e-2706-41c4-8495-d7005ac1f562-kube-api-access-f6rfl\" (UniqueName: \"kubernetes.io/projected/d058674e-2706-41c4-8495-d7005ac1f562-kube-api-access-f6rfl\") pod \"d058674e-2706-41c4-8495-d7005ac1f562\" (UID: \"d058674e-2706-41c4-8495-d7005ac1f562\") " Oct 31 00:19:54.544058 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:54.543935 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d058674e-2706-41c4-8495-d7005ac1f562-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d058674e-2706-41c4-8495-d7005ac1f562-istio-data\") pod \"d058674e-2706-41c4-8495-d7005ac1f562\" (UID: \"d058674e-2706-41c4-8495-d7005ac1f562\") " Oct 31 00:19:54.544058 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:54.543977 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d058674e-2706-41c4-8495-d7005ac1f562-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d058674e-2706-41c4-8495-d7005ac1f562-istio-envoy\") pod \"d058674e-2706-41c4-8495-d7005ac1f562\" (UID: \"d058674e-2706-41c4-8495-d7005ac1f562\") " Oct 31 00:19:54.544058 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:54.544011 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d058674e-2706-41c4-8495-d7005ac1f562-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d058674e-2706-41c4-8495-d7005ac1f562-istiod-ca-cert\") pod \"d058674e-2706-41c4-8495-d7005ac1f562\" (UID: \"d058674e-2706-41c4-8495-d7005ac1f562\") " Oct 31 00:19:54.544279 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:54.544069 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d058674e-2706-41c4-8495-d7005ac1f562-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d058674e-2706-41c4-8495-d7005ac1f562-credential-socket\") pod \"d058674e-2706-41c4-8495-d7005ac1f562\" (UID: \"d058674e-2706-41c4-8495-d7005ac1f562\") " Oct 31 00:19:54.544693 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:54.544653 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d058674e-2706-41c4-8495-d7005ac1f562-workload-socket" pod "d058674e-2706-41c4-8495-d7005ac1f562" (UID: "d058674e-2706-41c4-8495-d7005ac1f562"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:19:54.544693 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:54.544670 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d058674e-2706-41c4-8495-d7005ac1f562-credential-socket" pod "d058674e-2706-41c4-8495-d7005ac1f562" (UID: "d058674e-2706-41c4-8495-d7005ac1f562"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:19:54.544985 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:54.544885 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d058674e-2706-41c4-8495-d7005ac1f562-workload-certs" pod "d058674e-2706-41c4-8495-d7005ac1f562" (UID: "d058674e-2706-41c4-8495-d7005ac1f562"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:19:54.544985 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:54.544895 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d058674e-2706-41c4-8495-d7005ac1f562-istio-data" pod "d058674e-2706-41c4-8495-d7005ac1f562" (UID: "d058674e-2706-41c4-8495-d7005ac1f562"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:19:54.545457 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:54.545413 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d058674e-2706-41c4-8495-d7005ac1f562-istiod-ca-cert" pod "d058674e-2706-41c4-8495-d7005ac1f562" (UID: "d058674e-2706-41c4-8495-d7005ac1f562"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:19:54.547294 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:54.547259 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d058674e-2706-41c4-8495-d7005ac1f562-istio-envoy" pod "d058674e-2706-41c4-8495-d7005ac1f562" (UID: "d058674e-2706-41c4-8495-d7005ac1f562"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:19:54.548147 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:54.548121 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d058674e-2706-41c4-8495-d7005ac1f562-istio-podinfo" pod "d058674e-2706-41c4-8495-d7005ac1f562" (UID: "d058674e-2706-41c4-8495-d7005ac1f562"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:19:54.548414 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:54.548392 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d058674e-2706-41c4-8495-d7005ac1f562-istio-token" pod "d058674e-2706-41c4-8495-d7005ac1f562" (UID: "d058674e-2706-41c4-8495-d7005ac1f562"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:19:54.548489 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:54.548425 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d058674e-2706-41c4-8495-d7005ac1f562-kube-api-access-f6rfl" pod "d058674e-2706-41c4-8495-d7005ac1f562" (UID: "d058674e-2706-41c4-8495-d7005ac1f562"). InnerVolumeSpecName "kube-api-access-f6rfl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:19:54.646362 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:54.646301 4927 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d058674e-2706-41c4-8495-d7005ac1f562-istio-envoy\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:19:54.646362 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:54.646356 4927 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d058674e-2706-41c4-8495-d7005ac1f562-istiod-ca-cert\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:19:54.646573 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:54.646374 4927 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d058674e-2706-41c4-8495-d7005ac1f562-credential-socket\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:19:54.646573 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:54.646390 4927 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d058674e-2706-41c4-8495-d7005ac1f562-istio-token\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:19:54.646573 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:54.646403 4927 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d058674e-2706-41c4-8495-d7005ac1f562-istio-podinfo\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:19:54.646573 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:54.646417 4927 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d058674e-2706-41c4-8495-d7005ac1f562-workload-certs\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:19:54.646573 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:54.646431 4927 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d058674e-2706-41c4-8495-d7005ac1f562-workload-socket\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:19:54.646573 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:54.646446 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f6rfl\" (UniqueName: \"kubernetes.io/projected/d058674e-2706-41c4-8495-d7005ac1f562-kube-api-access-f6rfl\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:19:54.646573 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:54.646460 4927 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d058674e-2706-41c4-8495-d7005ac1f562-istio-data\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:19:54.836914 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:54.836831 4927 generic.go:356] "Generic (PLEG): container finished" podID="d058674e-2706-41c4-8495-d7005ac1f562" containerID="b467bcb4497c1dc2f83567a7020a33b6ac5a1e9219577fb1c368f8866aa3ff14" exitCode=0 Oct 31 00:19:54.836914 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:54.836884 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-460d4e-openshift-default-575d5fbfb7-ftjgz" Oct 31 00:19:54.837107 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:54.836980 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-460d4e-openshift-default-575d5fbfb7-ftjgz" event={"ID":"d058674e-2706-41c4-8495-d7005ac1f562","Type":"ContainerDied","Data":"b467bcb4497c1dc2f83567a7020a33b6ac5a1e9219577fb1c368f8866aa3ff14"} Oct 31 00:19:54.837107 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:54.837016 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-460d4e-openshift-default-575d5fbfb7-ftjgz" event={"ID":"d058674e-2706-41c4-8495-d7005ac1f562","Type":"ContainerDied","Data":"95945db155b79d9dfff9785f9dd1e0b9c807ee1831c3f7c07214a4d2f5146fac"} Oct 31 00:19:54.837107 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:54.837037 4927 scope.go:122] "RemoveContainer" containerID="b467bcb4497c1dc2f83567a7020a33b6ac5a1e9219577fb1c368f8866aa3ff14" Oct 31 00:19:54.851163 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:54.851143 4927 scope.go:122] "RemoveContainer" containerID="7ff637e1e8bfb168ede519666b1271d01cd05dc174570cadd1fb02b573ac1d6d" Oct 31 00:19:54.863580 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:54.863560 4927 scope.go:122] "RemoveContainer" containerID="b467bcb4497c1dc2f83567a7020a33b6ac5a1e9219577fb1c368f8866aa3ff14" Oct 31 00:19:54.863872 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:19:54.863850 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b467bcb4497c1dc2f83567a7020a33b6ac5a1e9219577fb1c368f8866aa3ff14\": container with ID starting with b467bcb4497c1dc2f83567a7020a33b6ac5a1e9219577fb1c368f8866aa3ff14 not found: ID does not exist" containerID="b467bcb4497c1dc2f83567a7020a33b6ac5a1e9219577fb1c368f8866aa3ff14" Oct 31 00:19:54.863937 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:54.863879 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b467bcb4497c1dc2f83567a7020a33b6ac5a1e9219577fb1c368f8866aa3ff14"} err="failed to get container status \"b467bcb4497c1dc2f83567a7020a33b6ac5a1e9219577fb1c368f8866aa3ff14\": rpc error: code = NotFound desc = could not find container \"b467bcb4497c1dc2f83567a7020a33b6ac5a1e9219577fb1c368f8866aa3ff14\": container with ID starting with b467bcb4497c1dc2f83567a7020a33b6ac5a1e9219577fb1c368f8866aa3ff14 not found: ID does not exist" Oct 31 00:19:54.863937 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:54.863897 4927 scope.go:122] "RemoveContainer" containerID="7ff637e1e8bfb168ede519666b1271d01cd05dc174570cadd1fb02b573ac1d6d" Oct 31 00:19:54.864158 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:19:54.864140 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ff637e1e8bfb168ede519666b1271d01cd05dc174570cadd1fb02b573ac1d6d\": container with ID starting with 7ff637e1e8bfb168ede519666b1271d01cd05dc174570cadd1fb02b573ac1d6d not found: ID does not exist" containerID="7ff637e1e8bfb168ede519666b1271d01cd05dc174570cadd1fb02b573ac1d6d" Oct 31 00:19:54.864212 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:54.864164 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ff637e1e8bfb168ede519666b1271d01cd05dc174570cadd1fb02b573ac1d6d"} err="failed to get container status \"7ff637e1e8bfb168ede519666b1271d01cd05dc174570cadd1fb02b573ac1d6d\": rpc error: code = NotFound desc = could not find container \"7ff637e1e8bfb168ede519666b1271d01cd05dc174570cadd1fb02b573ac1d6d\": container with ID starting with 7ff637e1e8bfb168ede519666b1271d01cd05dc174570cadd1fb02b573ac1d6d not found: ID does not exist" Oct 31 00:19:55.010326 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:55.010286 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-460d4e-openshift-default-575d5fbfb7-ftjgz"] Oct 31 00:19:55.015042 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:55.015014 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-460d4e-openshift-default-575d5fbfb7-ftjgz"] Oct 31 00:19:55.087980 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:55.087903 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d058674e-2706-41c4-8495-d7005ac1f562" path="/var/lib/kubelet/pods/d058674e-2706-41c4-8495-d7005ac1f562/volumes" Oct 31 00:19:56.555589 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:19:56.555556 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:20:01.205983 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:01.205888 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:20:01.581309 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:01.581224 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-f7b8d8-576dfbf9f9-d9d8l"] Oct 31 00:20:01.830368 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:01.830297 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:20:02.046650 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:02.046613 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:20:05.713353 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:05.709649 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:20:05.887369 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:05.887229 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:20:06.582477 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:06.582430 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-f7b8d8-576dfbf9f9-d9d8l" podUID="4644c742-d4f8-4b7c-89c6-05e38fc39063" containerName="payload-pre-processing" containerID="cri-o://12fede5ab004f064eaab33f83a2b575896370cea5ba7d5d5ef00c0a6c2caa99e" gracePeriod=25 Oct 31 00:20:06.869250 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:06.869223 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-f7b8d8-576dfbf9f9-d9d8l" Oct 31 00:20:06.911297 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:06.911265 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4644c742-d4f8-4b7c-89c6-05e38fc39063-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4644c742-d4f8-4b7c-89c6-05e38fc39063-plugins-config-volume\") pod \"4644c742-d4f8-4b7c-89c6-05e38fc39063\" (UID: \"4644c742-d4f8-4b7c-89c6-05e38fc39063\") " Oct 31 00:20:06.911554 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:06.911537 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4644c742-d4f8-4b7c-89c6-05e38fc39063-kube-api-access-j5mdr\" (UniqueName: \"kubernetes.io/projected/4644c742-d4f8-4b7c-89c6-05e38fc39063-kube-api-access-j5mdr\") pod \"4644c742-d4f8-4b7c-89c6-05e38fc39063\" (UID: \"4644c742-d4f8-4b7c-89c6-05e38fc39063\") " Oct 31 00:20:06.911790 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:06.911757 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4644c742-d4f8-4b7c-89c6-05e38fc39063-plugins-config-volume" pod "4644c742-d4f8-4b7c-89c6-05e38fc39063" (UID: "4644c742-d4f8-4b7c-89c6-05e38fc39063"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:20:06.912240 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:06.912217 4927 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4644c742-d4f8-4b7c-89c6-05e38fc39063-plugins-config-volume\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:20:06.914017 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:06.913984 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4644c742-d4f8-4b7c-89c6-05e38fc39063-kube-api-access-j5mdr" pod "4644c742-d4f8-4b7c-89c6-05e38fc39063" (UID: "4644c742-d4f8-4b7c-89c6-05e38fc39063"). InnerVolumeSpecName "kube-api-access-j5mdr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:20:07.009393 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:07.009322 4927 generic.go:356] "Generic (PLEG): container finished" podID="4644c742-d4f8-4b7c-89c6-05e38fc39063" containerID="12fede5ab004f064eaab33f83a2b575896370cea5ba7d5d5ef00c0a6c2caa99e" exitCode=0 Oct 31 00:20:07.009598 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:07.009510 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-f7b8d8-576dfbf9f9-d9d8l" event={"ID":"4644c742-d4f8-4b7c-89c6-05e38fc39063","Type":"ContainerDied","Data":"12fede5ab004f064eaab33f83a2b575896370cea5ba7d5d5ef00c0a6c2caa99e"} Oct 31 00:20:07.009674 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:07.009626 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-f7b8d8-576dfbf9f9-d9d8l" event={"ID":"4644c742-d4f8-4b7c-89c6-05e38fc39063","Type":"ContainerDied","Data":"d4799cc809c9403c58819283754d9cb0a145672104855bf2514cd031d9cf84f5"} Oct 31 00:20:07.009736 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:07.009672 4927 scope.go:122] "RemoveContainer" containerID="12fede5ab004f064eaab33f83a2b575896370cea5ba7d5d5ef00c0a6c2caa99e" Oct 31 00:20:07.009802 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:07.009747 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-f7b8d8-576dfbf9f9-d9d8l" Oct 31 00:20:07.014255 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:07.014220 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j5mdr\" (UniqueName: \"kubernetes.io/projected/4644c742-d4f8-4b7c-89c6-05e38fc39063-kube-api-access-j5mdr\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:20:07.026142 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:07.026108 4927 scope.go:122] "RemoveContainer" containerID="12fede5ab004f064eaab33f83a2b575896370cea5ba7d5d5ef00c0a6c2caa99e" Oct 31 00:20:07.026596 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:20:07.026571 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12fede5ab004f064eaab33f83a2b575896370cea5ba7d5d5ef00c0a6c2caa99e\": container with ID starting with 12fede5ab004f064eaab33f83a2b575896370cea5ba7d5d5ef00c0a6c2caa99e not found: ID does not exist" containerID="12fede5ab004f064eaab33f83a2b575896370cea5ba7d5d5ef00c0a6c2caa99e" Oct 31 00:20:07.026712 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:07.026603 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12fede5ab004f064eaab33f83a2b575896370cea5ba7d5d5ef00c0a6c2caa99e"} err="failed to get container status \"12fede5ab004f064eaab33f83a2b575896370cea5ba7d5d5ef00c0a6c2caa99e\": rpc error: code = NotFound desc = could not find container \"12fede5ab004f064eaab33f83a2b575896370cea5ba7d5d5ef00c0a6c2caa99e\": container with ID starting with 12fede5ab004f064eaab33f83a2b575896370cea5ba7d5d5ef00c0a6c2caa99e not found: ID does not exist" Oct 31 00:20:07.182189 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:07.182149 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-f7b8d8-576dfbf9f9-d9d8l"] Oct 31 00:20:07.186740 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:07.186710 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-f7b8d8-576dfbf9f9-d9d8l"] Oct 31 00:20:09.088429 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:09.088388 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4644c742-d4f8-4b7c-89c6-05e38fc39063" path="/var/lib/kubelet/pods/4644c742-d4f8-4b7c-89c6-05e38fc39063/volumes" Oct 31 00:20:10.504272 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:10.504237 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:20:15.094878 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:15.094843 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:20:19.679995 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:20:19.679956 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sqv69" containerName="extractor" Oct 31 00:20:19.709898 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:19.709866 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:20:22.355259 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:22.355202 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:20:22.834741 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:22.834700 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:20:23.889537 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:23.889485 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:20:24.057062 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:24.057024 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:20:24.586637 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:24.586587 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:20:25.015157 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:25.015114 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:20:25.169761 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:25.169718 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:20:28.654358 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:28.650856 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:20:29.050429 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:29.048477 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:20:29.261794 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:29.261751 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:20:30.449261 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:30.449226 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:20:30.760179 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:30.760090 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:20:30.921364 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:30.918572 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:20:31.295657 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:31.295626 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:20:31.476183 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:31.476148 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:20:32.750696 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:32.750659 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:20:33.398283 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:33.398194 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:20:33.946162 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:33.946120 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-d687ff-openshift-default-5c75fcb6d5-25thf"] Oct 31 00:20:33.947039 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:33.947014 4927 cpu_manager.go:414] "containerMap: removing container" podUID="7a19c11a-45ef-434e-9667-e75e17898834" containerName="istio-proxy" Oct 31 00:20:33.947165 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:33.947040 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a19c11a-45ef-434e-9667-e75e17898834" containerName="istio-proxy" Oct 31 00:20:33.947165 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:33.947087 4927 cpu_manager.go:414] "containerMap: removing container" podUID="d058674e-2706-41c4-8495-d7005ac1f562" containerName="istio-proxy" Oct 31 00:20:33.947165 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:33.947095 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d058674e-2706-41c4-8495-d7005ac1f562" containerName="istio-proxy" Oct 31 00:20:33.947165 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:33.947112 4927 cpu_manager.go:414] "containerMap: removing container" podUID="d058674e-2706-41c4-8495-d7005ac1f562" containerName="istio-proxy" Oct 31 00:20:33.947165 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:33.947121 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d058674e-2706-41c4-8495-d7005ac1f562" containerName="istio-proxy" Oct 31 00:20:33.947165 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:33.947135 4927 cpu_manager.go:414] "containerMap: removing container" podUID="4644c742-d4f8-4b7c-89c6-05e38fc39063" containerName="payload-pre-processing" Oct 31 00:20:33.947165 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:33.947141 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4644c742-d4f8-4b7c-89c6-05e38fc39063" containerName="payload-pre-processing" Oct 31 00:20:33.947587 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:33.947354 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a19c11a-45ef-434e-9667-e75e17898834" containerName="istio-proxy" Oct 31 00:20:33.947587 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:33.947378 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="d058674e-2706-41c4-8495-d7005ac1f562" containerName="istio-proxy" Oct 31 00:20:33.947587 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:33.947392 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="d058674e-2706-41c4-8495-d7005ac1f562" containerName="istio-proxy" Oct 31 00:20:33.947587 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:33.947403 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="4644c742-d4f8-4b7c-89c6-05e38fc39063" containerName="payload-pre-processing" Oct 31 00:20:33.947587 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:33.947417 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a19c11a-45ef-434e-9667-e75e17898834" containerName="istio-proxy" Oct 31 00:20:33.952876 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:33.952851 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-d687ff-openshift-default-5c75fcb6d5-25thf" Oct 31 00:20:34.013692 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:34.013644 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-d687ff-openshift-default-dockercfg-zswjq\"" Oct 31 00:20:34.067224 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:34.067171 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/360cd796-06da-49c1-9bd4-f2f41f04efeb-workload-certs\") pod \"e2e-api-b-d687ff-openshift-default-5c75fcb6d5-25thf\" (UID: \"360cd796-06da-49c1-9bd4-f2f41f04efeb\") " pod="openshift-ingress/e2e-api-b-d687ff-openshift-default-5c75fcb6d5-25thf" Oct 31 00:20:34.067449 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:34.067310 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/360cd796-06da-49c1-9bd4-f2f41f04efeb-istio-podinfo\") pod \"e2e-api-b-d687ff-openshift-default-5c75fcb6d5-25thf\" (UID: \"360cd796-06da-49c1-9bd4-f2f41f04efeb\") " pod="openshift-ingress/e2e-api-b-d687ff-openshift-default-5c75fcb6d5-25thf" Oct 31 00:20:34.067449 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:34.067359 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/360cd796-06da-49c1-9bd4-f2f41f04efeb-credential-socket\") pod \"e2e-api-b-d687ff-openshift-default-5c75fcb6d5-25thf\" (UID: \"360cd796-06da-49c1-9bd4-f2f41f04efeb\") " pod="openshift-ingress/e2e-api-b-d687ff-openshift-default-5c75fcb6d5-25thf" Oct 31 00:20:34.067449 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:34.067414 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/360cd796-06da-49c1-9bd4-f2f41f04efeb-istio-data\") pod \"e2e-api-b-d687ff-openshift-default-5c75fcb6d5-25thf\" (UID: \"360cd796-06da-49c1-9bd4-f2f41f04efeb\") " pod="openshift-ingress/e2e-api-b-d687ff-openshift-default-5c75fcb6d5-25thf" Oct 31 00:20:34.067641 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:34.067453 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/360cd796-06da-49c1-9bd4-f2f41f04efeb-istiod-ca-cert\") pod \"e2e-api-b-d687ff-openshift-default-5c75fcb6d5-25thf\" (UID: \"360cd796-06da-49c1-9bd4-f2f41f04efeb\") " pod="openshift-ingress/e2e-api-b-d687ff-openshift-default-5c75fcb6d5-25thf" Oct 31 00:20:34.067641 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:34.067556 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/360cd796-06da-49c1-9bd4-f2f41f04efeb-istio-envoy\") pod \"e2e-api-b-d687ff-openshift-default-5c75fcb6d5-25thf\" (UID: \"360cd796-06da-49c1-9bd4-f2f41f04efeb\") " pod="openshift-ingress/e2e-api-b-d687ff-openshift-default-5c75fcb6d5-25thf" Oct 31 00:20:34.067641 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:34.067606 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/360cd796-06da-49c1-9bd4-f2f41f04efeb-workload-socket\") pod \"e2e-api-b-d687ff-openshift-default-5c75fcb6d5-25thf\" (UID: \"360cd796-06da-49c1-9bd4-f2f41f04efeb\") " pod="openshift-ingress/e2e-api-b-d687ff-openshift-default-5c75fcb6d5-25thf" Oct 31 00:20:34.067805 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:34.067706 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pfz6l\" (UniqueName: \"kubernetes.io/projected/360cd796-06da-49c1-9bd4-f2f41f04efeb-kube-api-access-pfz6l\") pod \"e2e-api-b-d687ff-openshift-default-5c75fcb6d5-25thf\" (UID: \"360cd796-06da-49c1-9bd4-f2f41f04efeb\") " pod="openshift-ingress/e2e-api-b-d687ff-openshift-default-5c75fcb6d5-25thf" Oct 31 00:20:34.067805 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:34.067736 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/360cd796-06da-49c1-9bd4-f2f41f04efeb-istio-token\") pod \"e2e-api-b-d687ff-openshift-default-5c75fcb6d5-25thf\" (UID: \"360cd796-06da-49c1-9bd4-f2f41f04efeb\") " pod="openshift-ingress/e2e-api-b-d687ff-openshift-default-5c75fcb6d5-25thf" Oct 31 00:20:34.078120 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:34.074437 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:20:34.096839 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:34.096748 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-d687ff-openshift-default-5c75fcb6d5-25thf"] Oct 31 00:20:34.216402 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:34.185359 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/360cd796-06da-49c1-9bd4-f2f41f04efeb-istio-data\") pod \"e2e-api-b-d687ff-openshift-default-5c75fcb6d5-25thf\" (UID: \"360cd796-06da-49c1-9bd4-f2f41f04efeb\") " pod="openshift-ingress/e2e-api-b-d687ff-openshift-default-5c75fcb6d5-25thf" Oct 31 00:20:34.216402 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:34.185429 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/360cd796-06da-49c1-9bd4-f2f41f04efeb-istiod-ca-cert\") pod \"e2e-api-b-d687ff-openshift-default-5c75fcb6d5-25thf\" (UID: \"360cd796-06da-49c1-9bd4-f2f41f04efeb\") " pod="openshift-ingress/e2e-api-b-d687ff-openshift-default-5c75fcb6d5-25thf" Oct 31 00:20:34.216402 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:34.185980 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/360cd796-06da-49c1-9bd4-f2f41f04efeb-istio-data\") pod \"e2e-api-b-d687ff-openshift-default-5c75fcb6d5-25thf\" (UID: \"360cd796-06da-49c1-9bd4-f2f41f04efeb\") " pod="openshift-ingress/e2e-api-b-d687ff-openshift-default-5c75fcb6d5-25thf" Oct 31 00:20:34.216402 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:34.186390 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/360cd796-06da-49c1-9bd4-f2f41f04efeb-istio-envoy\") pod \"e2e-api-b-d687ff-openshift-default-5c75fcb6d5-25thf\" (UID: \"360cd796-06da-49c1-9bd4-f2f41f04efeb\") " pod="openshift-ingress/e2e-api-b-d687ff-openshift-default-5c75fcb6d5-25thf" Oct 31 00:20:34.216402 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:34.186448 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/360cd796-06da-49c1-9bd4-f2f41f04efeb-workload-socket\") pod \"e2e-api-b-d687ff-openshift-default-5c75fcb6d5-25thf\" (UID: \"360cd796-06da-49c1-9bd4-f2f41f04efeb\") " pod="openshift-ingress/e2e-api-b-d687ff-openshift-default-5c75fcb6d5-25thf" Oct 31 00:20:34.216402 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:34.186557 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfz6l\" (UniqueName: \"kubernetes.io/projected/360cd796-06da-49c1-9bd4-f2f41f04efeb-kube-api-access-pfz6l\") pod \"e2e-api-b-d687ff-openshift-default-5c75fcb6d5-25thf\" (UID: \"360cd796-06da-49c1-9bd4-f2f41f04efeb\") " pod="openshift-ingress/e2e-api-b-d687ff-openshift-default-5c75fcb6d5-25thf" Oct 31 00:20:34.216402 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:34.186584 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/360cd796-06da-49c1-9bd4-f2f41f04efeb-istio-token\") pod \"e2e-api-b-d687ff-openshift-default-5c75fcb6d5-25thf\" (UID: \"360cd796-06da-49c1-9bd4-f2f41f04efeb\") " pod="openshift-ingress/e2e-api-b-d687ff-openshift-default-5c75fcb6d5-25thf" Oct 31 00:20:34.216402 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:34.186614 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/360cd796-06da-49c1-9bd4-f2f41f04efeb-workload-certs\") pod \"e2e-api-b-d687ff-openshift-default-5c75fcb6d5-25thf\" (UID: \"360cd796-06da-49c1-9bd4-f2f41f04efeb\") " pod="openshift-ingress/e2e-api-b-d687ff-openshift-default-5c75fcb6d5-25thf" Oct 31 00:20:34.216402 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:34.186736 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/360cd796-06da-49c1-9bd4-f2f41f04efeb-istio-podinfo\") pod \"e2e-api-b-d687ff-openshift-default-5c75fcb6d5-25thf\" (UID: \"360cd796-06da-49c1-9bd4-f2f41f04efeb\") " pod="openshift-ingress/e2e-api-b-d687ff-openshift-default-5c75fcb6d5-25thf" Oct 31 00:20:34.216402 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:34.186768 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/360cd796-06da-49c1-9bd4-f2f41f04efeb-credential-socket\") pod \"e2e-api-b-d687ff-openshift-default-5c75fcb6d5-25thf\" (UID: \"360cd796-06da-49c1-9bd4-f2f41f04efeb\") " pod="openshift-ingress/e2e-api-b-d687ff-openshift-default-5c75fcb6d5-25thf" Oct 31 00:20:34.216402 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:34.187174 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/360cd796-06da-49c1-9bd4-f2f41f04efeb-credential-socket\") pod \"e2e-api-b-d687ff-openshift-default-5c75fcb6d5-25thf\" (UID: \"360cd796-06da-49c1-9bd4-f2f41f04efeb\") " pod="openshift-ingress/e2e-api-b-d687ff-openshift-default-5c75fcb6d5-25thf" Oct 31 00:20:34.216402 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:34.187828 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/360cd796-06da-49c1-9bd4-f2f41f04efeb-istiod-ca-cert\") pod \"e2e-api-b-d687ff-openshift-default-5c75fcb6d5-25thf\" (UID: \"360cd796-06da-49c1-9bd4-f2f41f04efeb\") " pod="openshift-ingress/e2e-api-b-d687ff-openshift-default-5c75fcb6d5-25thf" Oct 31 00:20:34.216402 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:34.188998 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/360cd796-06da-49c1-9bd4-f2f41f04efeb-workload-socket\") pod \"e2e-api-b-d687ff-openshift-default-5c75fcb6d5-25thf\" (UID: \"360cd796-06da-49c1-9bd4-f2f41f04efeb\") " pod="openshift-ingress/e2e-api-b-d687ff-openshift-default-5c75fcb6d5-25thf" Oct 31 00:20:34.216402 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:34.189652 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/360cd796-06da-49c1-9bd4-f2f41f04efeb-workload-certs\") pod \"e2e-api-b-d687ff-openshift-default-5c75fcb6d5-25thf\" (UID: \"360cd796-06da-49c1-9bd4-f2f41f04efeb\") " pod="openshift-ingress/e2e-api-b-d687ff-openshift-default-5c75fcb6d5-25thf" Oct 31 00:20:34.256771 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:34.254650 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/360cd796-06da-49c1-9bd4-f2f41f04efeb-istio-envoy\") pod \"e2e-api-b-d687ff-openshift-default-5c75fcb6d5-25thf\" (UID: \"360cd796-06da-49c1-9bd4-f2f41f04efeb\") " pod="openshift-ingress/e2e-api-b-d687ff-openshift-default-5c75fcb6d5-25thf" Oct 31 00:20:34.261675 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:34.260513 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/360cd796-06da-49c1-9bd4-f2f41f04efeb-istio-podinfo\") pod \"e2e-api-b-d687ff-openshift-default-5c75fcb6d5-25thf\" (UID: \"360cd796-06da-49c1-9bd4-f2f41f04efeb\") " pod="openshift-ingress/e2e-api-b-d687ff-openshift-default-5c75fcb6d5-25thf" Oct 31 00:20:34.266043 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:34.265271 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pfz6l\" (UniqueName: \"kubernetes.io/projected/360cd796-06da-49c1-9bd4-f2f41f04efeb-kube-api-access-pfz6l\") pod \"e2e-api-b-d687ff-openshift-default-5c75fcb6d5-25thf\" (UID: \"360cd796-06da-49c1-9bd4-f2f41f04efeb\") " pod="openshift-ingress/e2e-api-b-d687ff-openshift-default-5c75fcb6d5-25thf" Oct 31 00:20:34.267004 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:34.266976 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/360cd796-06da-49c1-9bd4-f2f41f04efeb-istio-token\") pod \"e2e-api-b-d687ff-openshift-default-5c75fcb6d5-25thf\" (UID: \"360cd796-06da-49c1-9bd4-f2f41f04efeb\") " pod="openshift-ingress/e2e-api-b-d687ff-openshift-default-5c75fcb6d5-25thf" Oct 31 00:20:34.295766 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:34.295733 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-d687ff-openshift-default-5c75fcb6d5-25thf" Oct 31 00:20:34.462193 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:34.462153 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:20:34.755698 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:34.755668 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-d687ff-openshift-default-5c75fcb6d5-25thf"] Oct 31 00:20:34.756827 ip-10-0-141-180 kubenswrapper[4927]: W1031 00:20:34.756794 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod360cd796_06da_49c1_9bd4_f2f41f04efeb.slice/crio-907068bcecaf2ef9dff3d90652492e2d0faf0e0877159994588f132d21d9099a WatchSource:0}: Error finding container 907068bcecaf2ef9dff3d90652492e2d0faf0e0877159994588f132d21d9099a: Status 404 returned error can't find the container with id 907068bcecaf2ef9dff3d90652492e2d0faf0e0877159994588f132d21d9099a Oct 31 00:20:34.764714 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:34.764671 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:20:34.764812 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:34.764764 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:20:34.764812 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:34.764804 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:20:34.828327 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:34.828297 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:20:35.024983 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:35.024752 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:20:35.510347 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:35.510290 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-d687ff-openshift-default-5c75fcb6d5-25thf" event={"ID":"360cd796-06da-49c1-9bd4-f2f41f04efeb","Type":"ContainerStarted","Data":"402a284d6077e106734bc3fd7369626a1d28550adc4df5e45092d3862efc2cf8"} Oct 31 00:20:35.510527 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:35.510384 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-d687ff-openshift-default-5c75fcb6d5-25thf" event={"ID":"360cd796-06da-49c1-9bd4-f2f41f04efeb","Type":"ContainerStarted","Data":"907068bcecaf2ef9dff3d90652492e2d0faf0e0877159994588f132d21d9099a"} Oct 31 00:20:35.689655 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:35.689596 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-d687ff-openshift-default-5c75fcb6d5-25thf" podStartSLOduration=2.68957977 podStartE2EDuration="2.68957977s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:20:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:20:35.635132427 +0000 UTC m=+2348.671463279" watchObservedRunningTime="2026-10-31 00:20:35.68957977 +0000 UTC m=+2348.725910592" Oct 31 00:20:35.853778 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:35.853683 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:20:36.296075 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:36.296040 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-d687ff-openshift-default-5c75fcb6d5-25thf" Oct 31 00:20:36.302056 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:36.302022 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-d687ff-openshift-default-5c75fcb6d5-25thf" Oct 31 00:20:36.521239 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:36.521199 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-d687ff-openshift-default-5c75fcb6d5-25thf" Oct 31 00:20:36.522941 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:36.522913 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-d687ff-openshift-default-5c75fcb6d5-25thf" Oct 31 00:20:38.278246 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:38.278207 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:20:38.489896 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:38.489862 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:20:38.541110 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:38.541021 4927 generic.go:356] "Generic (PLEG): container finished" podID="31803f91-4f1e-407f-b82e-386fd8a80be8" containerID="dac22421d14078433923c10beb17d05dee975d3befd8509fa2341aa6c6bfe2ae" exitCode=0 Oct 31 00:20:38.541289 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:38.541098 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ae3cd0-openshift-default-5dbcd797d5-hvcv4" event={"ID":"31803f91-4f1e-407f-b82e-386fd8a80be8","Type":"ContainerDied","Data":"dac22421d14078433923c10beb17d05dee975d3befd8509fa2341aa6c6bfe2ae"} Oct 31 00:20:38.541653 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:38.541628 4927 scope.go:122] "RemoveContainer" containerID="dac22421d14078433923c10beb17d05dee975d3befd8509fa2341aa6c6bfe2ae" Oct 31 00:20:38.543660 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:38.543623 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:20:38.543780 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:38.543712 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:20:38.543780 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:38.543758 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:20:38.672996 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:38.672963 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:20:38.892166 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:38.892084 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:20:39.557692 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:39.557653 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ae3cd0-openshift-default-5dbcd797d5-hvcv4" event={"ID":"31803f91-4f1e-407f-b82e-386fd8a80be8","Type":"ContainerStarted","Data":"5c6b29b49ef7ac09a23b9a5c1c6e55666fcfac059a961b651d90a5f50945e119"} Oct 31 00:20:39.994194 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:39.994159 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-ae3cd0-openshift-default-5dbcd797d5-hvcv4" Oct 31 00:20:39.995073 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:39.995049 4927 patch_prober.go:29] interesting pod/e2e-api-a-ae3cd0-openshift-default-5dbcd797d5-hvcv4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused" start-of-body= Oct 31 00:20:39.995147 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:39.995100 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ae3cd0-openshift-default-5dbcd797d5-hvcv4" podUID="31803f91-4f1e-407f-b82e-386fd8a80be8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused" Oct 31 00:20:40.953754 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:40.953717 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:20:41.002264 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:41.002219 4927 patch_prober.go:29] interesting pod/e2e-api-a-ae3cd0-openshift-default-5dbcd797d5-hvcv4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused" start-of-body= Oct 31 00:20:41.002766 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:41.002721 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ae3cd0-openshift-default-5dbcd797d5-hvcv4" podUID="31803f91-4f1e-407f-b82e-386fd8a80be8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused" Oct 31 00:20:41.849740 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:41.849703 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:20:42.002784 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:42.002729 4927 patch_prober.go:29] interesting pod/e2e-api-a-ae3cd0-openshift-default-5dbcd797d5-hvcv4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused" start-of-body= Oct 31 00:20:42.015771 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:42.003097 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:20:42.045443 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:42.024373 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ae3cd0-openshift-default-5dbcd797d5-hvcv4" podUID="31803f91-4f1e-407f-b82e-386fd8a80be8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused" Oct 31 00:20:42.048322 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:42.024408 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-ae3cd0-openshift-default-5dbcd797d5-hvcv4" Oct 31 00:20:42.994634 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:42.994595 4927 patch_prober.go:29] interesting pod/e2e-api-a-ae3cd0-openshift-default-5dbcd797d5-hvcv4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused" start-of-body= Oct 31 00:20:42.994802 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:42.994659 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ae3cd0-openshift-default-5dbcd797d5-hvcv4" podUID="31803f91-4f1e-407f-b82e-386fd8a80be8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused" Oct 31 00:20:43.994822 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:43.994783 4927 patch_prober.go:29] interesting pod/e2e-api-a-ae3cd0-openshift-default-5dbcd797d5-hvcv4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused" start-of-body= Oct 31 00:20:43.995370 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:43.994848 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ae3cd0-openshift-default-5dbcd797d5-hvcv4" podUID="31803f91-4f1e-407f-b82e-386fd8a80be8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused" Oct 31 00:20:44.994412 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:44.994372 4927 patch_prober.go:29] interesting pod/e2e-api-a-ae3cd0-openshift-default-5dbcd797d5-hvcv4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused" start-of-body= Oct 31 00:20:44.994641 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:44.994448 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ae3cd0-openshift-default-5dbcd797d5-hvcv4" podUID="31803f91-4f1e-407f-b82e-386fd8a80be8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused" Oct 31 00:20:45.744028 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:45.743991 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:20:45.902219 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:45.902183 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:20:45.999751 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:45.999664 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-ae3cd0-openshift-default-5dbcd797d5-hvcv4" Oct 31 00:20:46.000959 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:46.000933 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-ae3cd0-openshift-default-5dbcd797d5-hvcv4" Oct 31 00:20:46.507858 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:46.507819 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:20:47.580439 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:47.580404 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:20:49.680263 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:20:49.680220 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sqv69" containerName="extractor" Oct 31 00:20:50.946304 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:50.946268 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:20:52.534510 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:52.534474 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-fe01fa/distinct-2-w5-fe01fa-kserve-79b99966d8-dkhzd"] Oct 31 00:20:52.695289 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:52.695251 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:20:53.109042 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:53.108988 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-0d75be-7b66b84856-j28w4"] Oct 31 00:20:53.205527 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:53.205493 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-0d75be-75f9855f5-kftq4"] Oct 31 00:20:53.462575 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:53.462539 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:20:53.663810 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:53.663776 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:20:54.550035 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:54.549999 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-jpjkl"] Oct 31 00:20:54.551360 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:54.551319 4927 cpu_manager.go:414] "containerMap: removing container" podUID="7a19c11a-45ef-434e-9667-e75e17898834" containerName="istio-proxy" Oct 31 00:20:54.551515 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:54.551502 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a19c11a-45ef-434e-9667-e75e17898834" containerName="istio-proxy" Oct 31 00:20:54.597745 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:54.597705 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-jpjkl" Oct 31 00:20:54.644819 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:54.644792 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-mz295"] Oct 31 00:20:54.657890 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:54.657865 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-mz295" Oct 31 00:20:54.667678 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:54.667646 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/334942ae-4532-436c-b274-532b56c035ec-istio-token\") pod \"maas-default-gateway-openshift-default-8f6f87f59-jpjkl\" (UID: \"334942ae-4532-436c-b274-532b56c035ec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-jpjkl" Oct 31 00:20:54.668228 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:54.668204 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/334942ae-4532-436c-b274-532b56c035ec-workload-socket\") pod \"maas-default-gateway-openshift-default-8f6f87f59-jpjkl\" (UID: \"334942ae-4532-436c-b274-532b56c035ec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-jpjkl" Oct 31 00:20:54.668520 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:54.668491 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xc6h\" (UniqueName: \"kubernetes.io/projected/334942ae-4532-436c-b274-532b56c035ec-kube-api-access-5xc6h\") pod \"maas-default-gateway-openshift-default-8f6f87f59-jpjkl\" (UID: \"334942ae-4532-436c-b274-532b56c035ec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-jpjkl" Oct 31 00:20:54.668806 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:54.668789 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/334942ae-4532-436c-b274-532b56c035ec-istio-data\") pod \"maas-default-gateway-openshift-default-8f6f87f59-jpjkl\" (UID: \"334942ae-4532-436c-b274-532b56c035ec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-jpjkl" Oct 31 00:20:54.668994 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:54.668965 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/334942ae-4532-436c-b274-532b56c035ec-workload-certs\") pod \"maas-default-gateway-openshift-default-8f6f87f59-jpjkl\" (UID: \"334942ae-4532-436c-b274-532b56c035ec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-jpjkl" Oct 31 00:20:54.669178 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:54.669161 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/334942ae-4532-436c-b274-532b56c035ec-credential-socket\") pod \"maas-default-gateway-openshift-default-8f6f87f59-jpjkl\" (UID: \"334942ae-4532-436c-b274-532b56c035ec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-jpjkl" Oct 31 00:20:54.669388 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:54.669372 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/334942ae-4532-436c-b274-532b56c035ec-istio-envoy\") pod \"maas-default-gateway-openshift-default-8f6f87f59-jpjkl\" (UID: \"334942ae-4532-436c-b274-532b56c035ec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-jpjkl" Oct 31 00:20:54.669552 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:54.669531 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/334942ae-4532-436c-b274-532b56c035ec-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8f6f87f59-jpjkl\" (UID: \"334942ae-4532-436c-b274-532b56c035ec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-jpjkl" Oct 31 00:20:54.669682 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:54.669669 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/334942ae-4532-436c-b274-532b56c035ec-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8f6f87f59-jpjkl\" (UID: \"334942ae-4532-436c-b274-532b56c035ec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-jpjkl" Oct 31 00:20:54.714774 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:54.712250 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-jpjkl"] Oct 31 00:20:54.721095 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:54.721065 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-mz295"] Oct 31 00:20:54.770715 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:54.770680 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c76ae81c-b5d0-4c9b-af8f-0a95d2aabf9f-workload-socket\") pod \"maas-default-gateway-openshift-default-8f6f87f59-mz295\" (UID: \"c76ae81c-b5d0-4c9b-af8f-0a95d2aabf9f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-mz295" Oct 31 00:20:54.770715 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:54.770720 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c76ae81c-b5d0-4c9b-af8f-0a95d2aabf9f-istio-token\") pod \"maas-default-gateway-openshift-default-8f6f87f59-mz295\" (UID: \"c76ae81c-b5d0-4c9b-af8f-0a95d2aabf9f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-mz295" Oct 31 00:20:54.770976 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:54.770743 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/334942ae-4532-436c-b274-532b56c035ec-credential-socket\") pod \"maas-default-gateway-openshift-default-8f6f87f59-jpjkl\" (UID: \"334942ae-4532-436c-b274-532b56c035ec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-jpjkl" Oct 31 00:20:54.770976 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:54.770932 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c76ae81c-b5d0-4c9b-af8f-0a95d2aabf9f-workload-certs\") pod \"maas-default-gateway-openshift-default-8f6f87f59-mz295\" (UID: \"c76ae81c-b5d0-4c9b-af8f-0a95d2aabf9f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-mz295" Oct 31 00:20:54.770976 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:54.770976 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/334942ae-4532-436c-b274-532b56c035ec-istio-envoy\") pod \"maas-default-gateway-openshift-default-8f6f87f59-jpjkl\" (UID: \"334942ae-4532-436c-b274-532b56c035ec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-jpjkl" Oct 31 00:20:54.771160 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:54.771009 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/334942ae-4532-436c-b274-532b56c035ec-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8f6f87f59-jpjkl\" (UID: \"334942ae-4532-436c-b274-532b56c035ec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-jpjkl" Oct 31 00:20:54.771160 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:54.771029 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/334942ae-4532-436c-b274-532b56c035ec-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8f6f87f59-jpjkl\" (UID: \"334942ae-4532-436c-b274-532b56c035ec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-jpjkl" Oct 31 00:20:54.771160 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:54.771090 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/334942ae-4532-436c-b274-532b56c035ec-istio-token\") pod \"maas-default-gateway-openshift-default-8f6f87f59-jpjkl\" (UID: \"334942ae-4532-436c-b274-532b56c035ec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-jpjkl" Oct 31 00:20:54.771160 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:54.771127 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/334942ae-4532-436c-b274-532b56c035ec-workload-socket\") pod \"maas-default-gateway-openshift-default-8f6f87f59-jpjkl\" (UID: \"334942ae-4532-436c-b274-532b56c035ec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-jpjkl" Oct 31 00:20:54.771160 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:54.771152 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c76ae81c-b5d0-4c9b-af8f-0a95d2aabf9f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8f6f87f59-mz295\" (UID: \"c76ae81c-b5d0-4c9b-af8f-0a95d2aabf9f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-mz295" Oct 31 00:20:54.771476 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:54.771188 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c76ae81c-b5d0-4c9b-af8f-0a95d2aabf9f-credential-socket\") pod \"maas-default-gateway-openshift-default-8f6f87f59-mz295\" (UID: \"c76ae81c-b5d0-4c9b-af8f-0a95d2aabf9f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-mz295" Oct 31 00:20:54.771476 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:54.771188 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/334942ae-4532-436c-b274-532b56c035ec-credential-socket\") pod \"maas-default-gateway-openshift-default-8f6f87f59-jpjkl\" (UID: \"334942ae-4532-436c-b274-532b56c035ec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-jpjkl" Oct 31 00:20:54.771476 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:54.771304 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xc6h\" (UniqueName: \"kubernetes.io/projected/334942ae-4532-436c-b274-532b56c035ec-kube-api-access-5xc6h\") pod \"maas-default-gateway-openshift-default-8f6f87f59-jpjkl\" (UID: \"334942ae-4532-436c-b274-532b56c035ec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-jpjkl" Oct 31 00:20:54.771476 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:54.771381 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c76ae81c-b5d0-4c9b-af8f-0a95d2aabf9f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8f6f87f59-mz295\" (UID: \"c76ae81c-b5d0-4c9b-af8f-0a95d2aabf9f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-mz295" Oct 31 00:20:54.771476 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:54.771404 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6wgbs\" (UniqueName: \"kubernetes.io/projected/c76ae81c-b5d0-4c9b-af8f-0a95d2aabf9f-kube-api-access-6wgbs\") pod \"maas-default-gateway-openshift-default-8f6f87f59-mz295\" (UID: \"c76ae81c-b5d0-4c9b-af8f-0a95d2aabf9f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-mz295" Oct 31 00:20:54.771476 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:54.771436 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c76ae81c-b5d0-4c9b-af8f-0a95d2aabf9f-istio-data\") pod \"maas-default-gateway-openshift-default-8f6f87f59-mz295\" (UID: \"c76ae81c-b5d0-4c9b-af8f-0a95d2aabf9f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-mz295" Oct 31 00:20:54.771476 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:54.771456 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c76ae81c-b5d0-4c9b-af8f-0a95d2aabf9f-istio-envoy\") pod \"maas-default-gateway-openshift-default-8f6f87f59-mz295\" (UID: \"c76ae81c-b5d0-4c9b-af8f-0a95d2aabf9f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-mz295" Oct 31 00:20:54.771845 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:54.771484 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/334942ae-4532-436c-b274-532b56c035ec-istio-data\") pod \"maas-default-gateway-openshift-default-8f6f87f59-jpjkl\" (UID: \"334942ae-4532-436c-b274-532b56c035ec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-jpjkl" Oct 31 00:20:54.771845 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:54.771561 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/334942ae-4532-436c-b274-532b56c035ec-workload-certs\") pod \"maas-default-gateway-openshift-default-8f6f87f59-jpjkl\" (UID: \"334942ae-4532-436c-b274-532b56c035ec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-jpjkl" Oct 31 00:20:54.771845 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:54.771797 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/334942ae-4532-436c-b274-532b56c035ec-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8f6f87f59-jpjkl\" (UID: \"334942ae-4532-436c-b274-532b56c035ec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-jpjkl" Oct 31 00:20:54.772043 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:54.771933 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/334942ae-4532-436c-b274-532b56c035ec-workload-certs\") pod \"maas-default-gateway-openshift-default-8f6f87f59-jpjkl\" (UID: \"334942ae-4532-436c-b274-532b56c035ec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-jpjkl" Oct 31 00:20:54.772043 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:54.771487 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/334942ae-4532-436c-b274-532b56c035ec-workload-socket\") pod \"maas-default-gateway-openshift-default-8f6f87f59-jpjkl\" (UID: \"334942ae-4532-436c-b274-532b56c035ec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-jpjkl" Oct 31 00:20:54.772205 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:54.772184 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/334942ae-4532-436c-b274-532b56c035ec-istio-data\") pod \"maas-default-gateway-openshift-default-8f6f87f59-jpjkl\" (UID: \"334942ae-4532-436c-b274-532b56c035ec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-jpjkl" Oct 31 00:20:54.773678 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:54.773644 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/334942ae-4532-436c-b274-532b56c035ec-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8f6f87f59-jpjkl\" (UID: \"334942ae-4532-436c-b274-532b56c035ec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-jpjkl" Oct 31 00:20:54.774399 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:54.774377 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/334942ae-4532-436c-b274-532b56c035ec-istio-envoy\") pod \"maas-default-gateway-openshift-default-8f6f87f59-jpjkl\" (UID: \"334942ae-4532-436c-b274-532b56c035ec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-jpjkl" Oct 31 00:20:54.871610 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:54.871530 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xc6h\" (UniqueName: \"kubernetes.io/projected/334942ae-4532-436c-b274-532b56c035ec-kube-api-access-5xc6h\") pod \"maas-default-gateway-openshift-default-8f6f87f59-jpjkl\" (UID: \"334942ae-4532-436c-b274-532b56c035ec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-jpjkl" Oct 31 00:20:54.873123 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:54.873093 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c76ae81c-b5d0-4c9b-af8f-0a95d2aabf9f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8f6f87f59-mz295\" (UID: \"c76ae81c-b5d0-4c9b-af8f-0a95d2aabf9f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-mz295" Oct 31 00:20:54.874025 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:54.874005 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c76ae81c-b5d0-4c9b-af8f-0a95d2aabf9f-credential-socket\") pod \"maas-default-gateway-openshift-default-8f6f87f59-mz295\" (UID: \"c76ae81c-b5d0-4c9b-af8f-0a95d2aabf9f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-mz295" Oct 31 00:20:54.874251 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:54.874234 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c76ae81c-b5d0-4c9b-af8f-0a95d2aabf9f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8f6f87f59-mz295\" (UID: \"c76ae81c-b5d0-4c9b-af8f-0a95d2aabf9f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-mz295" Oct 31 00:20:54.874393 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:54.874374 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wgbs\" (UniqueName: \"kubernetes.io/projected/c76ae81c-b5d0-4c9b-af8f-0a95d2aabf9f-kube-api-access-6wgbs\") pod \"maas-default-gateway-openshift-default-8f6f87f59-mz295\" (UID: \"c76ae81c-b5d0-4c9b-af8f-0a95d2aabf9f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-mz295" Oct 31 00:20:54.874532 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:54.874518 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c76ae81c-b5d0-4c9b-af8f-0a95d2aabf9f-istio-data\") pod \"maas-default-gateway-openshift-default-8f6f87f59-mz295\" (UID: \"c76ae81c-b5d0-4c9b-af8f-0a95d2aabf9f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-mz295" Oct 31 00:20:54.874632 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:54.874620 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c76ae81c-b5d0-4c9b-af8f-0a95d2aabf9f-istio-envoy\") pod \"maas-default-gateway-openshift-default-8f6f87f59-mz295\" (UID: \"c76ae81c-b5d0-4c9b-af8f-0a95d2aabf9f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-mz295" Oct 31 00:20:54.874784 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:54.874772 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c76ae81c-b5d0-4c9b-af8f-0a95d2aabf9f-workload-socket\") pod \"maas-default-gateway-openshift-default-8f6f87f59-mz295\" (UID: \"c76ae81c-b5d0-4c9b-af8f-0a95d2aabf9f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-mz295" Oct 31 00:20:54.874883 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:54.874870 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c76ae81c-b5d0-4c9b-af8f-0a95d2aabf9f-istio-token\") pod \"maas-default-gateway-openshift-default-8f6f87f59-mz295\" (UID: \"c76ae81c-b5d0-4c9b-af8f-0a95d2aabf9f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-mz295" Oct 31 00:20:54.875009 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:54.874999 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c76ae81c-b5d0-4c9b-af8f-0a95d2aabf9f-workload-certs\") pod \"maas-default-gateway-openshift-default-8f6f87f59-mz295\" (UID: \"c76ae81c-b5d0-4c9b-af8f-0a95d2aabf9f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-mz295" Oct 31 00:20:54.875455 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:54.875435 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c76ae81c-b5d0-4c9b-af8f-0a95d2aabf9f-workload-certs\") pod \"maas-default-gateway-openshift-default-8f6f87f59-mz295\" (UID: \"c76ae81c-b5d0-4c9b-af8f-0a95d2aabf9f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-mz295" Oct 31 00:20:54.875559 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:54.873954 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c76ae81c-b5d0-4c9b-af8f-0a95d2aabf9f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8f6f87f59-mz295\" (UID: \"c76ae81c-b5d0-4c9b-af8f-0a95d2aabf9f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-mz295" Oct 31 00:20:54.875979 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:54.875887 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c76ae81c-b5d0-4c9b-af8f-0a95d2aabf9f-istio-data\") pod \"maas-default-gateway-openshift-default-8f6f87f59-mz295\" (UID: \"c76ae81c-b5d0-4c9b-af8f-0a95d2aabf9f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-mz295" Oct 31 00:20:54.876082 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:54.875902 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c76ae81c-b5d0-4c9b-af8f-0a95d2aabf9f-workload-socket\") pod \"maas-default-gateway-openshift-default-8f6f87f59-mz295\" (UID: \"c76ae81c-b5d0-4c9b-af8f-0a95d2aabf9f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-mz295" Oct 31 00:20:54.876163 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:54.876115 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c76ae81c-b5d0-4c9b-af8f-0a95d2aabf9f-credential-socket\") pod \"maas-default-gateway-openshift-default-8f6f87f59-mz295\" (UID: \"c76ae81c-b5d0-4c9b-af8f-0a95d2aabf9f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-mz295" Oct 31 00:20:54.884952 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:54.884924 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c76ae81c-b5d0-4c9b-af8f-0a95d2aabf9f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8f6f87f59-mz295\" (UID: \"c76ae81c-b5d0-4c9b-af8f-0a95d2aabf9f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-mz295" Oct 31 00:20:54.885495 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:54.885453 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c76ae81c-b5d0-4c9b-af8f-0a95d2aabf9f-istio-envoy\") pod \"maas-default-gateway-openshift-default-8f6f87f59-mz295\" (UID: \"c76ae81c-b5d0-4c9b-af8f-0a95d2aabf9f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-mz295" Oct 31 00:20:54.885946 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:54.885851 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/334942ae-4532-436c-b274-532b56c035ec-istio-token\") pod \"maas-default-gateway-openshift-default-8f6f87f59-jpjkl\" (UID: \"334942ae-4532-436c-b274-532b56c035ec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-jpjkl" Oct 31 00:20:54.963720 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:54.963683 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c76ae81c-b5d0-4c9b-af8f-0a95d2aabf9f-istio-token\") pod \"maas-default-gateway-openshift-default-8f6f87f59-mz295\" (UID: \"c76ae81c-b5d0-4c9b-af8f-0a95d2aabf9f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-mz295" Oct 31 00:20:54.965108 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:54.965074 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6wgbs\" (UniqueName: \"kubernetes.io/projected/c76ae81c-b5d0-4c9b-af8f-0a95d2aabf9f-kube-api-access-6wgbs\") pod \"maas-default-gateway-openshift-default-8f6f87f59-mz295\" (UID: \"c76ae81c-b5d0-4c9b-af8f-0a95d2aabf9f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-mz295" Oct 31 00:20:55.011739 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:55.011699 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-jpjkl" Oct 31 00:20:55.020758 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:55.020730 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-mz295" Oct 31 00:20:55.479362 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:55.479306 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-jpjkl"] Oct 31 00:20:55.481980 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:55.481944 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:20:55.482079 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:55.482012 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:20:55.482079 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:55.482040 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:20:55.483446 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:55.483423 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-mz295"] Oct 31 00:20:55.519777 ip-10-0-141-180 kubenswrapper[4927]: W1031 00:20:55.519749 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc76ae81c_b5d0_4c9b_af8f_0a95d2aabf9f.slice/crio-0e947dc1eb51d43213a3cb3682f5637c72a42f2f6b2ab978b8edc90aa3ea0071 WatchSource:0}: Error finding container 0e947dc1eb51d43213a3cb3682f5637c72a42f2f6b2ab978b8edc90aa3ea0071: Status 404 returned error can't find the container with id 0e947dc1eb51d43213a3cb3682f5637c72a42f2f6b2ab978b8edc90aa3ea0071 Oct 31 00:20:55.522731 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:55.522697 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:20:55.522837 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:55.522778 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:20:55.522837 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:55.522828 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:20:55.722738 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:55.722701 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-mz295" event={"ID":"c76ae81c-b5d0-4c9b-af8f-0a95d2aabf9f","Type":"ContainerStarted","Data":"f2735651c5eb3c78efea40369c5e52ff8edef4177ea7da229ac47810cbac400a"} Oct 31 00:20:55.723225 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:55.722764 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-mz295" event={"ID":"c76ae81c-b5d0-4c9b-af8f-0a95d2aabf9f","Type":"ContainerStarted","Data":"0e947dc1eb51d43213a3cb3682f5637c72a42f2f6b2ab978b8edc90aa3ea0071"} Oct 31 00:20:55.724704 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:55.724672 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-jpjkl" event={"ID":"334942ae-4532-436c-b274-532b56c035ec","Type":"ContainerStarted","Data":"f9740f71a450a8d486b1459a6121a8eb91adab0a8dd81fe391c0f9d113468e7b"} Oct 31 00:20:55.724832 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:55.724727 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-jpjkl" event={"ID":"334942ae-4532-436c-b274-532b56c035ec","Type":"ContainerStarted","Data":"4b33d1c39456bfabc4b0c5ed6e18a53ea62419d8ae6a6cb82175a243e46aacac"} Oct 31 00:20:55.851634 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:55.851504 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-mz295" podStartSLOduration=1.851485926 podStartE2EDuration="1.851485926s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:20:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:20:55.846396403 +0000 UTC m=+2368.882727222" watchObservedRunningTime="2026-10-31 00:20:55.851485926 +0000 UTC m=+2368.887816747" Oct 31 00:20:55.978745 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:55.978682 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-jpjkl" podStartSLOduration=1.9786632800000001 podStartE2EDuration="1.97866328s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:20:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:20:55.976737875 +0000 UTC m=+2369.013068732" watchObservedRunningTime="2026-10-31 00:20:55.97866328 +0000 UTC m=+2369.014994103" Oct 31 00:20:56.012422 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:56.012384 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-jpjkl" Oct 31 00:20:56.014360 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:56.014313 4927 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-jpjkl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.244:15021/healthz/ready\": dial tcp 10.132.0.244:15021: connect: connection refused" start-of-body= Oct 31 00:20:56.014515 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:56.014388 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-jpjkl" podUID="334942ae-4532-436c-b274-532b56c035ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.244:15021/healthz/ready\": dial tcp 10.132.0.244:15021: connect: connection refused" Oct 31 00:20:56.021507 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:56.021480 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-mz295" Oct 31 00:20:56.023729 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:56.023693 4927 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-mz295 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.245:15021/healthz/ready\": dial tcp 10.132.0.245:15021: connect: connection refused" start-of-body= Oct 31 00:20:56.023870 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:56.023754 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-mz295" podUID="c76ae81c-b5d0-4c9b-af8f-0a95d2aabf9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.245:15021/healthz/ready\": dial tcp 10.132.0.245:15021: connect: connection refused" Oct 31 00:20:56.098585 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:56.096795 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:20:57.015367 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:57.014800 4927 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-jpjkl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.244:15021/healthz/ready\": dial tcp 10.132.0.244:15021: connect: connection refused" start-of-body= Oct 31 00:20:57.015367 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:57.014869 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-jpjkl" podUID="334942ae-4532-436c-b274-532b56c035ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.244:15021/healthz/ready\": dial tcp 10.132.0.244:15021: connect: connection refused" Oct 31 00:20:57.035355 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:57.033803 4927 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-mz295 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.245:15021/healthz/ready\": dial tcp 10.132.0.245:15021: connect: connection refused" start-of-body= Oct 31 00:20:57.035355 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:57.033871 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-mz295" podUID="c76ae81c-b5d0-4c9b-af8f-0a95d2aabf9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.245:15021/healthz/ready\": dial tcp 10.132.0.245:15021: connect: connection refused" Oct 31 00:20:58.023358 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:58.016832 4927 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-jpjkl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.244:15021/healthz/ready\": dial tcp 10.132.0.244:15021: connect: connection refused" start-of-body= Oct 31 00:20:58.023358 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:58.016905 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-jpjkl" podUID="334942ae-4532-436c-b274-532b56c035ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.244:15021/healthz/ready\": dial tcp 10.132.0.244:15021: connect: connection refused" Oct 31 00:20:58.023358 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:58.021683 4927 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-mz295 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.245:15021/healthz/ready\": dial tcp 10.132.0.245:15021: connect: connection refused" start-of-body= Oct 31 00:20:58.023358 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:58.021735 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-mz295" podUID="c76ae81c-b5d0-4c9b-af8f-0a95d2aabf9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.245:15021/healthz/ready\": dial tcp 10.132.0.245:15021: connect: connection refused" Oct 31 00:20:58.113356 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:58.111389 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-0d75be-7b66b84856-j28w4" podUID="9fd4cd19-a1e6-455a-b36a-c9f58692b081" containerName="payload-processing" containerID="cri-o://da5ec8cc378a8c46a065d5f4772e22cf96a174e9c8b7517d5a74777de2b4de94" gracePeriod=25 Oct 31 00:20:58.205813 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:58.205769 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-0d75be-75f9855f5-kftq4" podUID="0b774910-1e71-40dd-9d93-026df8b5f9d2" containerName="payload-pre-processing" containerID="cri-o://fccf0c779f70085c9950db7547d74987e208d9579ea0ba23f9ad709ba3e8ae09" gracePeriod=25 Oct 31 00:20:58.561410 ip-10-0-141-180 kubenswrapper[4927]: W1031 00:20:58.560479 4927 logging.go:55] [core] [Channel #9053 SubChannel #9054]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.187:50052", ServerName: "10.132.0.187:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.187:50052: connect: connection refused" Oct 31 00:20:58.696481 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:58.696452 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-0d75be-75f9855f5-kftq4" Oct 31 00:20:58.785771 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:58.784533 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b774910-1e71-40dd-9d93-026df8b5f9d2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b774910-1e71-40dd-9d93-026df8b5f9d2-plugins-config-volume\") pod \"0b774910-1e71-40dd-9d93-026df8b5f9d2\" (UID: \"0b774910-1e71-40dd-9d93-026df8b5f9d2\") " Oct 31 00:20:58.785771 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:58.784598 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b774910-1e71-40dd-9d93-026df8b5f9d2-kube-api-access-r5h5r\" (UniqueName: \"kubernetes.io/projected/0b774910-1e71-40dd-9d93-026df8b5f9d2-kube-api-access-r5h5r\") pod \"0b774910-1e71-40dd-9d93-026df8b5f9d2\" (UID: \"0b774910-1e71-40dd-9d93-026df8b5f9d2\") " Oct 31 00:20:58.785771 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:58.785742 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b774910-1e71-40dd-9d93-026df8b5f9d2-plugins-config-volume" pod "0b774910-1e71-40dd-9d93-026df8b5f9d2" (UID: "0b774910-1e71-40dd-9d93-026df8b5f9d2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:20:58.788043 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:58.787993 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b774910-1e71-40dd-9d93-026df8b5f9d2-kube-api-access-r5h5r" pod "0b774910-1e71-40dd-9d93-026df8b5f9d2" (UID: "0b774910-1e71-40dd-9d93-026df8b5f9d2"). InnerVolumeSpecName "kube-api-access-r5h5r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:20:58.790771 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:58.790738 4927 generic.go:356] "Generic (PLEG): container finished" podID="0b774910-1e71-40dd-9d93-026df8b5f9d2" containerID="fccf0c779f70085c9950db7547d74987e208d9579ea0ba23f9ad709ba3e8ae09" exitCode=0 Oct 31 00:20:58.790908 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:58.790857 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-0d75be-75f9855f5-kftq4" Oct 31 00:20:58.791007 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:58.790981 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-0d75be-75f9855f5-kftq4" event={"ID":"0b774910-1e71-40dd-9d93-026df8b5f9d2","Type":"ContainerDied","Data":"fccf0c779f70085c9950db7547d74987e208d9579ea0ba23f9ad709ba3e8ae09"} Oct 31 00:20:58.791070 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:58.791044 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-0d75be-75f9855f5-kftq4" event={"ID":"0b774910-1e71-40dd-9d93-026df8b5f9d2","Type":"ContainerDied","Data":"4d6b38331795f6656fae91b16a0fa9ef17425b60c1c2aff55963229136e766ad"} Oct 31 00:20:58.791135 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:58.791076 4927 scope.go:122] "RemoveContainer" containerID="fccf0c779f70085c9950db7547d74987e208d9579ea0ba23f9ad709ba3e8ae09" Oct 31 00:20:58.793734 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:58.793712 4927 generic.go:356] "Generic (PLEG): container finished" podID="9fd4cd19-a1e6-455a-b36a-c9f58692b081" containerID="da5ec8cc378a8c46a065d5f4772e22cf96a174e9c8b7517d5a74777de2b4de94" exitCode=0 Oct 31 00:20:58.793853 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:58.793787 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-0d75be-7b66b84856-j28w4" event={"ID":"9fd4cd19-a1e6-455a-b36a-c9f58692b081","Type":"ContainerDied","Data":"da5ec8cc378a8c46a065d5f4772e22cf96a174e9c8b7517d5a74777de2b4de94"} Oct 31 00:20:58.888799 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:58.888713 4927 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b774910-1e71-40dd-9d93-026df8b5f9d2-plugins-config-volume\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:20:58.888799 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:58.888748 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r5h5r\" (UniqueName: \"kubernetes.io/projected/0b774910-1e71-40dd-9d93-026df8b5f9d2-kube-api-access-r5h5r\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:20:58.889255 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:58.889228 4927 scope.go:122] "RemoveContainer" containerID="fccf0c779f70085c9950db7547d74987e208d9579ea0ba23f9ad709ba3e8ae09" Oct 31 00:20:58.889729 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:20:58.889698 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fccf0c779f70085c9950db7547d74987e208d9579ea0ba23f9ad709ba3e8ae09\": container with ID starting with fccf0c779f70085c9950db7547d74987e208d9579ea0ba23f9ad709ba3e8ae09 not found: ID does not exist" containerID="fccf0c779f70085c9950db7547d74987e208d9579ea0ba23f9ad709ba3e8ae09" Oct 31 00:20:58.889839 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:58.889742 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fccf0c779f70085c9950db7547d74987e208d9579ea0ba23f9ad709ba3e8ae09"} err="failed to get container status \"fccf0c779f70085c9950db7547d74987e208d9579ea0ba23f9ad709ba3e8ae09\": rpc error: code = NotFound desc = could not find container \"fccf0c779f70085c9950db7547d74987e208d9579ea0ba23f9ad709ba3e8ae09\": container with ID starting with fccf0c779f70085c9950db7547d74987e208d9579ea0ba23f9ad709ba3e8ae09 not found: ID does not exist" Oct 31 00:20:58.903868 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:58.903844 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-0d75be-7b66b84856-j28w4" Oct 31 00:20:58.990156 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:58.990114 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9fd4cd19-a1e6-455a-b36a-c9f58692b081-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9fd4cd19-a1e6-455a-b36a-c9f58692b081-plugins-config-volume\") pod \"9fd4cd19-a1e6-455a-b36a-c9f58692b081\" (UID: \"9fd4cd19-a1e6-455a-b36a-c9f58692b081\") " Oct 31 00:20:58.990384 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:58.990254 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9fd4cd19-a1e6-455a-b36a-c9f58692b081-kube-api-access-nv8nj\" (UniqueName: \"kubernetes.io/projected/9fd4cd19-a1e6-455a-b36a-c9f58692b081-kube-api-access-nv8nj\") pod \"9fd4cd19-a1e6-455a-b36a-c9f58692b081\" (UID: \"9fd4cd19-a1e6-455a-b36a-c9f58692b081\") " Oct 31 00:20:58.991110 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:58.991079 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9fd4cd19-a1e6-455a-b36a-c9f58692b081-plugins-config-volume" pod "9fd4cd19-a1e6-455a-b36a-c9f58692b081" (UID: "9fd4cd19-a1e6-455a-b36a-c9f58692b081"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:20:58.992867 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:58.992842 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fd4cd19-a1e6-455a-b36a-c9f58692b081-kube-api-access-nv8nj" pod "9fd4cd19-a1e6-455a-b36a-c9f58692b081" (UID: "9fd4cd19-a1e6-455a-b36a-c9f58692b081"). InnerVolumeSpecName "kube-api-access-nv8nj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:20:58.995095 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:58.995062 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-0d75be-75f9855f5-kftq4"] Oct 31 00:20:58.999359 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:58.999323 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-0d75be-75f9855f5-kftq4"] Oct 31 00:20:59.026360 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:59.018592 4927 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-jpjkl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.244:15021/healthz/ready\": dial tcp 10.132.0.244:15021: connect: connection refused" start-of-body= Oct 31 00:20:59.027420 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:59.027383 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-jpjkl" podUID="334942ae-4532-436c-b274-532b56c035ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.244:15021/healthz/ready\": dial tcp 10.132.0.244:15021: connect: connection refused" Oct 31 00:20:59.050358 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:59.027272 4927 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-mz295 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.245:15021/healthz/ready\": dial tcp 10.132.0.245:15021: connect: connection refused" start-of-body= Oct 31 00:20:59.059572 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:59.059505 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-mz295" podUID="c76ae81c-b5d0-4c9b-af8f-0a95d2aabf9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.245:15021/healthz/ready\": dial tcp 10.132.0.245:15021: connect: connection refused" Oct 31 00:20:59.087693 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:59.087661 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b774910-1e71-40dd-9d93-026df8b5f9d2" path="/var/lib/kubelet/pods/0b774910-1e71-40dd-9d93-026df8b5f9d2/volumes" Oct 31 00:20:59.092194 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:59.092163 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nv8nj\" (UniqueName: \"kubernetes.io/projected/9fd4cd19-a1e6-455a-b36a-c9f58692b081-kube-api-access-nv8nj\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:20:59.092287 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:59.092227 4927 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9fd4cd19-a1e6-455a-b36a-c9f58692b081-plugins-config-volume\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:20:59.563358 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:59.562707 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-0d75be-7b66b84856-j28w4" podUID="9fd4cd19-a1e6-455a-b36a-c9f58692b081" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.187:50052\" within 1s: context deadline exceeded" Oct 31 00:20:59.563358 ip-10-0-141-180 kubenswrapper[4927]: W1031 00:20:59.563074 4927 logging.go:55] [core] [Channel #9053 SubChannel #9054]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.187:50052", ServerName: "10.132.0.187:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.187:50052: operation was canceled" Oct 31 00:20:59.677387 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:59.677344 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-fe01fa-698f78d94b-4wmlb"] Oct 31 00:20:59.803955 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:59.803922 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-0d75be-7b66b84856-j28w4" event={"ID":"9fd4cd19-a1e6-455a-b36a-c9f58692b081","Type":"ContainerDied","Data":"d753420c050efb812f8d81e84f4a5c5d0084d2adf1946eabe46f1cede1245bcc"} Oct 31 00:20:59.803955 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:59.803944 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-0d75be-7b66b84856-j28w4" Oct 31 00:20:59.804222 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:59.803981 4927 scope.go:122] "RemoveContainer" containerID="da5ec8cc378a8c46a065d5f4772e22cf96a174e9c8b7517d5a74777de2b4de94" Oct 31 00:20:59.986315 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:59.986281 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-0d75be-7b66b84856-j28w4"] Oct 31 00:20:59.990125 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:20:59.990104 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-0d75be-7b66b84856-j28w4"] Oct 31 00:21:00.017806 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:21:00.017762 4927 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-jpjkl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.244:15021/healthz/ready\": dial tcp 10.132.0.244:15021: connect: connection refused" start-of-body= Oct 31 00:21:00.018007 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:21:00.017843 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-jpjkl" podUID="334942ae-4532-436c-b274-532b56c035ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.244:15021/healthz/ready\": dial tcp 10.132.0.244:15021: connect: connection refused" Oct 31 00:21:00.051356 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:21:00.043054 4927 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-mz295 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.245:15021/healthz/ready\": dial tcp 10.132.0.245:15021: connect: connection refused" start-of-body= Oct 31 00:21:00.051356 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:21:00.043113 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-mz295" podUID="c76ae81c-b5d0-4c9b-af8f-0a95d2aabf9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.245:15021/healthz/ready\": dial tcp 10.132.0.245:15021: connect: connection refused" Oct 31 00:21:00.057156 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:21:00.057116 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:21:00.336372 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:21:00.313002 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:21:01.019357 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:21:01.015778 4927 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-jpjkl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.244:15021/healthz/ready\": dial tcp 10.132.0.244:15021: connect: connection refused" start-of-body= Oct 31 00:21:01.019357 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:21:01.015847 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-jpjkl" podUID="334942ae-4532-436c-b274-532b56c035ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.244:15021/healthz/ready\": dial tcp 10.132.0.244:15021: connect: connection refused" Oct 31 00:21:01.027355 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:21:01.024699 4927 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-mz295 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.245:15021/healthz/ready\": dial tcp 10.132.0.245:15021: connect: connection refused" start-of-body= Oct 31 00:21:01.027355 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:21:01.024755 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-mz295" podUID="c76ae81c-b5d0-4c9b-af8f-0a95d2aabf9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.245:15021/healthz/ready\": dial tcp 10.132.0.245:15021: connect: connection refused" Oct 31 00:21:01.130367 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:21:01.126661 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9fd4cd19-a1e6-455a-b36a-c9f58692b081" path="/var/lib/kubelet/pods/9fd4cd19-a1e6-455a-b36a-c9f58692b081/volumes" Oct 31 00:21:01.563215 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:21:01.563178 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:21:02.012828 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:21:02.012783 4927 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-jpjkl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.244:15021/healthz/ready\": dial tcp 10.132.0.244:15021: connect: connection refused" start-of-body= Oct 31 00:21:02.013192 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:21:02.013161 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-jpjkl" podUID="334942ae-4532-436c-b274-532b56c035ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.244:15021/healthz/ready\": dial tcp 10.132.0.244:15021: connect: connection refused" Oct 31 00:21:02.022134 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:21:02.022098 4927 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-mz295 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.245:15021/healthz/ready\": dial tcp 10.132.0.245:15021: connect: connection refused" start-of-body= Oct 31 00:21:02.022325 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:21:02.022155 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-mz295" podUID="c76ae81c-b5d0-4c9b-af8f-0a95d2aabf9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.245:15021/healthz/ready\": dial tcp 10.132.0.245:15021: connect: connection refused" Oct 31 00:21:03.012985 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:21:03.012940 4927 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-jpjkl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.244:15021/healthz/ready\": dial tcp 10.132.0.244:15021: connect: connection refused" start-of-body= Oct 31 00:21:03.013584 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:21:03.013012 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-jpjkl" podUID="334942ae-4532-436c-b274-532b56c035ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.244:15021/healthz/ready\": dial tcp 10.132.0.244:15021: connect: connection refused" Oct 31 00:21:03.021489 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:21:03.021448 4927 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-mz295 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.245:15021/healthz/ready\": dial tcp 10.132.0.245:15021: connect: connection refused" start-of-body= Oct 31 00:21:03.021688 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:21:03.021515 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-mz295" podUID="c76ae81c-b5d0-4c9b-af8f-0a95d2aabf9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.245:15021/healthz/ready\": dial tcp 10.132.0.245:15021: connect: connection refused" Oct 31 00:21:04.678075 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:21:04.677987 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w5-fe01fa-698f78d94b-4wmlb" podUID="518493d9-43fd-451c-abc6-fa2db3010c48" containerName="payload-processing" containerID="cri-o://9f32da5fe52b33933496ee738a53fb3b95afe1e248b6cfe85bd542ea9684a90b" gracePeriod=25 Oct 31 00:21:04.933468 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:21:04.933387 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-jpjkl" Oct 31 00:21:04.934380 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:21:04.934257 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-jpjkl" Oct 31 00:21:04.935578 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:21:04.935552 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-jpjkl" Oct 31 00:21:05.023005 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:21:05.022964 4927 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-mz295 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.245:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:21:05.023193 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:21:05.023034 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-mz295" podUID="c76ae81c-b5d0-4c9b-af8f-0a95d2aabf9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.245:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:21:05.048809 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:21:05.048777 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-mz295" Oct 31 00:21:05.049426 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:21:05.049397 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-mz295" Oct 31 00:21:05.050804 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:21:05.050771 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-mz295" Oct 31 00:21:06.182586 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:21:06.182546 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:21:07.556069 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:21:07.555983 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-fe01fa/distinct-2-w5-fe01fa-kserve-79b99966d8-dkhzd" podUID="d5c1c9e8-ce47-4868-860d-b8413ff74ab5" containerName="main" containerID="cri-o://a3f6dc6ac8204c37462a61c318ca30feb277f09f4ed9fbe6e854ef2d90b48a4f" gracePeriod=45 Oct 31 00:21:07.837446 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:21:07.837416 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-fe01fa/distinct-2-w5-fe01fa-kserve-79b99966d8-dkhzd" Oct 31 00:21:07.870955 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:21:07.870927 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5c1c9e8-ce47-4868-860d-b8413ff74ab5-kube-api-access-ksfpk\" (UniqueName: \"kubernetes.io/projected/d5c1c9e8-ce47-4868-860d-b8413ff74ab5-kube-api-access-ksfpk\") pod \"d5c1c9e8-ce47-4868-860d-b8413ff74ab5\" (UID: \"d5c1c9e8-ce47-4868-860d-b8413ff74ab5\") " Oct 31 00:21:07.871114 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:21:07.871013 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5c1c9e8-ce47-4868-860d-b8413ff74ab5-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d5c1c9e8-ce47-4868-860d-b8413ff74ab5-dshm\") pod \"d5c1c9e8-ce47-4868-860d-b8413ff74ab5\" (UID: \"d5c1c9e8-ce47-4868-860d-b8413ff74ab5\") " Oct 31 00:21:07.871114 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:21:07.871090 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5c1c9e8-ce47-4868-860d-b8413ff74ab5-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d5c1c9e8-ce47-4868-860d-b8413ff74ab5-tmp-dir\") pod \"d5c1c9e8-ce47-4868-860d-b8413ff74ab5\" (UID: \"d5c1c9e8-ce47-4868-860d-b8413ff74ab5\") " Oct 31 00:21:07.871248 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:21:07.871186 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5c1c9e8-ce47-4868-860d-b8413ff74ab5-home\" (UniqueName: \"kubernetes.io/empty-dir/d5c1c9e8-ce47-4868-860d-b8413ff74ab5-home\") pod \"d5c1c9e8-ce47-4868-860d-b8413ff74ab5\" (UID: \"d5c1c9e8-ce47-4868-860d-b8413ff74ab5\") " Oct 31 00:21:07.871307 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:21:07.871249 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d5c1c9e8-ce47-4868-860d-b8413ff74ab5-tls-certs\" (UniqueName: \"kubernetes.io/secret/d5c1c9e8-ce47-4868-860d-b8413ff74ab5-tls-certs\") pod \"d5c1c9e8-ce47-4868-860d-b8413ff74ab5\" (UID: \"d5c1c9e8-ce47-4868-860d-b8413ff74ab5\") " Oct 31 00:21:07.871307 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:21:07.871287 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5c1c9e8-ce47-4868-860d-b8413ff74ab5-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d5c1c9e8-ce47-4868-860d-b8413ff74ab5-model-cache\") pod \"d5c1c9e8-ce47-4868-860d-b8413ff74ab5\" (UID: \"d5c1c9e8-ce47-4868-860d-b8413ff74ab5\") " Oct 31 00:21:07.871457 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:21:07.871370 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5c1c9e8-ce47-4868-860d-b8413ff74ab5-tmp-dir" pod "d5c1c9e8-ce47-4868-860d-b8413ff74ab5" (UID: "d5c1c9e8-ce47-4868-860d-b8413ff74ab5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:21:07.871532 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:21:07.871456 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5c1c9e8-ce47-4868-860d-b8413ff74ab5-home" pod "d5c1c9e8-ce47-4868-860d-b8413ff74ab5" (UID: "d5c1c9e8-ce47-4868-860d-b8413ff74ab5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:21:07.871647 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:21:07.871625 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5c1c9e8-ce47-4868-860d-b8413ff74ab5-model-cache" pod "d5c1c9e8-ce47-4868-860d-b8413ff74ab5" (UID: "d5c1c9e8-ce47-4868-860d-b8413ff74ab5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:21:07.871900 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:21:07.871882 4927 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d5c1c9e8-ce47-4868-860d-b8413ff74ab5-model-cache\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:21:07.871962 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:21:07.871907 4927 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d5c1c9e8-ce47-4868-860d-b8413ff74ab5-tmp-dir\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:21:07.871962 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:21:07.871921 4927 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d5c1c9e8-ce47-4868-860d-b8413ff74ab5-home\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:21:07.873454 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:21:07.873434 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5c1c9e8-ce47-4868-860d-b8413ff74ab5-tls-certs" pod "d5c1c9e8-ce47-4868-860d-b8413ff74ab5" (UID: "d5c1c9e8-ce47-4868-860d-b8413ff74ab5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:21:07.873661 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:21:07.873637 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5c1c9e8-ce47-4868-860d-b8413ff74ab5-dshm" pod "d5c1c9e8-ce47-4868-860d-b8413ff74ab5" (UID: "d5c1c9e8-ce47-4868-860d-b8413ff74ab5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:21:07.873661 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:21:07.873649 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5c1c9e8-ce47-4868-860d-b8413ff74ab5-kube-api-access-ksfpk" pod "d5c1c9e8-ce47-4868-860d-b8413ff74ab5" (UID: "d5c1c9e8-ce47-4868-860d-b8413ff74ab5"). InnerVolumeSpecName "kube-api-access-ksfpk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:21:07.931911 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:21:07.931871 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w5-fe01fa-698f78d94b-4wmlb" podUID="518493d9-43fd-451c-abc6-fa2db3010c48" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 00:21:07.940402 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:21:07.940368 4927 generic.go:356] "Generic (PLEG): container finished" podID="d5c1c9e8-ce47-4868-860d-b8413ff74ab5" containerID="a3f6dc6ac8204c37462a61c318ca30feb277f09f4ed9fbe6e854ef2d90b48a4f" exitCode=0 Oct 31 00:21:07.940558 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:21:07.940411 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-fe01fa/distinct-2-w5-fe01fa-kserve-79b99966d8-dkhzd" Oct 31 00:21:07.940558 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:21:07.940453 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-fe01fa/distinct-2-w5-fe01fa-kserve-79b99966d8-dkhzd" event={"ID":"d5c1c9e8-ce47-4868-860d-b8413ff74ab5","Type":"ContainerDied","Data":"a3f6dc6ac8204c37462a61c318ca30feb277f09f4ed9fbe6e854ef2d90b48a4f"} Oct 31 00:21:07.940558 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:21:07.940505 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-fe01fa/distinct-2-w5-fe01fa-kserve-79b99966d8-dkhzd" event={"ID":"d5c1c9e8-ce47-4868-860d-b8413ff74ab5","Type":"ContainerDied","Data":"3eefe03eb843117afeb5f40d3a01897fec7324076d59182319b448a52dc8d5e0"} Oct 31 00:21:07.940558 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:21:07.940528 4927 scope.go:122] "RemoveContainer" containerID="a3f6dc6ac8204c37462a61c318ca30feb277f09f4ed9fbe6e854ef2d90b48a4f" Oct 31 00:21:07.952201 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:21:07.952175 4927 scope.go:122] "RemoveContainer" containerID="a3f6dc6ac8204c37462a61c318ca30feb277f09f4ed9fbe6e854ef2d90b48a4f" Oct 31 00:21:07.952535 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:21:07.952511 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3f6dc6ac8204c37462a61c318ca30feb277f09f4ed9fbe6e854ef2d90b48a4f\": container with ID starting with a3f6dc6ac8204c37462a61c318ca30feb277f09f4ed9fbe6e854ef2d90b48a4f not found: ID does not exist" containerID="a3f6dc6ac8204c37462a61c318ca30feb277f09f4ed9fbe6e854ef2d90b48a4f" Oct 31 00:21:07.952601 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:21:07.952546 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3f6dc6ac8204c37462a61c318ca30feb277f09f4ed9fbe6e854ef2d90b48a4f"} err="failed to get container status \"a3f6dc6ac8204c37462a61c318ca30feb277f09f4ed9fbe6e854ef2d90b48a4f\": rpc error: code = NotFound desc = could not find container \"a3f6dc6ac8204c37462a61c318ca30feb277f09f4ed9fbe6e854ef2d90b48a4f\": container with ID starting with a3f6dc6ac8204c37462a61c318ca30feb277f09f4ed9fbe6e854ef2d90b48a4f not found: ID does not exist" Oct 31 00:21:07.973820 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:21:07.973791 4927 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d5c1c9e8-ce47-4868-860d-b8413ff74ab5-tls-certs\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:21:07.973820 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:21:07.973818 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ksfpk\" (UniqueName: \"kubernetes.io/projected/d5c1c9e8-ce47-4868-860d-b8413ff74ab5-kube-api-access-ksfpk\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:21:07.973820 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:21:07.973829 4927 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d5c1c9e8-ce47-4868-860d-b8413ff74ab5-dshm\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:21:08.090705 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:21:08.090614 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-fe01fa/distinct-2-w5-fe01fa-kserve-79b99966d8-dkhzd"] Oct 31 00:21:08.093570 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:21:08.093538 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-fe01fa/distinct-2-w5-fe01fa-kserve-79b99966d8-dkhzd"] Oct 31 00:21:09.089171 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:21:09.089123 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d5c1c9e8-ce47-4868-860d-b8413ff74ab5" path="/var/lib/kubelet/pods/d5c1c9e8-ce47-4868-860d-b8413ff74ab5/volumes" Oct 31 00:21:11.189624 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:21:11.188811 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:21:15.832892 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:21:15.832853 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:21:17.266090 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:21:17.266055 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:21:17.931619 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:21:17.931575 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w5-fe01fa-698f78d94b-4wmlb" podUID="518493d9-43fd-451c-abc6-fa2db3010c48" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 00:21:17.992523 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:21:17.992481 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:21:18.859357 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:21:18.858815 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:21:19.018758 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:21:19.018654 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:21:19.682574 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:21:19.682535 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sqv69" containerName="extractor" Oct 31 00:21:20.697514 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:21:20.697477 4927 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/system.slice/system-systemd\\\\x2dcoredump.slice/systemd-coredump@21-42521-0.service\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a91e846_1ef4_4063_860e_46cdbf225ced.slice/crio-conmon-75c18b5577f51dca639c1901a93d0bd173d4a1772389f80ba0fe87b53cab3d52.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a91e846_1ef4_4063_860e_46cdbf225ced.slice/crio-75c18b5577f51dca639c1901a93d0bd173d4a1772389f80ba0fe87b53cab3d52.scope\": RecentStats: unable to find data in memory cache]" Oct 31 00:21:20.745356 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:21:20.744935 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:21:21.107469 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:21:21.107429 4927 generic.go:356] "Generic (PLEG): container finished" podID="8a91e846-1ef4-4063-860e-46cdbf225ced" containerID="75c18b5577f51dca639c1901a93d0bd173d4a1772389f80ba0fe87b53cab3d52" exitCode=0 Oct 31 00:21:21.107693 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:21:21.107511 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ddf61a-openshift-default-f57f54bc-4gvfk" event={"ID":"8a91e846-1ef4-4063-860e-46cdbf225ced","Type":"ContainerDied","Data":"75c18b5577f51dca639c1901a93d0bd173d4a1772389f80ba0fe87b53cab3d52"} Oct 31 00:21:21.108026 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:21:21.108004 4927 scope.go:122] "RemoveContainer" containerID="75c18b5577f51dca639c1901a93d0bd173d4a1772389f80ba0fe87b53cab3d52" Oct 31 00:21:21.110207 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:21:21.110170 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:21:21.110320 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:21:21.110279 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:21:21.110424 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:21:21.110325 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:21:22.170215 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:21:22.170177 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ddf61a-openshift-default-f57f54bc-4gvfk" event={"ID":"8a91e846-1ef4-4063-860e-46cdbf225ced","Type":"ContainerStarted","Data":"5f78ac7307756be3414509c8fea905737c9432c9e9f25c7f910b02b7bcb9c51e"} Oct 31 00:21:23.076185 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:21:23.076142 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:21:23.090026 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:21:23.089995 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-ddf61a-openshift-default-f57f54bc-4gvfk" Oct 31 00:21:23.091516 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:21:23.091493 4927 patch_prober.go:29] interesting pod/e2e-inf-a-ddf61a-openshift-default-f57f54bc-4gvfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.242:15021/healthz/ready\": dial tcp 10.132.0.242:15021: connect: connection refused" start-of-body= Oct 31 00:21:23.091618 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:21:23.091538 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ddf61a-openshift-default-f57f54bc-4gvfk" podUID="8a91e846-1ef4-4063-860e-46cdbf225ced" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.242:15021/healthz/ready\": dial tcp 10.132.0.242:15021: connect: connection refused" Oct 31 00:21:23.291200 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:21:23.291165 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:21:23.556768 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:21:23.556736 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-ddf61a-openshift-default-f57f54bc-4gvfk" Oct 31 00:21:24.090078 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:21:24.090043 4927 patch_prober.go:29] interesting pod/e2e-inf-a-ddf61a-openshift-default-f57f54bc-4gvfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.242:15021/healthz/ready\": dial tcp 10.132.0.242:15021: connect: connection refused" start-of-body= Oct 31 00:21:24.090291 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:21:24.090108 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ddf61a-openshift-default-f57f54bc-4gvfk" podUID="8a91e846-1ef4-4063-860e-46cdbf225ced" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.242:15021/healthz/ready\": dial tcp 10.132.0.242:15021: connect: connection refused" Oct 31 00:21:25.043471 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:21:25.043443 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-fe01fa-698f78d94b-4wmlb" Oct 31 00:21:25.090084 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:21:25.090050 4927 patch_prober.go:29] interesting pod/e2e-inf-a-ddf61a-openshift-default-f57f54bc-4gvfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.242:15021/healthz/ready\": dial tcp 10.132.0.242:15021: connect: connection refused" start-of-body= Oct 31 00:21:25.090238 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:21:25.090103 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ddf61a-openshift-default-f57f54bc-4gvfk" podUID="8a91e846-1ef4-4063-860e-46cdbf225ced" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.242:15021/healthz/ready\": dial tcp 10.132.0.242:15021: connect: connection refused" Oct 31 00:21:25.166957 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:21:25.166916 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/518493d9-43fd-451c-abc6-fa2db3010c48-kube-api-access-clv8m\" (UniqueName: \"kubernetes.io/projected/518493d9-43fd-451c-abc6-fa2db3010c48-kube-api-access-clv8m\") pod \"518493d9-43fd-451c-abc6-fa2db3010c48\" (UID: \"518493d9-43fd-451c-abc6-fa2db3010c48\") " Oct 31 00:21:25.167173 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:21:25.167028 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/518493d9-43fd-451c-abc6-fa2db3010c48-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/518493d9-43fd-451c-abc6-fa2db3010c48-plugins-config-volume\") pod \"518493d9-43fd-451c-abc6-fa2db3010c48\" (UID: \"518493d9-43fd-451c-abc6-fa2db3010c48\") " Oct 31 00:21:25.168183 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:21:25.168154 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/518493d9-43fd-451c-abc6-fa2db3010c48-plugins-config-volume" pod "518493d9-43fd-451c-abc6-fa2db3010c48" (UID: "518493d9-43fd-451c-abc6-fa2db3010c48"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:21:25.177437 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:21:25.177403 4927 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/518493d9-43fd-451c-abc6-fa2db3010c48-plugins-config-volume\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:21:25.178540 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:21:25.178515 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/518493d9-43fd-451c-abc6-fa2db3010c48-kube-api-access-clv8m" pod "518493d9-43fd-451c-abc6-fa2db3010c48" (UID: "518493d9-43fd-451c-abc6-fa2db3010c48"). InnerVolumeSpecName "kube-api-access-clv8m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:21:25.194460 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:21:25.194427 4927 generic.go:356] "Generic (PLEG): container finished" podID="518493d9-43fd-451c-abc6-fa2db3010c48" containerID="9f32da5fe52b33933496ee738a53fb3b95afe1e248b6cfe85bd542ea9684a90b" exitCode=0 Oct 31 00:21:25.194622 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:21:25.194483 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-fe01fa-698f78d94b-4wmlb" Oct 31 00:21:25.194622 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:21:25.194498 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-fe01fa-698f78d94b-4wmlb" event={"ID":"518493d9-43fd-451c-abc6-fa2db3010c48","Type":"ContainerDied","Data":"9f32da5fe52b33933496ee738a53fb3b95afe1e248b6cfe85bd542ea9684a90b"} Oct 31 00:21:25.194622 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:21:25.194565 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-fe01fa-698f78d94b-4wmlb" event={"ID":"518493d9-43fd-451c-abc6-fa2db3010c48","Type":"ContainerDied","Data":"6ce510ecf01fbfa10ec89b3e89c8c21eb9098d2a787abed48dfbab8351cdd661"} Oct 31 00:21:25.194622 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:21:25.194596 4927 scope.go:122] "RemoveContainer" containerID="9f32da5fe52b33933496ee738a53fb3b95afe1e248b6cfe85bd542ea9684a90b" Oct 31 00:21:25.234798 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:21:25.234764 4927 scope.go:122] "RemoveContainer" containerID="9f32da5fe52b33933496ee738a53fb3b95afe1e248b6cfe85bd542ea9684a90b" Oct 31 00:21:25.235297 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:21:25.235266 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f32da5fe52b33933496ee738a53fb3b95afe1e248b6cfe85bd542ea9684a90b\": container with ID starting with 9f32da5fe52b33933496ee738a53fb3b95afe1e248b6cfe85bd542ea9684a90b not found: ID does not exist" containerID="9f32da5fe52b33933496ee738a53fb3b95afe1e248b6cfe85bd542ea9684a90b" Oct 31 00:21:25.235421 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:21:25.235322 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f32da5fe52b33933496ee738a53fb3b95afe1e248b6cfe85bd542ea9684a90b"} err="failed to get container status \"9f32da5fe52b33933496ee738a53fb3b95afe1e248b6cfe85bd542ea9684a90b\": rpc error: code = NotFound desc = could not find container \"9f32da5fe52b33933496ee738a53fb3b95afe1e248b6cfe85bd542ea9684a90b\": container with ID starting with 9f32da5fe52b33933496ee738a53fb3b95afe1e248b6cfe85bd542ea9684a90b not found: ID does not exist" Oct 31 00:21:25.278904 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:21:25.278821 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-clv8m\" (UniqueName: \"kubernetes.io/projected/518493d9-43fd-451c-abc6-fa2db3010c48-kube-api-access-clv8m\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:21:25.365347 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:21:25.365299 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-fe01fa-698f78d94b-4wmlb"] Oct 31 00:21:25.369107 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:21:25.369078 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-fe01fa-698f78d94b-4wmlb"] Oct 31 00:21:25.930353 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:21:25.928922 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:21:26.090030 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:21:26.089992 4927 patch_prober.go:29] interesting pod/e2e-inf-a-ddf61a-openshift-default-f57f54bc-4gvfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.242:15021/healthz/ready\": dial tcp 10.132.0.242:15021: connect: connection refused" start-of-body= Oct 31 00:21:26.090424 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:21:26.090043 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ddf61a-openshift-default-f57f54bc-4gvfk" podUID="8a91e846-1ef4-4063-860e-46cdbf225ced" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.242:15021/healthz/ready\": dial tcp 10.132.0.242:15021: connect: connection refused" Oct 31 00:21:26.187724 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:21:26.187633 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:21:26.410638 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:21:26.410497 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:21:27.096370 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:21:27.096315 4927 patch_prober.go:29] interesting pod/e2e-inf-a-ddf61a-openshift-default-f57f54bc-4gvfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.242:15021/healthz/ready\": dial tcp 10.132.0.242:15021: connect: connection refused" start-of-body= Oct 31 00:21:27.096855 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:21:27.096405 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ddf61a-openshift-default-f57f54bc-4gvfk" podUID="8a91e846-1ef4-4063-860e-46cdbf225ced" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.242:15021/healthz/ready\": dial tcp 10.132.0.242:15021: connect: connection refused" Oct 31 00:21:27.099106 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:21:27.099078 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="518493d9-43fd-451c-abc6-fa2db3010c48" path="/var/lib/kubelet/pods/518493d9-43fd-451c-abc6-fa2db3010c48/volumes" Oct 31 00:21:28.094754 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:21:28.094724 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-ddf61a-openshift-default-f57f54bc-4gvfk" Oct 31 00:21:28.095751 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:21:28.095731 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-ddf61a-openshift-default-f57f54bc-4gvfk" Oct 31 00:21:28.651494 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:21:28.651468 4927 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-pqt2c_6382dfb5-a363-47a2-bc79-0b96f7bec0c3/console-operator/3.log" Oct 31 00:21:29.661912 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:21:29.661883 4927 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-pqt2c_6382dfb5-a363-47a2-bc79-0b96f7bec0c3/console-operator/3.log" Oct 31 00:21:29.916451 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:21:29.916270 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:21:30.072311 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:21:30.072277 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:21:30.794279 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:21:30.794243 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:21:30.971106 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:21:30.966422 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:21:31.644144 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:21:31.644105 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:21:35.937530 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:21:35.937494 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:21:36.130942 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:21:36.130888 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:21:36.329217 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:21:36.329124 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:21:37.222421 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:21:37.222374 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:21:37.405170 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:21:37.405117 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:21:41.008186 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:21:41.008146 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:21:45.151520 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:21:45.151480 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-0d75be-5c9666c858-4qdvv"] Oct 31 00:21:45.152034 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:21:45.151746 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-0d75be-5c9666c858-4qdvv" podUID="08022abe-d8cb-4d22-b531-d3f823e938ff" containerName="maas-api" containerID="cri-o://e5aa7e51db05d9d85bb1de7256afff7f3121a7c7d8b5c405e4df1c6a850a6655" gracePeriod=30 Oct 31 00:21:45.231382 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:21:45.231347 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:21:45.412986 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:21:45.412958 4927 generic.go:356] "Generic (PLEG): container finished" podID="08022abe-d8cb-4d22-b531-d3f823e938ff" containerID="e5aa7e51db05d9d85bb1de7256afff7f3121a7c7d8b5c405e4df1c6a850a6655" exitCode=0 Oct 31 00:21:45.413281 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:21:45.413264 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-0d75be-5c9666c858-4qdvv" event={"ID":"08022abe-d8cb-4d22-b531-d3f823e938ff","Type":"ContainerDied","Data":"e5aa7e51db05d9d85bb1de7256afff7f3121a7c7d8b5c405e4df1c6a850a6655"} Oct 31 00:21:45.508772 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:21:45.508744 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-0d75be-5c9666c858-4qdvv" Oct 31 00:21:45.598106 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:21:45.597226 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/08022abe-d8cb-4d22-b531-d3f823e938ff-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/08022abe-d8cb-4d22-b531-d3f823e938ff-maas-api-tls\") pod \"08022abe-d8cb-4d22-b531-d3f823e938ff\" (UID: \"08022abe-d8cb-4d22-b531-d3f823e938ff\") " Oct 31 00:21:45.598106 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:21:45.597280 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/08022abe-d8cb-4d22-b531-d3f823e938ff-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/08022abe-d8cb-4d22-b531-d3f823e938ff-openshift-service-ca\") pod \"08022abe-d8cb-4d22-b531-d3f823e938ff\" (UID: \"08022abe-d8cb-4d22-b531-d3f823e938ff\") " Oct 31 00:21:45.598106 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:21:45.597478 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/08022abe-d8cb-4d22-b531-d3f823e938ff-metrics-certs\" (UniqueName: \"kubernetes.io/secret/08022abe-d8cb-4d22-b531-d3f823e938ff-metrics-certs\") pod \"08022abe-d8cb-4d22-b531-d3f823e938ff\" (UID: \"08022abe-d8cb-4d22-b531-d3f823e938ff\") " Oct 31 00:21:45.598106 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:21:45.597733 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08022abe-d8cb-4d22-b531-d3f823e938ff-kube-api-access-54n9c\" (UniqueName: \"kubernetes.io/projected/08022abe-d8cb-4d22-b531-d3f823e938ff-kube-api-access-54n9c\") pod \"08022abe-d8cb-4d22-b531-d3f823e938ff\" (UID: \"08022abe-d8cb-4d22-b531-d3f823e938ff\") " Oct 31 00:21:45.598780 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:21:45.598744 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08022abe-d8cb-4d22-b531-d3f823e938ff-openshift-service-ca" pod "08022abe-d8cb-4d22-b531-d3f823e938ff" (UID: "08022abe-d8cb-4d22-b531-d3f823e938ff"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:21:45.601678 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:21:45.601651 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/08022abe-d8cb-4d22-b531-d3f823e938ff-metrics-certs" pod "08022abe-d8cb-4d22-b531-d3f823e938ff" (UID: "08022abe-d8cb-4d22-b531-d3f823e938ff"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:21:45.601832 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:21:45.601811 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08022abe-d8cb-4d22-b531-d3f823e938ff-kube-api-access-54n9c" pod "08022abe-d8cb-4d22-b531-d3f823e938ff" (UID: "08022abe-d8cb-4d22-b531-d3f823e938ff"). InnerVolumeSpecName "kube-api-access-54n9c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:21:45.642356 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:21:45.628947 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/08022abe-d8cb-4d22-b531-d3f823e938ff-maas-api-tls" pod "08022abe-d8cb-4d22-b531-d3f823e938ff" (UID: "08022abe-d8cb-4d22-b531-d3f823e938ff"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:21:45.700481 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:21:45.700398 4927 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/08022abe-d8cb-4d22-b531-d3f823e938ff-metrics-certs\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:21:45.700481 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:21:45.700439 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-54n9c\" (UniqueName: \"kubernetes.io/projected/08022abe-d8cb-4d22-b531-d3f823e938ff-kube-api-access-54n9c\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:21:45.700481 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:21:45.700455 4927 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/08022abe-d8cb-4d22-b531-d3f823e938ff-maas-api-tls\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:21:45.700481 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:21:45.700471 4927 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/08022abe-d8cb-4d22-b531-d3f823e938ff-openshift-service-ca\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:21:46.421925 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:21:46.421893 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-0d75be-5c9666c858-4qdvv" Oct 31 00:21:46.422444 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:21:46.421929 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-0d75be-5c9666c858-4qdvv" event={"ID":"08022abe-d8cb-4d22-b531-d3f823e938ff","Type":"ContainerDied","Data":"019e3d977bf4317bb20ae98acac8cdb2a449b7f3c99f47d252b391ad013652b5"} Oct 31 00:21:46.422444 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:21:46.421988 4927 scope.go:122] "RemoveContainer" containerID="e5aa7e51db05d9d85bb1de7256afff7f3121a7c7d8b5c405e4df1c6a850a6655" Oct 31 00:21:46.593427 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:21:46.593397 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-0d75be-5c9666c858-4qdvv"] Oct 31 00:21:46.646835 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:21:46.646781 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-0d75be-5c9666c858-4qdvv"] Oct 31 00:21:46.660169 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:21:46.660132 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-d687ff-6f87cb4d9c-m557f"] Oct 31 00:21:46.660915 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:21:46.660892 4927 cpu_manager.go:414] "containerMap: removing container" podUID="9fd4cd19-a1e6-455a-b36a-c9f58692b081" containerName="payload-processing" Oct 31 00:21:46.660915 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:21:46.660915 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9fd4cd19-a1e6-455a-b36a-c9f58692b081" containerName="payload-processing" Oct 31 00:21:46.661112 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:21:46.660969 4927 cpu_manager.go:414] "containerMap: removing container" podUID="0b774910-1e71-40dd-9d93-026df8b5f9d2" containerName="payload-pre-processing" Oct 31 00:21:46.661112 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:21:46.660979 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b774910-1e71-40dd-9d93-026df8b5f9d2" containerName="payload-pre-processing" Oct 31 00:21:46.661112 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:21:46.660997 4927 cpu_manager.go:414] "containerMap: removing container" podUID="d5c1c9e8-ce47-4868-860d-b8413ff74ab5" containerName="main" Oct 31 00:21:46.661112 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:21:46.661005 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5c1c9e8-ce47-4868-860d-b8413ff74ab5" containerName="main" Oct 31 00:21:46.661112 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:21:46.661031 4927 cpu_manager.go:414] "containerMap: removing container" podUID="518493d9-43fd-451c-abc6-fa2db3010c48" containerName="payload-processing" Oct 31 00:21:46.661112 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:21:46.661038 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="518493d9-43fd-451c-abc6-fa2db3010c48" containerName="payload-processing" Oct 31 00:21:46.661112 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:21:46.661048 4927 cpu_manager.go:414] "containerMap: removing container" podUID="08022abe-d8cb-4d22-b531-d3f823e938ff" containerName="maas-api" Oct 31 00:21:46.661112 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:21:46.661055 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08022abe-d8cb-4d22-b531-d3f823e938ff" containerName="maas-api" Oct 31 00:21:46.661579 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:21:46.661231 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="9fd4cd19-a1e6-455a-b36a-c9f58692b081" containerName="payload-processing" Oct 31 00:21:46.661579 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:21:46.661253 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b774910-1e71-40dd-9d93-026df8b5f9d2" containerName="payload-pre-processing" Oct 31 00:21:46.661579 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:21:46.661265 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5c1c9e8-ce47-4868-860d-b8413ff74ab5" containerName="main" Oct 31 00:21:46.661579 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:21:46.661280 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="518493d9-43fd-451c-abc6-fa2db3010c48" containerName="payload-processing" Oct 31 00:21:46.661579 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:21:46.661290 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="08022abe-d8cb-4d22-b531-d3f823e938ff" containerName="maas-api" Oct 31 00:21:46.666607 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:21:46.666584 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d687ff-6f87cb4d9c-m557f" Oct 31 00:21:46.720695 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:21:46.720662 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/29148bdd-fd25-4c41-bf85-bfa6dc2489d8-maas-api-tls\") pod \"maas-api-e2e-api-b-d687ff-6f87cb4d9c-m557f\" (UID: \"29148bdd-fd25-4c41-bf85-bfa6dc2489d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d687ff-6f87cb4d9c-m557f" Oct 31 00:21:46.720845 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:21:46.720732 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/29148bdd-fd25-4c41-bf85-bfa6dc2489d8-openshift-service-ca\") pod \"maas-api-e2e-api-b-d687ff-6f87cb4d9c-m557f\" (UID: \"29148bdd-fd25-4c41-bf85-bfa6dc2489d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d687ff-6f87cb4d9c-m557f" Oct 31 00:21:46.720898 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:21:46.720872 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86nlx\" (UniqueName: \"kubernetes.io/projected/29148bdd-fd25-4c41-bf85-bfa6dc2489d8-kube-api-access-86nlx\") pod \"maas-api-e2e-api-b-d687ff-6f87cb4d9c-m557f\" (UID: \"29148bdd-fd25-4c41-bf85-bfa6dc2489d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d687ff-6f87cb4d9c-m557f" Oct 31 00:21:46.721066 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:21:46.721041 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/29148bdd-fd25-4c41-bf85-bfa6dc2489d8-metrics-certs\") pod \"maas-api-e2e-api-b-d687ff-6f87cb4d9c-m557f\" (UID: \"29148bdd-fd25-4c41-bf85-bfa6dc2489d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d687ff-6f87cb4d9c-m557f" Oct 31 00:21:46.722476 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:21:46.722428 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-d687ff-serving-cert\"" Oct 31 00:21:46.729110 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:21:46.729085 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-d687ff-6f87cb4d9c-m557f"] Oct 31 00:21:46.748135 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:21:46.748108 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:21:46.822472 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:21:46.822438 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/29148bdd-fd25-4c41-bf85-bfa6dc2489d8-metrics-certs\") pod \"maas-api-e2e-api-b-d687ff-6f87cb4d9c-m557f\" (UID: \"29148bdd-fd25-4c41-bf85-bfa6dc2489d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d687ff-6f87cb4d9c-m557f" Oct 31 00:21:46.822641 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:21:46.822526 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/29148bdd-fd25-4c41-bf85-bfa6dc2489d8-maas-api-tls\") pod \"maas-api-e2e-api-b-d687ff-6f87cb4d9c-m557f\" (UID: \"29148bdd-fd25-4c41-bf85-bfa6dc2489d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d687ff-6f87cb4d9c-m557f" Oct 31 00:21:46.822641 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:21:46.822580 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/29148bdd-fd25-4c41-bf85-bfa6dc2489d8-openshift-service-ca\") pod \"maas-api-e2e-api-b-d687ff-6f87cb4d9c-m557f\" (UID: \"29148bdd-fd25-4c41-bf85-bfa6dc2489d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d687ff-6f87cb4d9c-m557f" Oct 31 00:21:46.822768 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:21:46.822657 4927 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-d687ff-serving-cert: secret "maas-api-e2e-api-b-d687ff-serving-cert" not found Oct 31 00:21:46.822768 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:21:46.822695 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86nlx\" (UniqueName: \"kubernetes.io/projected/29148bdd-fd25-4c41-bf85-bfa6dc2489d8-kube-api-access-86nlx\") pod \"maas-api-e2e-api-b-d687ff-6f87cb4d9c-m557f\" (UID: \"29148bdd-fd25-4c41-bf85-bfa6dc2489d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d687ff-6f87cb4d9c-m557f" Oct 31 00:21:46.822768 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:21:46.822711 4927 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/29148bdd-fd25-4c41-bf85-bfa6dc2489d8-maas-api-tls podName:29148bdd-fd25-4c41-bf85-bfa6dc2489d8 nodeName:}" failed. No retries permitted until 2026-10-31 00:21:47.32269744 +0000 UTC m=+2420.359028241 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/29148bdd-fd25-4c41-bf85-bfa6dc2489d8-maas-api-tls") pod "maas-api-e2e-api-b-d687ff-6f87cb4d9c-m557f" (UID: "29148bdd-fd25-4c41-bf85-bfa6dc2489d8") : secret "maas-api-e2e-api-b-d687ff-serving-cert" not found Oct 31 00:21:46.823293 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:21:46.823269 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/29148bdd-fd25-4c41-bf85-bfa6dc2489d8-openshift-service-ca\") pod \"maas-api-e2e-api-b-d687ff-6f87cb4d9c-m557f\" (UID: \"29148bdd-fd25-4c41-bf85-bfa6dc2489d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d687ff-6f87cb4d9c-m557f" Oct 31 00:21:46.824819 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:21:46.824802 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/29148bdd-fd25-4c41-bf85-bfa6dc2489d8-metrics-certs\") pod \"maas-api-e2e-api-b-d687ff-6f87cb4d9c-m557f\" (UID: \"29148bdd-fd25-4c41-bf85-bfa6dc2489d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d687ff-6f87cb4d9c-m557f" Oct 31 00:21:46.920355 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:21:46.915266 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-86nlx\" (UniqueName: \"kubernetes.io/projected/29148bdd-fd25-4c41-bf85-bfa6dc2489d8-kube-api-access-86nlx\") pod \"maas-api-e2e-api-b-d687ff-6f87cb4d9c-m557f\" (UID: \"29148bdd-fd25-4c41-bf85-bfa6dc2489d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d687ff-6f87cb4d9c-m557f" Oct 31 00:21:46.931419 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:21:46.923708 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:21:47.089059 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:21:47.089024 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="08022abe-d8cb-4d22-b531-d3f823e938ff" path="/var/lib/kubelet/pods/08022abe-d8cb-4d22-b531-d3f823e938ff/volumes" Oct 31 00:21:47.093197 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:21:47.093169 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:21:47.329696 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:21:47.329609 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/29148bdd-fd25-4c41-bf85-bfa6dc2489d8-maas-api-tls\") pod \"maas-api-e2e-api-b-d687ff-6f87cb4d9c-m557f\" (UID: \"29148bdd-fd25-4c41-bf85-bfa6dc2489d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d687ff-6f87cb4d9c-m557f" Oct 31 00:21:47.332339 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:21:47.332294 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/29148bdd-fd25-4c41-bf85-bfa6dc2489d8-maas-api-tls\") pod \"maas-api-e2e-api-b-d687ff-6f87cb4d9c-m557f\" (UID: \"29148bdd-fd25-4c41-bf85-bfa6dc2489d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d687ff-6f87cb4d9c-m557f" Oct 31 00:21:47.580811 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:21:47.580729 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d687ff-6f87cb4d9c-m557f" Oct 31 00:21:47.670394 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:21:47.669642 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:21:47.987404 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:21:47.987371 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-d687ff-6f87cb4d9c-m557f"] Oct 31 00:21:47.988131 ip-10-0-141-180 kubenswrapper[4927]: W1031 00:21:47.988107 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29148bdd_fd25_4c41_bf85_bfa6dc2489d8.slice/crio-e58421fc668a27cc8498b2031025ea611be0b794221940d36045a6fbfaef4a23 WatchSource:0}: Error finding container e58421fc668a27cc8498b2031025ea611be0b794221940d36045a6fbfaef4a23: Status 404 returned error can't find the container with id e58421fc668a27cc8498b2031025ea611be0b794221940d36045a6fbfaef4a23 Oct 31 00:21:47.989923 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:21:47.989899 4927 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 00:21:48.439984 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:21:48.439953 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d687ff-6f87cb4d9c-m557f" event={"ID":"29148bdd-fd25-4c41-bf85-bfa6dc2489d8","Type":"ContainerStarted","Data":"e58421fc668a27cc8498b2031025ea611be0b794221940d36045a6fbfaef4a23"} Oct 31 00:21:49.449454 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:21:49.449422 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d687ff-6f87cb4d9c-m557f" event={"ID":"29148bdd-fd25-4c41-bf85-bfa6dc2489d8","Type":"ContainerStarted","Data":"0a417e90b14f56265c300803c8bd93e02dd45a54af137d382980caf0cd25b9be"} Oct 31 00:21:49.449919 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:21:49.449541 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d687ff-6f87cb4d9c-m557f" Oct 31 00:21:49.582570 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:21:49.582514 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d687ff-6f87cb4d9c-m557f" podStartSLOduration=2.654409052 podStartE2EDuration="3.582498712s" totalImagesPullingTime="928.08966ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:21:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:21:49.57913149 +0000 UTC m=+2422.615462336" watchObservedRunningTime="2026-10-31 00:21:49.582498712 +0000 UTC m=+2422.618829534" Oct 31 00:21:49.683811 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:21:49.683767 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sqv69" containerName="extractor" Oct 31 00:21:52.057580 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:21:52.057519 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:21:54.201244 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:21:54.201211 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:21:54.356992 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:21:54.356953 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:21:54.581593 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:21:54.581517 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-0d75be-openshift-default-57f99d9cbc-smttn"] Oct 31 00:21:54.581803 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:21:54.581774 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w6-0d75be-openshift-default-57f99d9cbc-smttn" podUID="61bb8f6c-7dd6-4e87-ac44-5374dd11c0dc" containerName="istio-proxy" containerID="cri-o://17083e75a5e87c8c1227a281fb6b74a9245389f2e1d42a190bbcaaad71dba081" gracePeriod=30 Oct 31 00:21:54.591055 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:21:54.591021 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:21:55.083169 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:21:55.083132 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:21:55.464911 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:21:55.464876 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d687ff-6f87cb4d9c-m557f" Oct 31 00:21:57.077120 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:21:57.077046 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:21:59.868942 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:21:59.868911 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-0d75be-openshift-default-57f99d9cbc-smttn" Oct 31 00:22:00.011579 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:22:00.011545 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/61bb8f6c-7dd6-4e87-ac44-5374dd11c0dc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61bb8f6c-7dd6-4e87-ac44-5374dd11c0dc-istiod-ca-cert\") pod \"61bb8f6c-7dd6-4e87-ac44-5374dd11c0dc\" (UID: \"61bb8f6c-7dd6-4e87-ac44-5374dd11c0dc\") " Oct 31 00:22:00.011751 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:22:00.011645 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61bb8f6c-7dd6-4e87-ac44-5374dd11c0dc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61bb8f6c-7dd6-4e87-ac44-5374dd11c0dc-workload-certs\") pod \"61bb8f6c-7dd6-4e87-ac44-5374dd11c0dc\" (UID: \"61bb8f6c-7dd6-4e87-ac44-5374dd11c0dc\") " Oct 31 00:22:00.011751 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:22:00.011686 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61bb8f6c-7dd6-4e87-ac44-5374dd11c0dc-istio-token\" (UniqueName: \"kubernetes.io/projected/61bb8f6c-7dd6-4e87-ac44-5374dd11c0dc-istio-token\") pod \"61bb8f6c-7dd6-4e87-ac44-5374dd11c0dc\" (UID: \"61bb8f6c-7dd6-4e87-ac44-5374dd11c0dc\") " Oct 31 00:22:00.011751 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:22:00.011726 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/61bb8f6c-7dd6-4e87-ac44-5374dd11c0dc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61bb8f6c-7dd6-4e87-ac44-5374dd11c0dc-istio-podinfo\") pod \"61bb8f6c-7dd6-4e87-ac44-5374dd11c0dc\" (UID: \"61bb8f6c-7dd6-4e87-ac44-5374dd11c0dc\") " Oct 31 00:22:00.011968 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:22:00.011761 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61bb8f6c-7dd6-4e87-ac44-5374dd11c0dc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61bb8f6c-7dd6-4e87-ac44-5374dd11c0dc-istio-envoy\") pod \"61bb8f6c-7dd6-4e87-ac44-5374dd11c0dc\" (UID: \"61bb8f6c-7dd6-4e87-ac44-5374dd11c0dc\") " Oct 31 00:22:00.011968 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:22:00.011868 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61bb8f6c-7dd6-4e87-ac44-5374dd11c0dc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61bb8f6c-7dd6-4e87-ac44-5374dd11c0dc-credential-socket\") pod \"61bb8f6c-7dd6-4e87-ac44-5374dd11c0dc\" (UID: \"61bb8f6c-7dd6-4e87-ac44-5374dd11c0dc\") " Oct 31 00:22:00.011968 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:22:00.011947 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61bb8f6c-7dd6-4e87-ac44-5374dd11c0dc-kube-api-access-9ld68\" (UniqueName: \"kubernetes.io/projected/61bb8f6c-7dd6-4e87-ac44-5374dd11c0dc-kube-api-access-9ld68\") pod \"61bb8f6c-7dd6-4e87-ac44-5374dd11c0dc\" (UID: \"61bb8f6c-7dd6-4e87-ac44-5374dd11c0dc\") " Oct 31 00:22:00.012133 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:22:00.012006 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61bb8f6c-7dd6-4e87-ac44-5374dd11c0dc-workload-certs" pod "61bb8f6c-7dd6-4e87-ac44-5374dd11c0dc" (UID: "61bb8f6c-7dd6-4e87-ac44-5374dd11c0dc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:22:00.012133 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:22:00.012046 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61bb8f6c-7dd6-4e87-ac44-5374dd11c0dc-istiod-ca-cert" pod "61bb8f6c-7dd6-4e87-ac44-5374dd11c0dc" (UID: "61bb8f6c-7dd6-4e87-ac44-5374dd11c0dc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:22:00.012133 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:22:00.012063 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61bb8f6c-7dd6-4e87-ac44-5374dd11c0dc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61bb8f6c-7dd6-4e87-ac44-5374dd11c0dc-istio-data\") pod \"61bb8f6c-7dd6-4e87-ac44-5374dd11c0dc\" (UID: \"61bb8f6c-7dd6-4e87-ac44-5374dd11c0dc\") " Oct 31 00:22:00.012133 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:22:00.012125 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61bb8f6c-7dd6-4e87-ac44-5374dd11c0dc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61bb8f6c-7dd6-4e87-ac44-5374dd11c0dc-workload-socket\") pod \"61bb8f6c-7dd6-4e87-ac44-5374dd11c0dc\" (UID: \"61bb8f6c-7dd6-4e87-ac44-5374dd11c0dc\") " Oct 31 00:22:00.012380 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:22:00.012269 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61bb8f6c-7dd6-4e87-ac44-5374dd11c0dc-credential-socket" pod "61bb8f6c-7dd6-4e87-ac44-5374dd11c0dc" (UID: "61bb8f6c-7dd6-4e87-ac44-5374dd11c0dc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:22:00.012538 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:22:00.012514 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61bb8f6c-7dd6-4e87-ac44-5374dd11c0dc-istio-data" pod "61bb8f6c-7dd6-4e87-ac44-5374dd11c0dc" (UID: "61bb8f6c-7dd6-4e87-ac44-5374dd11c0dc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:22:00.012845 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:22:00.012703 4927 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61bb8f6c-7dd6-4e87-ac44-5374dd11c0dc-credential-socket\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:22:00.012845 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:22:00.012726 4927 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61bb8f6c-7dd6-4e87-ac44-5374dd11c0dc-istio-data\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:22:00.012845 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:22:00.012741 4927 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61bb8f6c-7dd6-4e87-ac44-5374dd11c0dc-istiod-ca-cert\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:22:00.012845 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:22:00.012756 4927 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61bb8f6c-7dd6-4e87-ac44-5374dd11c0dc-workload-certs\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:22:00.013130 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:22:00.012864 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61bb8f6c-7dd6-4e87-ac44-5374dd11c0dc-workload-socket" pod "61bb8f6c-7dd6-4e87-ac44-5374dd11c0dc" (UID: "61bb8f6c-7dd6-4e87-ac44-5374dd11c0dc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:22:00.015006 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:22:00.014970 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61bb8f6c-7dd6-4e87-ac44-5374dd11c0dc-kube-api-access-9ld68" pod "61bb8f6c-7dd6-4e87-ac44-5374dd11c0dc" (UID: "61bb8f6c-7dd6-4e87-ac44-5374dd11c0dc"). InnerVolumeSpecName "kube-api-access-9ld68". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:22:00.015125 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:22:00.015088 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61bb8f6c-7dd6-4e87-ac44-5374dd11c0dc-istio-envoy" pod "61bb8f6c-7dd6-4e87-ac44-5374dd11c0dc" (UID: "61bb8f6c-7dd6-4e87-ac44-5374dd11c0dc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:22:00.015667 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:22:00.015644 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/61bb8f6c-7dd6-4e87-ac44-5374dd11c0dc-istio-podinfo" pod "61bb8f6c-7dd6-4e87-ac44-5374dd11c0dc" (UID: "61bb8f6c-7dd6-4e87-ac44-5374dd11c0dc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:22:00.021574 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:22:00.021542 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61bb8f6c-7dd6-4e87-ac44-5374dd11c0dc-istio-token" pod "61bb8f6c-7dd6-4e87-ac44-5374dd11c0dc" (UID: "61bb8f6c-7dd6-4e87-ac44-5374dd11c0dc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:22:00.057902 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:22:00.057861 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-35a187/distinct-2-w6-35a187-kserve-7ff4b6dc55-944g6"] Oct 31 00:22:00.114800 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:22:00.114713 4927 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61bb8f6c-7dd6-4e87-ac44-5374dd11c0dc-workload-socket\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:22:00.115042 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:22:00.115010 4927 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61bb8f6c-7dd6-4e87-ac44-5374dd11c0dc-istio-token\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:22:00.115042 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:22:00.115042 4927 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61bb8f6c-7dd6-4e87-ac44-5374dd11c0dc-istio-podinfo\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:22:00.115256 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:22:00.115057 4927 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61bb8f6c-7dd6-4e87-ac44-5374dd11c0dc-istio-envoy\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:22:00.115256 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:22:00.115071 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9ld68\" (UniqueName: \"kubernetes.io/projected/61bb8f6c-7dd6-4e87-ac44-5374dd11c0dc-kube-api-access-9ld68\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:22:00.198746 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:22:00.198709 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:22:00.538219 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:22:00.538180 4927 generic.go:356] "Generic (PLEG): container finished" podID="61bb8f6c-7dd6-4e87-ac44-5374dd11c0dc" containerID="17083e75a5e87c8c1227a281fb6b74a9245389f2e1d42a190bbcaaad71dba081" exitCode=0 Oct 31 00:22:00.538477 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:22:00.538407 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-0d75be-openshift-default-57f99d9cbc-smttn" event={"ID":"61bb8f6c-7dd6-4e87-ac44-5374dd11c0dc","Type":"ContainerDied","Data":"17083e75a5e87c8c1227a281fb6b74a9245389f2e1d42a190bbcaaad71dba081"} Oct 31 00:22:00.538477 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:22:00.538465 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-0d75be-openshift-default-57f99d9cbc-smttn" event={"ID":"61bb8f6c-7dd6-4e87-ac44-5374dd11c0dc","Type":"ContainerDied","Data":"e53c66abfd5c9cbb206a1f340d71f045df7db48f695aa9448d089ba541d0e7f2"} Oct 31 00:22:00.538628 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:22:00.538500 4927 scope.go:122] "RemoveContainer" containerID="17083e75a5e87c8c1227a281fb6b74a9245389f2e1d42a190bbcaaad71dba081" Oct 31 00:22:00.538693 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:22:00.538666 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-0d75be-openshift-default-57f99d9cbc-smttn" Oct 31 00:22:00.551299 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:22:00.551260 4927 scope.go:122] "RemoveContainer" containerID="294e83eb65d8d48cedf6847168059337f49522ad8337e6d31732cabd4f76172e" Oct 31 00:22:00.561279 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:22:00.561259 4927 scope.go:122] "RemoveContainer" containerID="17083e75a5e87c8c1227a281fb6b74a9245389f2e1d42a190bbcaaad71dba081" Oct 31 00:22:00.561619 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:22:00.561599 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"17083e75a5e87c8c1227a281fb6b74a9245389f2e1d42a190bbcaaad71dba081\": container with ID starting with 17083e75a5e87c8c1227a281fb6b74a9245389f2e1d42a190bbcaaad71dba081 not found: ID does not exist" containerID="17083e75a5e87c8c1227a281fb6b74a9245389f2e1d42a190bbcaaad71dba081" Oct 31 00:22:00.561683 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:22:00.561628 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17083e75a5e87c8c1227a281fb6b74a9245389f2e1d42a190bbcaaad71dba081"} err="failed to get container status \"17083e75a5e87c8c1227a281fb6b74a9245389f2e1d42a190bbcaaad71dba081\": rpc error: code = NotFound desc = could not find container \"17083e75a5e87c8c1227a281fb6b74a9245389f2e1d42a190bbcaaad71dba081\": container with ID starting with 17083e75a5e87c8c1227a281fb6b74a9245389f2e1d42a190bbcaaad71dba081 not found: ID does not exist" Oct 31 00:22:00.561683 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:22:00.561645 4927 scope.go:122] "RemoveContainer" containerID="294e83eb65d8d48cedf6847168059337f49522ad8337e6d31732cabd4f76172e" Oct 31 00:22:00.561911 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:22:00.561889 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"294e83eb65d8d48cedf6847168059337f49522ad8337e6d31732cabd4f76172e\": container with ID starting with 294e83eb65d8d48cedf6847168059337f49522ad8337e6d31732cabd4f76172e not found: ID does not exist" containerID="294e83eb65d8d48cedf6847168059337f49522ad8337e6d31732cabd4f76172e" Oct 31 00:22:00.561963 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:22:00.561919 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"294e83eb65d8d48cedf6847168059337f49522ad8337e6d31732cabd4f76172e"} err="failed to get container status \"294e83eb65d8d48cedf6847168059337f49522ad8337e6d31732cabd4f76172e\": rpc error: code = NotFound desc = could not find container \"294e83eb65d8d48cedf6847168059337f49522ad8337e6d31732cabd4f76172e\": container with ID starting with 294e83eb65d8d48cedf6847168059337f49522ad8337e6d31732cabd4f76172e not found: ID does not exist" Oct 31 00:22:00.722245 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:22:00.722201 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-0d75be-openshift-default-57f99d9cbc-smttn"] Oct 31 00:22:00.772901 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:22:00.772867 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-0d75be-openshift-default-57f99d9cbc-smttn"] Oct 31 00:22:01.087646 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:22:01.087611 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61bb8f6c-7dd6-4e87-ac44-5374dd11c0dc" path="/var/lib/kubelet/pods/61bb8f6c-7dd6-4e87-ac44-5374dd11c0dc/volumes" Oct 31 00:22:02.119194 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:22:02.119154 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:22:03.936249 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:22:03.936180 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:22:04.392730 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:22:04.392653 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:22:04.942759 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:22:04.942729 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-37abc0-openshift-default-65b888884f-pqjfl"] Oct 31 00:22:04.943436 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:22:04.943421 4927 cpu_manager.go:414] "containerMap: removing container" podUID="61bb8f6c-7dd6-4e87-ac44-5374dd11c0dc" containerName="istio-proxy" Oct 31 00:22:04.943436 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:22:04.943436 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61bb8f6c-7dd6-4e87-ac44-5374dd11c0dc" containerName="istio-proxy" Oct 31 00:22:04.943539 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:22:04.943462 4927 cpu_manager.go:414] "containerMap: removing container" podUID="61bb8f6c-7dd6-4e87-ac44-5374dd11c0dc" containerName="istio-proxy" Oct 31 00:22:04.943539 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:22:04.943468 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61bb8f6c-7dd6-4e87-ac44-5374dd11c0dc" containerName="istio-proxy" Oct 31 00:22:04.943607 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:22:04.943565 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="61bb8f6c-7dd6-4e87-ac44-5374dd11c0dc" containerName="istio-proxy" Oct 31 00:22:04.943607 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:22:04.943577 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="61bb8f6c-7dd6-4e87-ac44-5374dd11c0dc" containerName="istio-proxy" Oct 31 00:22:04.948928 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:22:04.948908 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-37abc0-openshift-default-65b888884f-pqjfl" Oct 31 00:22:05.003192 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:22:05.002413 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84baa184-a53a-4f9f-8597-fd7c95e04d0d-credential-socket\") pod \"e2e-inf-b-37abc0-openshift-default-65b888884f-pqjfl\" (UID: \"84baa184-a53a-4f9f-8597-fd7c95e04d0d\") " pod="openshift-ingress/e2e-inf-b-37abc0-openshift-default-65b888884f-pqjfl" Oct 31 00:22:05.003192 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:22:05.002510 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84baa184-a53a-4f9f-8597-fd7c95e04d0d-istio-data\") pod \"e2e-inf-b-37abc0-openshift-default-65b888884f-pqjfl\" (UID: \"84baa184-a53a-4f9f-8597-fd7c95e04d0d\") " pod="openshift-ingress/e2e-inf-b-37abc0-openshift-default-65b888884f-pqjfl" Oct 31 00:22:05.003192 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:22:05.002605 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8tb6t\" (UniqueName: \"kubernetes.io/projected/84baa184-a53a-4f9f-8597-fd7c95e04d0d-kube-api-access-8tb6t\") pod \"e2e-inf-b-37abc0-openshift-default-65b888884f-pqjfl\" (UID: \"84baa184-a53a-4f9f-8597-fd7c95e04d0d\") " pod="openshift-ingress/e2e-inf-b-37abc0-openshift-default-65b888884f-pqjfl" Oct 31 00:22:05.003192 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:22:05.002729 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84baa184-a53a-4f9f-8597-fd7c95e04d0d-workload-socket\") pod \"e2e-inf-b-37abc0-openshift-default-65b888884f-pqjfl\" (UID: \"84baa184-a53a-4f9f-8597-fd7c95e04d0d\") " pod="openshift-ingress/e2e-inf-b-37abc0-openshift-default-65b888884f-pqjfl" Oct 31 00:22:05.003192 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:22:05.002765 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84baa184-a53a-4f9f-8597-fd7c95e04d0d-istiod-ca-cert\") pod \"e2e-inf-b-37abc0-openshift-default-65b888884f-pqjfl\" (UID: \"84baa184-a53a-4f9f-8597-fd7c95e04d0d\") " pod="openshift-ingress/e2e-inf-b-37abc0-openshift-default-65b888884f-pqjfl" Oct 31 00:22:05.003192 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:22:05.002837 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84baa184-a53a-4f9f-8597-fd7c95e04d0d-istio-envoy\") pod \"e2e-inf-b-37abc0-openshift-default-65b888884f-pqjfl\" (UID: \"84baa184-a53a-4f9f-8597-fd7c95e04d0d\") " pod="openshift-ingress/e2e-inf-b-37abc0-openshift-default-65b888884f-pqjfl" Oct 31 00:22:05.003192 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:22:05.002881 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84baa184-a53a-4f9f-8597-fd7c95e04d0d-istio-podinfo\") pod \"e2e-inf-b-37abc0-openshift-default-65b888884f-pqjfl\" (UID: \"84baa184-a53a-4f9f-8597-fd7c95e04d0d\") " pod="openshift-ingress/e2e-inf-b-37abc0-openshift-default-65b888884f-pqjfl" Oct 31 00:22:05.003192 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:22:05.002925 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84baa184-a53a-4f9f-8597-fd7c95e04d0d-istio-token\") pod \"e2e-inf-b-37abc0-openshift-default-65b888884f-pqjfl\" (UID: \"84baa184-a53a-4f9f-8597-fd7c95e04d0d\") " pod="openshift-ingress/e2e-inf-b-37abc0-openshift-default-65b888884f-pqjfl" Oct 31 00:22:05.003192 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:22:05.002954 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84baa184-a53a-4f9f-8597-fd7c95e04d0d-workload-certs\") pod \"e2e-inf-b-37abc0-openshift-default-65b888884f-pqjfl\" (UID: \"84baa184-a53a-4f9f-8597-fd7c95e04d0d\") " pod="openshift-ingress/e2e-inf-b-37abc0-openshift-default-65b888884f-pqjfl" Oct 31 00:22:05.043690 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:22:05.043654 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-37abc0-openshift-default-dockercfg-xq2rm\"" Oct 31 00:22:05.083722 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:22:05.077446 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:22:05.105734 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:22:05.104159 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84baa184-a53a-4f9f-8597-fd7c95e04d0d-workload-socket\") pod \"e2e-inf-b-37abc0-openshift-default-65b888884f-pqjfl\" (UID: \"84baa184-a53a-4f9f-8597-fd7c95e04d0d\") " pod="openshift-ingress/e2e-inf-b-37abc0-openshift-default-65b888884f-pqjfl" Oct 31 00:22:05.105734 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:22:05.104215 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84baa184-a53a-4f9f-8597-fd7c95e04d0d-istiod-ca-cert\") pod \"e2e-inf-b-37abc0-openshift-default-65b888884f-pqjfl\" (UID: \"84baa184-a53a-4f9f-8597-fd7c95e04d0d\") " pod="openshift-ingress/e2e-inf-b-37abc0-openshift-default-65b888884f-pqjfl" Oct 31 00:22:05.105734 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:22:05.104279 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84baa184-a53a-4f9f-8597-fd7c95e04d0d-istio-envoy\") pod \"e2e-inf-b-37abc0-openshift-default-65b888884f-pqjfl\" (UID: \"84baa184-a53a-4f9f-8597-fd7c95e04d0d\") " pod="openshift-ingress/e2e-inf-b-37abc0-openshift-default-65b888884f-pqjfl" Oct 31 00:22:05.105734 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:22:05.104357 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84baa184-a53a-4f9f-8597-fd7c95e04d0d-istio-podinfo\") pod \"e2e-inf-b-37abc0-openshift-default-65b888884f-pqjfl\" (UID: \"84baa184-a53a-4f9f-8597-fd7c95e04d0d\") " pod="openshift-ingress/e2e-inf-b-37abc0-openshift-default-65b888884f-pqjfl" Oct 31 00:22:05.105734 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:22:05.104402 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84baa184-a53a-4f9f-8597-fd7c95e04d0d-istio-token\") pod \"e2e-inf-b-37abc0-openshift-default-65b888884f-pqjfl\" (UID: \"84baa184-a53a-4f9f-8597-fd7c95e04d0d\") " pod="openshift-ingress/e2e-inf-b-37abc0-openshift-default-65b888884f-pqjfl" Oct 31 00:22:05.105734 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:22:05.104430 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84baa184-a53a-4f9f-8597-fd7c95e04d0d-workload-certs\") pod \"e2e-inf-b-37abc0-openshift-default-65b888884f-pqjfl\" (UID: \"84baa184-a53a-4f9f-8597-fd7c95e04d0d\") " pod="openshift-ingress/e2e-inf-b-37abc0-openshift-default-65b888884f-pqjfl" Oct 31 00:22:05.105734 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:22:05.104482 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84baa184-a53a-4f9f-8597-fd7c95e04d0d-credential-socket\") pod \"e2e-inf-b-37abc0-openshift-default-65b888884f-pqjfl\" (UID: \"84baa184-a53a-4f9f-8597-fd7c95e04d0d\") " pod="openshift-ingress/e2e-inf-b-37abc0-openshift-default-65b888884f-pqjfl" Oct 31 00:22:05.105734 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:22:05.104554 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84baa184-a53a-4f9f-8597-fd7c95e04d0d-istio-data\") pod \"e2e-inf-b-37abc0-openshift-default-65b888884f-pqjfl\" (UID: \"84baa184-a53a-4f9f-8597-fd7c95e04d0d\") " pod="openshift-ingress/e2e-inf-b-37abc0-openshift-default-65b888884f-pqjfl" Oct 31 00:22:05.108924 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:22:05.106428 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8tb6t\" (UniqueName: \"kubernetes.io/projected/84baa184-a53a-4f9f-8597-fd7c95e04d0d-kube-api-access-8tb6t\") pod \"e2e-inf-b-37abc0-openshift-default-65b888884f-pqjfl\" (UID: \"84baa184-a53a-4f9f-8597-fd7c95e04d0d\") " pod="openshift-ingress/e2e-inf-b-37abc0-openshift-default-65b888884f-pqjfl" Oct 31 00:22:05.108924 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:22:05.107473 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84baa184-a53a-4f9f-8597-fd7c95e04d0d-workload-socket\") pod \"e2e-inf-b-37abc0-openshift-default-65b888884f-pqjfl\" (UID: \"84baa184-a53a-4f9f-8597-fd7c95e04d0d\") " pod="openshift-ingress/e2e-inf-b-37abc0-openshift-default-65b888884f-pqjfl" Oct 31 00:22:05.108924 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:22:05.108162 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84baa184-a53a-4f9f-8597-fd7c95e04d0d-istiod-ca-cert\") pod \"e2e-inf-b-37abc0-openshift-default-65b888884f-pqjfl\" (UID: \"84baa184-a53a-4f9f-8597-fd7c95e04d0d\") " pod="openshift-ingress/e2e-inf-b-37abc0-openshift-default-65b888884f-pqjfl" Oct 31 00:22:05.114134 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:22:05.114099 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84baa184-a53a-4f9f-8597-fd7c95e04d0d-workload-certs\") pod \"e2e-inf-b-37abc0-openshift-default-65b888884f-pqjfl\" (UID: \"84baa184-a53a-4f9f-8597-fd7c95e04d0d\") " pod="openshift-ingress/e2e-inf-b-37abc0-openshift-default-65b888884f-pqjfl" Oct 31 00:22:05.149905 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:22:05.143077 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84baa184-a53a-4f9f-8597-fd7c95e04d0d-credential-socket\") pod \"e2e-inf-b-37abc0-openshift-default-65b888884f-pqjfl\" (UID: \"84baa184-a53a-4f9f-8597-fd7c95e04d0d\") " pod="openshift-ingress/e2e-inf-b-37abc0-openshift-default-65b888884f-pqjfl" Oct 31 00:22:05.149905 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:22:05.143724 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84baa184-a53a-4f9f-8597-fd7c95e04d0d-istio-data\") pod \"e2e-inf-b-37abc0-openshift-default-65b888884f-pqjfl\" (UID: \"84baa184-a53a-4f9f-8597-fd7c95e04d0d\") " pod="openshift-ingress/e2e-inf-b-37abc0-openshift-default-65b888884f-pqjfl" Oct 31 00:22:05.151069 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:22:05.151013 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84baa184-a53a-4f9f-8597-fd7c95e04d0d-istio-podinfo\") pod \"e2e-inf-b-37abc0-openshift-default-65b888884f-pqjfl\" (UID: \"84baa184-a53a-4f9f-8597-fd7c95e04d0d\") " pod="openshift-ingress/e2e-inf-b-37abc0-openshift-default-65b888884f-pqjfl" Oct 31 00:22:05.183236 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:22:05.183083 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-37abc0-openshift-default-65b888884f-pqjfl"] Oct 31 00:22:05.208109 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:22:05.208006 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8tb6t\" (UniqueName: \"kubernetes.io/projected/84baa184-a53a-4f9f-8597-fd7c95e04d0d-kube-api-access-8tb6t\") pod \"e2e-inf-b-37abc0-openshift-default-65b888884f-pqjfl\" (UID: \"84baa184-a53a-4f9f-8597-fd7c95e04d0d\") " pod="openshift-ingress/e2e-inf-b-37abc0-openshift-default-65b888884f-pqjfl" Oct 31 00:22:05.264689 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:22:05.264465 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84baa184-a53a-4f9f-8597-fd7c95e04d0d-istio-envoy\") pod \"e2e-inf-b-37abc0-openshift-default-65b888884f-pqjfl\" (UID: \"84baa184-a53a-4f9f-8597-fd7c95e04d0d\") " pod="openshift-ingress/e2e-inf-b-37abc0-openshift-default-65b888884f-pqjfl" Oct 31 00:22:05.298835 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:22:05.298050 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84baa184-a53a-4f9f-8597-fd7c95e04d0d-istio-token\") pod \"e2e-inf-b-37abc0-openshift-default-65b888884f-pqjfl\" (UID: \"84baa184-a53a-4f9f-8597-fd7c95e04d0d\") " pod="openshift-ingress/e2e-inf-b-37abc0-openshift-default-65b888884f-pqjfl" Oct 31 00:22:05.389421 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:22:05.388074 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:22:05.575775 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:22:05.575676 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-37abc0-openshift-default-65b888884f-pqjfl" Oct 31 00:22:06.022945 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:22:06.022915 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-37abc0-openshift-default-65b888884f-pqjfl"] Oct 31 00:22:06.024227 ip-10-0-141-180 kubenswrapper[4927]: W1031 00:22:06.024197 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84baa184_a53a_4f9f_8597_fd7c95e04d0d.slice/crio-2d079763b5f56d836caf12656cc34019e9d9d0e3df1d94081507ace72c9f18c3 WatchSource:0}: Error finding container 2d079763b5f56d836caf12656cc34019e9d9d0e3df1d94081507ace72c9f18c3: Status 404 returned error can't find the container with id 2d079763b5f56d836caf12656cc34019e9d9d0e3df1d94081507ace72c9f18c3 Oct 31 00:22:06.027019 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:22:06.026982 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:22:06.027107 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:22:06.027058 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:22:06.027107 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:22:06.027091 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:22:06.592151 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:22:06.592117 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-37abc0-openshift-default-65b888884f-pqjfl" event={"ID":"84baa184-a53a-4f9f-8597-fd7c95e04d0d","Type":"ContainerStarted","Data":"15a2819b990fb45280a7b8b986029e7fb518a58ba0c3030b8b70906d3f091550"} Oct 31 00:22:06.592357 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:22:06.592169 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-37abc0-openshift-default-65b888884f-pqjfl" event={"ID":"84baa184-a53a-4f9f-8597-fd7c95e04d0d","Type":"ContainerStarted","Data":"2d079763b5f56d836caf12656cc34019e9d9d0e3df1d94081507ace72c9f18c3"} Oct 31 00:22:06.758317 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:22:06.758266 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-37abc0-openshift-default-65b888884f-pqjfl" podStartSLOduration=2.75825109 podStartE2EDuration="2.75825109s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:22:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:22:06.754164409 +0000 UTC m=+2439.790495231" watchObservedRunningTime="2026-10-31 00:22:06.75825109 +0000 UTC m=+2439.794581912" Oct 31 00:22:07.140782 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:22:07.140740 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:22:07.205355 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:22:07.201806 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-35a187-7887c98dff-5h96p"] Oct 31 00:22:07.435784 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:22:07.435738 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:22:07.593378 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:22:07.586891 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-37abc0-openshift-default-65b888884f-pqjfl" Oct 31 00:22:07.617354 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:22:07.603070 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:22:07.622351 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:22:07.619618 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-37abc0-openshift-default-65b888884f-pqjfl" Oct 31 00:22:08.623227 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:22:08.623189 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-37abc0-openshift-default-65b888884f-pqjfl" Oct 31 00:22:08.624377 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:22:08.624323 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-37abc0-openshift-default-65b888884f-pqjfl" Oct 31 00:22:09.605969 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:22:09.605927 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:22:09.818254 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:22:09.818221 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:22:12.203270 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:22:12.203184 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-35a187-7887c98dff-5h96p" podUID="2b69ba41-3647-4827-9acc-8f00b05d299a" containerName="payload-pre-processing" containerID="cri-o://007f441c941b9c34c2417a05dde72d89ab751077d63f5cef4d19c802ede34a78" gracePeriod=25 Oct 31 00:22:12.427345 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:22:12.427302 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:22:13.375414 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:22:13.375382 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:22:13.592391 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:22:13.592323 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:22:14.043638 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:22:14.043599 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-35a187-7887c98dff-5h96p" podUID="2b69ba41-3647-4827-9acc-8f00b05d299a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 00:22:15.085565 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:22:15.085501 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-35a187/distinct-2-w6-35a187-kserve-7ff4b6dc55-944g6" podUID="429ee092-42d8-475e-b0c6-b5a9930766fc" containerName="main" containerID="cri-o://395cd9912d86ee5a63a52e819da9b8d1baf7ae77f4add38912ff14a53f428ec1" gracePeriod=45 Oct 31 00:22:15.101940 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:22:15.101905 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-35a187/distinct-2-w6-35a187-kserve-7ff4b6dc55-944g6" podUID="429ee092-42d8-475e-b0c6-b5a9930766fc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.113:8000/ready\": dial tcp 10.132.0.113:8000: connect: connection refused" Oct 31 00:22:15.379312 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:22:15.379284 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-35a187/distinct-2-w6-35a187-kserve-7ff4b6dc55-944g6" Oct 31 00:22:15.462103 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:22:15.462070 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/429ee092-42d8-475e-b0c6-b5a9930766fc-dshm\" (UniqueName: \"kubernetes.io/empty-dir/429ee092-42d8-475e-b0c6-b5a9930766fc-dshm\") pod \"429ee092-42d8-475e-b0c6-b5a9930766fc\" (UID: \"429ee092-42d8-475e-b0c6-b5a9930766fc\") " Oct 31 00:22:15.462281 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:22:15.462117 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/429ee092-42d8-475e-b0c6-b5a9930766fc-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/429ee092-42d8-475e-b0c6-b5a9930766fc-tmp-dir\") pod \"429ee092-42d8-475e-b0c6-b5a9930766fc\" (UID: \"429ee092-42d8-475e-b0c6-b5a9930766fc\") " Oct 31 00:22:15.462281 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:22:15.462205 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/429ee092-42d8-475e-b0c6-b5a9930766fc-kube-api-access-rkn2x\" (UniqueName: \"kubernetes.io/projected/429ee092-42d8-475e-b0c6-b5a9930766fc-kube-api-access-rkn2x\") pod \"429ee092-42d8-475e-b0c6-b5a9930766fc\" (UID: \"429ee092-42d8-475e-b0c6-b5a9930766fc\") " Oct 31 00:22:15.462281 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:22:15.462261 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/429ee092-42d8-475e-b0c6-b5a9930766fc-home\" (UniqueName: \"kubernetes.io/empty-dir/429ee092-42d8-475e-b0c6-b5a9930766fc-home\") pod \"429ee092-42d8-475e-b0c6-b5a9930766fc\" (UID: \"429ee092-42d8-475e-b0c6-b5a9930766fc\") " Oct 31 00:22:15.462439 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:22:15.462396 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/429ee092-42d8-475e-b0c6-b5a9930766fc-tls-certs\" (UniqueName: \"kubernetes.io/secret/429ee092-42d8-475e-b0c6-b5a9930766fc-tls-certs\") pod \"429ee092-42d8-475e-b0c6-b5a9930766fc\" (UID: \"429ee092-42d8-475e-b0c6-b5a9930766fc\") " Oct 31 00:22:15.462439 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:22:15.462405 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/429ee092-42d8-475e-b0c6-b5a9930766fc-tmp-dir" pod "429ee092-42d8-475e-b0c6-b5a9930766fc" (UID: "429ee092-42d8-475e-b0c6-b5a9930766fc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:22:15.462558 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:22:15.462457 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/429ee092-42d8-475e-b0c6-b5a9930766fc-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/429ee092-42d8-475e-b0c6-b5a9930766fc-model-cache\") pod \"429ee092-42d8-475e-b0c6-b5a9930766fc\" (UID: \"429ee092-42d8-475e-b0c6-b5a9930766fc\") " Oct 31 00:22:15.462621 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:22:15.462568 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/429ee092-42d8-475e-b0c6-b5a9930766fc-home" pod "429ee092-42d8-475e-b0c6-b5a9930766fc" (UID: "429ee092-42d8-475e-b0c6-b5a9930766fc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:22:15.462763 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:22:15.462729 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/429ee092-42d8-475e-b0c6-b5a9930766fc-model-cache" pod "429ee092-42d8-475e-b0c6-b5a9930766fc" (UID: "429ee092-42d8-475e-b0c6-b5a9930766fc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:22:15.463009 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:22:15.462990 4927 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/429ee092-42d8-475e-b0c6-b5a9930766fc-model-cache\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:22:15.463073 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:22:15.463013 4927 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/429ee092-42d8-475e-b0c6-b5a9930766fc-tmp-dir\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:22:15.463073 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:22:15.463022 4927 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/429ee092-42d8-475e-b0c6-b5a9930766fc-home\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:22:15.464708 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:22:15.464670 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/429ee092-42d8-475e-b0c6-b5a9930766fc-tls-certs" pod "429ee092-42d8-475e-b0c6-b5a9930766fc" (UID: "429ee092-42d8-475e-b0c6-b5a9930766fc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:22:15.464796 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:22:15.464748 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/429ee092-42d8-475e-b0c6-b5a9930766fc-dshm" pod "429ee092-42d8-475e-b0c6-b5a9930766fc" (UID: "429ee092-42d8-475e-b0c6-b5a9930766fc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:22:15.464796 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:22:15.464757 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/429ee092-42d8-475e-b0c6-b5a9930766fc-kube-api-access-rkn2x" pod "429ee092-42d8-475e-b0c6-b5a9930766fc" (UID: "429ee092-42d8-475e-b0c6-b5a9930766fc"). InnerVolumeSpecName "kube-api-access-rkn2x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:22:15.564107 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:22:15.564067 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rkn2x\" (UniqueName: \"kubernetes.io/projected/429ee092-42d8-475e-b0c6-b5a9930766fc-kube-api-access-rkn2x\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:22:15.564107 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:22:15.564099 4927 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/429ee092-42d8-475e-b0c6-b5a9930766fc-tls-certs\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:22:15.564107 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:22:15.564112 4927 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/429ee092-42d8-475e-b0c6-b5a9930766fc-dshm\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:22:15.691113 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:22:15.691081 4927 generic.go:356] "Generic (PLEG): container finished" podID="429ee092-42d8-475e-b0c6-b5a9930766fc" containerID="395cd9912d86ee5a63a52e819da9b8d1baf7ae77f4add38912ff14a53f428ec1" exitCode=0 Oct 31 00:22:15.691302 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:22:15.691125 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-35a187/distinct-2-w6-35a187-kserve-7ff4b6dc55-944g6" Oct 31 00:22:15.691302 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:22:15.691165 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-35a187/distinct-2-w6-35a187-kserve-7ff4b6dc55-944g6" event={"ID":"429ee092-42d8-475e-b0c6-b5a9930766fc","Type":"ContainerDied","Data":"395cd9912d86ee5a63a52e819da9b8d1baf7ae77f4add38912ff14a53f428ec1"} Oct 31 00:22:15.691302 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:22:15.691217 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-35a187/distinct-2-w6-35a187-kserve-7ff4b6dc55-944g6" event={"ID":"429ee092-42d8-475e-b0c6-b5a9930766fc","Type":"ContainerDied","Data":"c940816fcfa592cfa2a3b0fe41de09398a92f0c33cb06e601a0535d40dcd1dd1"} Oct 31 00:22:15.691302 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:22:15.691240 4927 scope.go:122] "RemoveContainer" containerID="395cd9912d86ee5a63a52e819da9b8d1baf7ae77f4add38912ff14a53f428ec1" Oct 31 00:22:15.703886 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:22:15.703867 4927 scope.go:122] "RemoveContainer" containerID="395cd9912d86ee5a63a52e819da9b8d1baf7ae77f4add38912ff14a53f428ec1" Oct 31 00:22:15.704152 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:22:15.704133 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"395cd9912d86ee5a63a52e819da9b8d1baf7ae77f4add38912ff14a53f428ec1\": container with ID starting with 395cd9912d86ee5a63a52e819da9b8d1baf7ae77f4add38912ff14a53f428ec1 not found: ID does not exist" containerID="395cd9912d86ee5a63a52e819da9b8d1baf7ae77f4add38912ff14a53f428ec1" Oct 31 00:22:15.704199 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:22:15.704158 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"395cd9912d86ee5a63a52e819da9b8d1baf7ae77f4add38912ff14a53f428ec1"} err="failed to get container status \"395cd9912d86ee5a63a52e819da9b8d1baf7ae77f4add38912ff14a53f428ec1\": rpc error: code = NotFound desc = could not find container \"395cd9912d86ee5a63a52e819da9b8d1baf7ae77f4add38912ff14a53f428ec1\": container with ID starting with 395cd9912d86ee5a63a52e819da9b8d1baf7ae77f4add38912ff14a53f428ec1 not found: ID does not exist" Oct 31 00:22:15.863057 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:22:15.863021 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-35a187/distinct-2-w6-35a187-kserve-7ff4b6dc55-944g6"] Oct 31 00:22:15.869157 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:22:15.869119 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-35a187/distinct-2-w6-35a187-kserve-7ff4b6dc55-944g6"] Oct 31 00:22:17.090438 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:22:17.090405 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="429ee092-42d8-475e-b0c6-b5a9930766fc" path="/var/lib/kubelet/pods/429ee092-42d8-475e-b0c6-b5a9930766fc/volumes" Oct 31 00:22:17.426344 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:22:17.426307 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:22:19.684224 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:22:19.684181 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sqv69" containerName="extractor" Oct 31 00:22:21.852068 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:22:21.852012 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:22:23.382481 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:22:23.382447 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:22:23.636576 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:22:23.636485 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:22:23.788958 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:22:23.788926 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:22:23.902091 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:22:23.901987 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-35a187-7887c98dff-5h96p" podUID="2b69ba41-3647-4827-9acc-8f00b05d299a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 00:22:24.324704 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:22:24.324662 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:22:27.668560 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:22:27.668527 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:22:27.825013 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:22:27.824974 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:22:28.664419 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:22:28.658063 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:22:31.017128 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:22:31.017093 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:22:31.234533 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:22:31.234441 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:22:31.403274 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:22:31.403184 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-d44pq"] Oct 31 00:22:31.403476 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:22:31.403440 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-d44pq" podUID="f08dd946-14b1-4546-8fae-01564f3c6908" containerName="istio-proxy" containerID="cri-o://e3b3bb0f5eb795718da9b14e5db0c35736c935970bc957b86792314b0c63c301" gracePeriod=30 Oct 31 00:22:31.436958 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:22:31.436922 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:22:32.101456 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:22:32.101418 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:22:32.410295 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:22:32.410258 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:22:32.486855 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:22:32.486822 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-35a187-7887c98dff-5h96p" Oct 31 00:22:32.637581 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:22:32.637494 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2b69ba41-3647-4827-9acc-8f00b05d299a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b69ba41-3647-4827-9acc-8f00b05d299a-plugins-config-volume\") pod \"2b69ba41-3647-4827-9acc-8f00b05d299a\" (UID: \"2b69ba41-3647-4827-9acc-8f00b05d299a\") " Oct 31 00:22:32.637737 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:22:32.637706 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b69ba41-3647-4827-9acc-8f00b05d299a-kube-api-access-d7v8k\" (UniqueName: \"kubernetes.io/projected/2b69ba41-3647-4827-9acc-8f00b05d299a-kube-api-access-d7v8k\") pod \"2b69ba41-3647-4827-9acc-8f00b05d299a\" (UID: \"2b69ba41-3647-4827-9acc-8f00b05d299a\") " Oct 31 00:22:32.638226 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:22:32.638193 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b69ba41-3647-4827-9acc-8f00b05d299a-plugins-config-volume" pod "2b69ba41-3647-4827-9acc-8f00b05d299a" (UID: "2b69ba41-3647-4827-9acc-8f00b05d299a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:22:32.640184 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:22:32.640156 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b69ba41-3647-4827-9acc-8f00b05d299a-kube-api-access-d7v8k" pod "2b69ba41-3647-4827-9acc-8f00b05d299a" (UID: "2b69ba41-3647-4827-9acc-8f00b05d299a"). InnerVolumeSpecName "kube-api-access-d7v8k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:22:32.740000 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:22:32.739964 4927 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b69ba41-3647-4827-9acc-8f00b05d299a-plugins-config-volume\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:22:32.740000 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:22:32.739995 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d7v8k\" (UniqueName: \"kubernetes.io/projected/2b69ba41-3647-4827-9acc-8f00b05d299a-kube-api-access-d7v8k\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:22:32.861416 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:22:32.861381 4927 generic.go:356] "Generic (PLEG): container finished" podID="2b69ba41-3647-4827-9acc-8f00b05d299a" containerID="007f441c941b9c34c2417a05dde72d89ab751077d63f5cef4d19c802ede34a78" exitCode=0 Oct 31 00:22:32.861588 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:22:32.861426 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-35a187-7887c98dff-5h96p" Oct 31 00:22:32.861588 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:22:32.861464 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-35a187-7887c98dff-5h96p" event={"ID":"2b69ba41-3647-4827-9acc-8f00b05d299a","Type":"ContainerDied","Data":"007f441c941b9c34c2417a05dde72d89ab751077d63f5cef4d19c802ede34a78"} Oct 31 00:22:32.861588 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:22:32.861516 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-35a187-7887c98dff-5h96p" event={"ID":"2b69ba41-3647-4827-9acc-8f00b05d299a","Type":"ContainerDied","Data":"e9f0c3286c8a3ec29795f18bf914dffece828606fbea756ec3a21fa1410351c2"} Oct 31 00:22:32.861588 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:22:32.861539 4927 scope.go:122] "RemoveContainer" containerID="007f441c941b9c34c2417a05dde72d89ab751077d63f5cef4d19c802ede34a78" Oct 31 00:22:32.872643 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:22:32.872626 4927 scope.go:122] "RemoveContainer" containerID="007f441c941b9c34c2417a05dde72d89ab751077d63f5cef4d19c802ede34a78" Oct 31 00:22:32.872896 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:22:32.872878 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"007f441c941b9c34c2417a05dde72d89ab751077d63f5cef4d19c802ede34a78\": container with ID starting with 007f441c941b9c34c2417a05dde72d89ab751077d63f5cef4d19c802ede34a78 not found: ID does not exist" containerID="007f441c941b9c34c2417a05dde72d89ab751077d63f5cef4d19c802ede34a78" Oct 31 00:22:32.872938 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:22:32.872906 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"007f441c941b9c34c2417a05dde72d89ab751077d63f5cef4d19c802ede34a78"} err="failed to get container status \"007f441c941b9c34c2417a05dde72d89ab751077d63f5cef4d19c802ede34a78\": rpc error: code = NotFound desc = could not find container \"007f441c941b9c34c2417a05dde72d89ab751077d63f5cef4d19c802ede34a78\": container with ID starting with 007f441c941b9c34c2417a05dde72d89ab751077d63f5cef4d19c802ede34a78 not found: ID does not exist" Oct 31 00:22:33.032506 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:22:33.032474 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-35a187-7887c98dff-5h96p"] Oct 31 00:22:33.036569 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:22:33.036542 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-35a187-7887c98dff-5h96p"] Oct 31 00:22:33.087524 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:22:33.087491 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b69ba41-3647-4827-9acc-8f00b05d299a" path="/var/lib/kubelet/pods/2b69ba41-3647-4827-9acc-8f00b05d299a/volumes" Oct 31 00:22:33.382670 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:22:33.382593 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:22:33.543248 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:22:33.543211 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:22:33.696958 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:22:33.696924 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:22:36.802876 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:22:36.802847 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-d44pq" Oct 31 00:22:36.896501 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:22:36.896461 4927 generic.go:356] "Generic (PLEG): container finished" podID="f08dd946-14b1-4546-8fae-01564f3c6908" containerID="e3b3bb0f5eb795718da9b14e5db0c35736c935970bc957b86792314b0c63c301" exitCode=0 Oct 31 00:22:36.896699 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:22:36.896623 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-d44pq" event={"ID":"f08dd946-14b1-4546-8fae-01564f3c6908","Type":"ContainerDied","Data":"e3b3bb0f5eb795718da9b14e5db0c35736c935970bc957b86792314b0c63c301"} Oct 31 00:22:36.896699 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:22:36.896673 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-d44pq" event={"ID":"f08dd946-14b1-4546-8fae-01564f3c6908","Type":"ContainerDied","Data":"5538873b02928199d71fb3aa254fb440d83f678402b4489fcbc65810ce66b646"} Oct 31 00:22:36.896849 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:22:36.896701 4927 scope.go:122] "RemoveContainer" containerID="e3b3bb0f5eb795718da9b14e5db0c35736c935970bc957b86792314b0c63c301" Oct 31 00:22:36.896849 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:22:36.896846 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-d44pq" Oct 31 00:22:36.910962 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:22:36.910876 4927 scope.go:122] "RemoveContainer" containerID="e3b3bb0f5eb795718da9b14e5db0c35736c935970bc957b86792314b0c63c301" Oct 31 00:22:36.911261 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:22:36.911219 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3b3bb0f5eb795718da9b14e5db0c35736c935970bc957b86792314b0c63c301\": container with ID starting with e3b3bb0f5eb795718da9b14e5db0c35736c935970bc957b86792314b0c63c301 not found: ID does not exist" containerID="e3b3bb0f5eb795718da9b14e5db0c35736c935970bc957b86792314b0c63c301" Oct 31 00:22:36.911418 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:22:36.911260 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3b3bb0f5eb795718da9b14e5db0c35736c935970bc957b86792314b0c63c301"} err="failed to get container status \"e3b3bb0f5eb795718da9b14e5db0c35736c935970bc957b86792314b0c63c301\": rpc error: code = NotFound desc = could not find container \"e3b3bb0f5eb795718da9b14e5db0c35736c935970bc957b86792314b0c63c301\": container with ID starting with e3b3bb0f5eb795718da9b14e5db0c35736c935970bc957b86792314b0c63c301 not found: ID does not exist" Oct 31 00:22:37.000095 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:22:36.999136 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f08dd946-14b1-4546-8fae-01564f3c6908-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f08dd946-14b1-4546-8fae-01564f3c6908-istio-podinfo\") pod \"f08dd946-14b1-4546-8fae-01564f3c6908\" (UID: \"f08dd946-14b1-4546-8fae-01564f3c6908\") " Oct 31 00:22:37.000095 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:22:36.999206 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f08dd946-14b1-4546-8fae-01564f3c6908-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f08dd946-14b1-4546-8fae-01564f3c6908-istiod-ca-cert\") pod \"f08dd946-14b1-4546-8fae-01564f3c6908\" (UID: \"f08dd946-14b1-4546-8fae-01564f3c6908\") " Oct 31 00:22:37.000095 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:22:36.999235 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f08dd946-14b1-4546-8fae-01564f3c6908-kube-api-access-jf9bt\" (UniqueName: \"kubernetes.io/projected/f08dd946-14b1-4546-8fae-01564f3c6908-kube-api-access-jf9bt\") pod \"f08dd946-14b1-4546-8fae-01564f3c6908\" (UID: \"f08dd946-14b1-4546-8fae-01564f3c6908\") " Oct 31 00:22:37.000095 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:22:36.999267 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f08dd946-14b1-4546-8fae-01564f3c6908-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f08dd946-14b1-4546-8fae-01564f3c6908-workload-certs\") pod \"f08dd946-14b1-4546-8fae-01564f3c6908\" (UID: \"f08dd946-14b1-4546-8fae-01564f3c6908\") " Oct 31 00:22:37.000095 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:22:36.999287 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f08dd946-14b1-4546-8fae-01564f3c6908-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f08dd946-14b1-4546-8fae-01564f3c6908-istio-data\") pod \"f08dd946-14b1-4546-8fae-01564f3c6908\" (UID: \"f08dd946-14b1-4546-8fae-01564f3c6908\") " Oct 31 00:22:37.000095 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:22:36.999437 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f08dd946-14b1-4546-8fae-01564f3c6908-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f08dd946-14b1-4546-8fae-01564f3c6908-istio-envoy\") pod \"f08dd946-14b1-4546-8fae-01564f3c6908\" (UID: \"f08dd946-14b1-4546-8fae-01564f3c6908\") " Oct 31 00:22:37.000095 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:22:36.999476 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f08dd946-14b1-4546-8fae-01564f3c6908-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f08dd946-14b1-4546-8fae-01564f3c6908-workload-socket\") pod \"f08dd946-14b1-4546-8fae-01564f3c6908\" (UID: \"f08dd946-14b1-4546-8fae-01564f3c6908\") " Oct 31 00:22:37.000095 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:22:36.999539 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f08dd946-14b1-4546-8fae-01564f3c6908-istio-token\" (UniqueName: \"kubernetes.io/projected/f08dd946-14b1-4546-8fae-01564f3c6908-istio-token\") pod \"f08dd946-14b1-4546-8fae-01564f3c6908\" (UID: \"f08dd946-14b1-4546-8fae-01564f3c6908\") " Oct 31 00:22:37.000095 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:22:36.999615 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f08dd946-14b1-4546-8fae-01564f3c6908-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f08dd946-14b1-4546-8fae-01564f3c6908-credential-socket\") pod \"f08dd946-14b1-4546-8fae-01564f3c6908\" (UID: \"f08dd946-14b1-4546-8fae-01564f3c6908\") " Oct 31 00:22:37.000095 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:22:36.999910 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f08dd946-14b1-4546-8fae-01564f3c6908-workload-certs" pod "f08dd946-14b1-4546-8fae-01564f3c6908" (UID: "f08dd946-14b1-4546-8fae-01564f3c6908"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:22:37.000843 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:22:37.000111 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f08dd946-14b1-4546-8fae-01564f3c6908-workload-socket" pod "f08dd946-14b1-4546-8fae-01564f3c6908" (UID: "f08dd946-14b1-4546-8fae-01564f3c6908"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:22:37.000843 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:22:37.000133 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f08dd946-14b1-4546-8fae-01564f3c6908-istio-data" pod "f08dd946-14b1-4546-8fae-01564f3c6908" (UID: "f08dd946-14b1-4546-8fae-01564f3c6908"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:22:37.000843 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:22:37.000356 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f08dd946-14b1-4546-8fae-01564f3c6908-istiod-ca-cert" pod "f08dd946-14b1-4546-8fae-01564f3c6908" (UID: "f08dd946-14b1-4546-8fae-01564f3c6908"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:22:37.000843 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:22:37.000539 4927 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f08dd946-14b1-4546-8fae-01564f3c6908-workload-socket\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:22:37.000843 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:22:37.000561 4927 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f08dd946-14b1-4546-8fae-01564f3c6908-istiod-ca-cert\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:22:37.000843 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:22:37.000577 4927 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f08dd946-14b1-4546-8fae-01564f3c6908-workload-certs\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:22:37.000843 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:22:37.000593 4927 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f08dd946-14b1-4546-8fae-01564f3c6908-istio-data\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:22:37.001393 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:22:37.001352 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f08dd946-14b1-4546-8fae-01564f3c6908-credential-socket" pod "f08dd946-14b1-4546-8fae-01564f3c6908" (UID: "f08dd946-14b1-4546-8fae-01564f3c6908"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:22:37.002679 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:22:37.002641 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f08dd946-14b1-4546-8fae-01564f3c6908-istio-envoy" pod "f08dd946-14b1-4546-8fae-01564f3c6908" (UID: "f08dd946-14b1-4546-8fae-01564f3c6908"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:22:37.002679 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:22:37.002657 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f08dd946-14b1-4546-8fae-01564f3c6908-istio-podinfo" pod "f08dd946-14b1-4546-8fae-01564f3c6908" (UID: "f08dd946-14b1-4546-8fae-01564f3c6908"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:22:37.003386 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:22:37.002741 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f08dd946-14b1-4546-8fae-01564f3c6908-kube-api-access-jf9bt" pod "f08dd946-14b1-4546-8fae-01564f3c6908" (UID: "f08dd946-14b1-4546-8fae-01564f3c6908"). InnerVolumeSpecName "kube-api-access-jf9bt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:22:37.003678 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:22:37.003657 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f08dd946-14b1-4546-8fae-01564f3c6908-istio-token" pod "f08dd946-14b1-4546-8fae-01564f3c6908" (UID: "f08dd946-14b1-4546-8fae-01564f3c6908"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:22:37.078384 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:22:37.078322 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:22:37.101825 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:22:37.101789 4927 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f08dd946-14b1-4546-8fae-01564f3c6908-istio-token\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:22:37.101825 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:22:37.101822 4927 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f08dd946-14b1-4546-8fae-01564f3c6908-credential-socket\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:22:37.102072 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:22:37.101838 4927 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f08dd946-14b1-4546-8fae-01564f3c6908-istio-podinfo\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:22:37.102072 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:22:37.101854 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jf9bt\" (UniqueName: \"kubernetes.io/projected/f08dd946-14b1-4546-8fae-01564f3c6908-kube-api-access-jf9bt\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:22:37.102072 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:22:37.101870 4927 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f08dd946-14b1-4546-8fae-01564f3c6908-istio-envoy\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:22:37.292407 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:22:37.292292 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:22:37.322266 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:22:37.322227 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-d44pq"] Oct 31 00:22:37.326158 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:22:37.326133 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-d44pq"] Oct 31 00:22:38.339009 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:22:38.338973 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:22:38.655103 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:22:38.655003 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:22:39.088588 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:22:39.088549 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f08dd946-14b1-4546-8fae-01564f3c6908" path="/var/lib/kubelet/pods/f08dd946-14b1-4546-8fae-01564f3c6908/volumes" Oct 31 00:22:43.298297 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:22:43.298262 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:22:44.296865 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:22:44.296827 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:22:44.459167 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:22:44.459130 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:22:47.909700 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:22:47.909640 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:22:49.678941 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:22:49.678903 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sqv69" containerName="extractor" Oct 31 00:22:52.531213 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:22:52.531177 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:22:57.285568 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:22:57.285533 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:22:57.519355 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:22:57.518168 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:23:02.503248 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:23:02.503202 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:23:03.580534 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:23:03.580498 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:23:03.778779 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:23:03.778738 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:23:07.167804 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:23:07.167766 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:23:07.434162 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:23:07.434077 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:23:12.209772 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:23:12.209727 4927 generic.go:356] "Generic (PLEG): container finished" podID="360cd796-06da-49c1-9bd4-f2f41f04efeb" containerID="402a284d6077e106734bc3fd7369626a1d28550adc4df5e45092d3862efc2cf8" exitCode=0 Oct 31 00:23:12.210303 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:23:12.209795 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-d687ff-openshift-default-5c75fcb6d5-25thf" event={"ID":"360cd796-06da-49c1-9bd4-f2f41f04efeb","Type":"ContainerDied","Data":"402a284d6077e106734bc3fd7369626a1d28550adc4df5e45092d3862efc2cf8"} Oct 31 00:23:12.210406 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:23:12.210351 4927 scope.go:122] "RemoveContainer" containerID="402a284d6077e106734bc3fd7369626a1d28550adc4df5e45092d3862efc2cf8" Oct 31 00:23:12.212186 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:23:12.212154 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:23:12.212308 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:23:12.212225 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:23:12.212308 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:23:12.212260 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:23:12.366885 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:23:12.366840 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:23:13.239180 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:23:13.239149 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-d687ff-openshift-default-5c75fcb6d5-25thf" event={"ID":"360cd796-06da-49c1-9bd4-f2f41f04efeb","Type":"ContainerStarted","Data":"4d2982d9bc716ac015cb59bdf6cfcd1346e09345f5cb2be03e87a4815f52b7da"} Oct 31 00:23:13.296143 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:23:13.296113 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-d687ff-openshift-default-5c75fcb6d5-25thf" Oct 31 00:23:13.296316 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:23:13.296277 4927 patch_prober.go:29] interesting pod/e2e-api-b-d687ff-openshift-default-5c75fcb6d5-25thf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.243:15021/healthz/ready\": dial tcp 10.132.0.243:15021: connect: connection refused" start-of-body= Oct 31 00:23:13.296402 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:23:13.296379 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-d687ff-openshift-default-5c75fcb6d5-25thf" podUID="360cd796-06da-49c1-9bd4-f2f41f04efeb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.243:15021/healthz/ready\": dial tcp 10.132.0.243:15021: connect: connection refused" Oct 31 00:23:14.296623 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:23:14.296586 4927 patch_prober.go:29] interesting pod/e2e-api-b-d687ff-openshift-default-5c75fcb6d5-25thf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.243:15021/healthz/ready\": dial tcp 10.132.0.243:15021: connect: connection refused" start-of-body= Oct 31 00:23:14.297092 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:23:14.296649 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-d687ff-openshift-default-5c75fcb6d5-25thf" podUID="360cd796-06da-49c1-9bd4-f2f41f04efeb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.243:15021/healthz/ready\": dial tcp 10.132.0.243:15021: connect: connection refused" Oct 31 00:23:15.087384 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:23:15.087346 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85672d8b-zrbx4"] Oct 31 00:23:15.140146 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:23:15.140104 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85672d8b-zrbx4"] Oct 31 00:23:15.297032 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:23:15.296994 4927 patch_prober.go:29] interesting pod/e2e-api-b-d687ff-openshift-default-5c75fcb6d5-25thf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.243:15021/healthz/ready\": dial tcp 10.132.0.243:15021: connect: connection refused" start-of-body= Oct 31 00:23:15.297468 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:23:15.297048 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-d687ff-openshift-default-5c75fcb6d5-25thf" podUID="360cd796-06da-49c1-9bd4-f2f41f04efeb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.243:15021/healthz/ready\": dial tcp 10.132.0.243:15021: connect: connection refused" Oct 31 00:23:16.296939 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:23:16.296904 4927 patch_prober.go:29] interesting pod/e2e-api-b-d687ff-openshift-default-5c75fcb6d5-25thf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.243:15021/healthz/ready\": dial tcp 10.132.0.243:15021: connect: connection refused" start-of-body= Oct 31 00:23:16.297176 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:23:16.296975 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-d687ff-openshift-default-5c75fcb6d5-25thf" podUID="360cd796-06da-49c1-9bd4-f2f41f04efeb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.243:15021/healthz/ready\": dial tcp 10.132.0.243:15021: connect: connection refused" Oct 31 00:23:17.090111 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:23:17.090079 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="025fab84-c395-489e-9662-3d524a5681a4" path="/var/lib/kubelet/pods/025fab84-c395-489e-9662-3d524a5681a4/volumes" Oct 31 00:23:17.250629 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:23:17.250592 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:23:17.298150 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:23:17.298113 4927 patch_prober.go:29] interesting pod/e2e-api-b-d687ff-openshift-default-5c75fcb6d5-25thf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.243:15021/healthz/ready\": dial tcp 10.132.0.243:15021: connect: connection refused" start-of-body= Oct 31 00:23:17.298815 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:23:17.298780 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-d687ff-openshift-default-5c75fcb6d5-25thf" podUID="360cd796-06da-49c1-9bd4-f2f41f04efeb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.243:15021/healthz/ready\": dial tcp 10.132.0.243:15021: connect: connection refused" Oct 31 00:23:18.296596 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:23:18.296559 4927 patch_prober.go:29] interesting pod/e2e-api-b-d687ff-openshift-default-5c75fcb6d5-25thf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.243:15021/healthz/ready\": dial tcp 10.132.0.243:15021: connect: connection refused" start-of-body= Oct 31 00:23:18.296792 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:23:18.296620 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-d687ff-openshift-default-5c75fcb6d5-25thf" podUID="360cd796-06da-49c1-9bd4-f2f41f04efeb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.243:15021/healthz/ready\": dial tcp 10.132.0.243:15021: connect: connection refused" Oct 31 00:23:19.301449 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:23:19.301419 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-d687ff-openshift-default-5c75fcb6d5-25thf" Oct 31 00:23:19.301892 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:23:19.301873 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-d687ff-openshift-default-5c75fcb6d5-25thf" Oct 31 00:23:19.302520 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:23:19.302501 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-d687ff-openshift-default-5c75fcb6d5-25thf" Oct 31 00:23:19.688955 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:23:19.688911 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sqv69" containerName="extractor" Oct 31 00:23:20.044833 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:23:20.044747 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-92e375-75c9fbbc84-nt42r"] Oct 31 00:23:20.159768 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:23:20.159731 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-92e375-6d87d5b947-4249b"] Oct 31 00:23:20.421064 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:23:20.421020 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:23:20.588037 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:23:20.587468 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:23:21.732797 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:23:21.732766 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:23:21.893307 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:23:21.893266 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:23:25.045652 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:23:25.045590 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-92e375-75c9fbbc84-nt42r" podUID="89af18c8-54a0-4dc6-a07b-07bd814ad172" containerName="payload-processing" containerID="cri-o://a7cdfc7579807ed78d04ae294b5e936bd955089beef99761085d5d97ab097598" gracePeriod=25 Oct 31 00:23:25.160705 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:23:25.160664 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-92e375-6d87d5b947-4249b" podUID="0be20326-c5ad-48ff-8126-f4a4e067a098" containerName="payload-pre-processing" containerID="cri-o://277c89dc3a3bc7533aaccb0eca7958b43eede8f1e53e48baec2be6df32fa8e5c" gracePeriod=25 Oct 31 00:23:25.337491 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:23:25.337461 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-92e375-75c9fbbc84-nt42r" Oct 31 00:23:25.383498 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:23:25.383447 4927 generic.go:356] "Generic (PLEG): container finished" podID="0be20326-c5ad-48ff-8126-f4a4e067a098" containerID="277c89dc3a3bc7533aaccb0eca7958b43eede8f1e53e48baec2be6df32fa8e5c" exitCode=0 Oct 31 00:23:25.383760 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:23:25.383734 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-92e375-6d87d5b947-4249b" event={"ID":"0be20326-c5ad-48ff-8126-f4a4e067a098","Type":"ContainerDied","Data":"277c89dc3a3bc7533aaccb0eca7958b43eede8f1e53e48baec2be6df32fa8e5c"} Oct 31 00:23:25.386521 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:23:25.386106 4927 generic.go:356] "Generic (PLEG): container finished" podID="89af18c8-54a0-4dc6-a07b-07bd814ad172" containerID="a7cdfc7579807ed78d04ae294b5e936bd955089beef99761085d5d97ab097598" exitCode=0 Oct 31 00:23:25.386521 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:23:25.386168 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-92e375-75c9fbbc84-nt42r" event={"ID":"89af18c8-54a0-4dc6-a07b-07bd814ad172","Type":"ContainerDied","Data":"a7cdfc7579807ed78d04ae294b5e936bd955089beef99761085d5d97ab097598"} Oct 31 00:23:25.386521 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:23:25.386179 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-92e375-75c9fbbc84-nt42r" Oct 31 00:23:25.386521 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:23:25.386218 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-92e375-75c9fbbc84-nt42r" event={"ID":"89af18c8-54a0-4dc6-a07b-07bd814ad172","Type":"ContainerDied","Data":"4ae0fd4c8daedb66927ee99df1bbf5564d3d4fb7abaa5e9d9b831d2431837714"} Oct 31 00:23:25.386521 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:23:25.386292 4927 scope.go:122] "RemoveContainer" containerID="a7cdfc7579807ed78d04ae294b5e936bd955089beef99761085d5d97ab097598" Oct 31 00:23:25.403041 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:23:25.403016 4927 scope.go:122] "RemoveContainer" containerID="a7cdfc7579807ed78d04ae294b5e936bd955089beef99761085d5d97ab097598" Oct 31 00:23:25.403514 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:23:25.403477 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7cdfc7579807ed78d04ae294b5e936bd955089beef99761085d5d97ab097598\": container with ID starting with a7cdfc7579807ed78d04ae294b5e936bd955089beef99761085d5d97ab097598 not found: ID does not exist" containerID="a7cdfc7579807ed78d04ae294b5e936bd955089beef99761085d5d97ab097598" Oct 31 00:23:25.403647 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:23:25.403525 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7cdfc7579807ed78d04ae294b5e936bd955089beef99761085d5d97ab097598"} err="failed to get container status \"a7cdfc7579807ed78d04ae294b5e936bd955089beef99761085d5d97ab097598\": rpc error: code = NotFound desc = could not find container \"a7cdfc7579807ed78d04ae294b5e936bd955089beef99761085d5d97ab097598\": container with ID starting with a7cdfc7579807ed78d04ae294b5e936bd955089beef99761085d5d97ab097598 not found: ID does not exist" Oct 31 00:23:25.447001 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:23:25.446974 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-92e375-6d87d5b947-4249b" Oct 31 00:23:25.460129 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:23:25.460102 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89af18c8-54a0-4dc6-a07b-07bd814ad172-kube-api-access-qh6td\" (UniqueName: \"kubernetes.io/projected/89af18c8-54a0-4dc6-a07b-07bd814ad172-kube-api-access-qh6td\") pod \"89af18c8-54a0-4dc6-a07b-07bd814ad172\" (UID: \"89af18c8-54a0-4dc6-a07b-07bd814ad172\") " Oct 31 00:23:25.461108 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:23:25.460383 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/89af18c8-54a0-4dc6-a07b-07bd814ad172-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89af18c8-54a0-4dc6-a07b-07bd814ad172-plugins-config-volume\") pod \"89af18c8-54a0-4dc6-a07b-07bd814ad172\" (UID: \"89af18c8-54a0-4dc6-a07b-07bd814ad172\") " Oct 31 00:23:25.461108 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:23:25.460764 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89af18c8-54a0-4dc6-a07b-07bd814ad172-plugins-config-volume" pod "89af18c8-54a0-4dc6-a07b-07bd814ad172" (UID: "89af18c8-54a0-4dc6-a07b-07bd814ad172"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:23:25.461108 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:23:25.461075 4927 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89af18c8-54a0-4dc6-a07b-07bd814ad172-plugins-config-volume\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:23:25.462887 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:23:25.462865 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89af18c8-54a0-4dc6-a07b-07bd814ad172-kube-api-access-qh6td" pod "89af18c8-54a0-4dc6-a07b-07bd814ad172" (UID: "89af18c8-54a0-4dc6-a07b-07bd814ad172"). InnerVolumeSpecName "kube-api-access-qh6td". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:23:25.561896 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:23:25.561818 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0be20326-c5ad-48ff-8126-f4a4e067a098-kube-api-access-qt6vd\" (UniqueName: \"kubernetes.io/projected/0be20326-c5ad-48ff-8126-f4a4e067a098-kube-api-access-qt6vd\") pod \"0be20326-c5ad-48ff-8126-f4a4e067a098\" (UID: \"0be20326-c5ad-48ff-8126-f4a4e067a098\") " Oct 31 00:23:25.562053 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:23:25.561953 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0be20326-c5ad-48ff-8126-f4a4e067a098-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0be20326-c5ad-48ff-8126-f4a4e067a098-plugins-config-volume\") pod \"0be20326-c5ad-48ff-8126-f4a4e067a098\" (UID: \"0be20326-c5ad-48ff-8126-f4a4e067a098\") " Oct 31 00:23:25.562348 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:23:25.562321 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qh6td\" (UniqueName: \"kubernetes.io/projected/89af18c8-54a0-4dc6-a07b-07bd814ad172-kube-api-access-qh6td\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:23:25.562472 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:23:25.562324 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0be20326-c5ad-48ff-8126-f4a4e067a098-plugins-config-volume" pod "0be20326-c5ad-48ff-8126-f4a4e067a098" (UID: "0be20326-c5ad-48ff-8126-f4a4e067a098"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:23:25.564008 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:23:25.563990 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0be20326-c5ad-48ff-8126-f4a4e067a098-kube-api-access-qt6vd" pod "0be20326-c5ad-48ff-8126-f4a4e067a098" (UID: "0be20326-c5ad-48ff-8126-f4a4e067a098"). InnerVolumeSpecName "kube-api-access-qt6vd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:23:25.664060 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:23:25.664026 4927 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0be20326-c5ad-48ff-8126-f4a4e067a098-plugins-config-volume\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:23:25.664060 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:23:25.664058 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qt6vd\" (UniqueName: \"kubernetes.io/projected/0be20326-c5ad-48ff-8126-f4a4e067a098-kube-api-access-qt6vd\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:23:25.814175 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:23:25.814080 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-92e375-75c9fbbc84-nt42r"] Oct 31 00:23:25.817953 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:23:25.817924 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-92e375-75c9fbbc84-nt42r"] Oct 31 00:23:25.893901 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:23:25.893862 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:23:26.397172 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:23:26.397088 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-92e375-6d87d5b947-4249b" Oct 31 00:23:26.397172 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:23:26.397125 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-92e375-6d87d5b947-4249b" event={"ID":"0be20326-c5ad-48ff-8126-f4a4e067a098","Type":"ContainerDied","Data":"a2d490366ed6d9bc35c78afccb5a84daa1ab0bbf79bac75446f3de0046960b15"} Oct 31 00:23:26.397669 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:23:26.397202 4927 scope.go:122] "RemoveContainer" containerID="277c89dc3a3bc7533aaccb0eca7958b43eede8f1e53e48baec2be6df32fa8e5c" Oct 31 00:23:26.568653 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:23:26.568604 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-92e375-6d87d5b947-4249b"] Oct 31 00:23:26.572089 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:23:26.572069 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-92e375-6d87d5b947-4249b"] Oct 31 00:23:27.087767 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:23:27.087733 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0be20326-c5ad-48ff-8126-f4a4e067a098" path="/var/lib/kubelet/pods/0be20326-c5ad-48ff-8126-f4a4e067a098/volumes" Oct 31 00:23:27.088200 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:23:27.088185 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="89af18c8-54a0-4dc6-a07b-07bd814ad172" path="/var/lib/kubelet/pods/89af18c8-54a0-4dc6-a07b-07bd814ad172/volumes" Oct 31 00:23:27.368269 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:23:27.368149 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:23:27.664856 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:23:27.664818 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:23:27.829120 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:23:27.829080 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:23:28.362265 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:23:28.362227 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:23:32.444915 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:23:32.444877 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:23:32.630163 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:23:32.630131 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:23:32.806033 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:23:32.805943 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:23:34.995787 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:23:34.995754 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:23:35.168266 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:23:35.168232 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:23:35.384440 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:23:35.384351 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:23:37.544245 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:23:37.544207 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:23:42.107921 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:23:42.107870 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:23:46.796343 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:23:46.796299 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:23:49.679731 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:23:49.679697 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sqv69" containerName="extractor" Oct 31 00:23:51.366909 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:23:51.366874 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:23:54.111970 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:23:54.111213 4927 patch_prober.go:29] interesting pod/e2e-inf-b-37abc0-openshift-default-65b888884f-pqjfl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.247:15021/healthz/ready\": read tcp 10.132.0.2:58978->10.132.0.247:15021: read: connection reset by peer" start-of-body= Oct 31 00:23:54.111970 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:23:54.111290 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-37abc0-openshift-default-65b888884f-pqjfl" podUID="84baa184-a53a-4f9f-8597-fd7c95e04d0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.247:15021/healthz/ready\": read tcp 10.132.0.2:58978->10.132.0.247:15021: read: connection reset by peer" Oct 31 00:23:54.582579 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:23:54.582546 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:23:54.750799 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:23:54.750767 4927 generic.go:356] "Generic (PLEG): container finished" podID="84baa184-a53a-4f9f-8597-fd7c95e04d0d" containerID="15a2819b990fb45280a7b8b986029e7fb518a58ba0c3030b8b70906d3f091550" exitCode=0 Oct 31 00:23:54.751003 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:23:54.750828 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-37abc0-openshift-default-65b888884f-pqjfl" event={"ID":"84baa184-a53a-4f9f-8597-fd7c95e04d0d","Type":"ContainerDied","Data":"15a2819b990fb45280a7b8b986029e7fb518a58ba0c3030b8b70906d3f091550"} Oct 31 00:23:54.751289 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:23:54.751258 4927 scope.go:122] "RemoveContainer" containerID="15a2819b990fb45280a7b8b986029e7fb518a58ba0c3030b8b70906d3f091550" Oct 31 00:23:54.753845 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:23:54.753792 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:23:54.754085 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:23:54.754063 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:23:54.754260 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:23:54.754234 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:23:55.760474 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:23:55.760436 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-37abc0-openshift-default-65b888884f-pqjfl" event={"ID":"84baa184-a53a-4f9f-8597-fd7c95e04d0d","Type":"ContainerStarted","Data":"98a82c5790d247e201ae11d6fb373788135e95863729e3435f0fa5b6ccd122b6"} Oct 31 00:23:56.326941 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:23:56.326903 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:23:56.576262 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:23:56.576225 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-37abc0-openshift-default-65b888884f-pqjfl" Oct 31 00:23:56.576482 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:23:56.576362 4927 patch_prober.go:29] interesting pod/e2e-inf-b-37abc0-openshift-default-65b888884f-pqjfl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.247:15021/healthz/ready\": dial tcp 10.132.0.247:15021: connect: connection refused" start-of-body= Oct 31 00:23:56.576482 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:23:56.576444 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-37abc0-openshift-default-65b888884f-pqjfl" podUID="84baa184-a53a-4f9f-8597-fd7c95e04d0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.247:15021/healthz/ready\": dial tcp 10.132.0.247:15021: connect: connection refused" Oct 31 00:23:56.733438 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:23:56.733394 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:23:57.576586 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:23:57.576548 4927 patch_prober.go:29] interesting pod/e2e-inf-b-37abc0-openshift-default-65b888884f-pqjfl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.247:15021/healthz/ready\": dial tcp 10.132.0.247:15021: connect: connection refused" start-of-body= Oct 31 00:23:57.577073 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:23:57.576621 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-37abc0-openshift-default-65b888884f-pqjfl" podUID="84baa184-a53a-4f9f-8597-fd7c95e04d0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.247:15021/healthz/ready\": dial tcp 10.132.0.247:15021: connect: connection refused" Oct 31 00:23:58.576237 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:23:58.576198 4927 patch_prober.go:29] interesting pod/e2e-inf-b-37abc0-openshift-default-65b888884f-pqjfl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.247:15021/healthz/ready\": dial tcp 10.132.0.247:15021: connect: connection refused" start-of-body= Oct 31 00:23:58.576465 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:23:58.576262 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-37abc0-openshift-default-65b888884f-pqjfl" podUID="84baa184-a53a-4f9f-8597-fd7c95e04d0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.247:15021/healthz/ready\": dial tcp 10.132.0.247:15021: connect: connection refused" Oct 31 00:23:59.577017 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:23:59.576977 4927 patch_prober.go:29] interesting pod/e2e-inf-b-37abc0-openshift-default-65b888884f-pqjfl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.247:15021/healthz/ready\": dial tcp 10.132.0.247:15021: connect: connection refused" start-of-body= Oct 31 00:23:59.577557 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:23:59.577033 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-37abc0-openshift-default-65b888884f-pqjfl" podUID="84baa184-a53a-4f9f-8597-fd7c95e04d0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.247:15021/healthz/ready\": dial tcp 10.132.0.247:15021: connect: connection refused" Oct 31 00:24:00.576989 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:24:00.576950 4927 patch_prober.go:29] interesting pod/e2e-inf-b-37abc0-openshift-default-65b888884f-pqjfl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.247:15021/healthz/ready\": dial tcp 10.132.0.247:15021: connect: connection refused" start-of-body= Oct 31 00:24:00.577175 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:24:00.577013 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-37abc0-openshift-default-65b888884f-pqjfl" podUID="84baa184-a53a-4f9f-8597-fd7c95e04d0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.247:15021/healthz/ready\": dial tcp 10.132.0.247:15021: connect: connection refused" Oct 31 00:24:01.248114 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:24:01.248077 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:24:01.580976 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:24:01.580903 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-37abc0-openshift-default-65b888884f-pqjfl" Oct 31 00:24:01.581409 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:24:01.581351 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-37abc0-openshift-default-65b888884f-pqjfl" Oct 31 00:24:01.582009 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:24:01.581992 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-37abc0-openshift-default-65b888884f-pqjfl" Oct 31 00:24:06.365353 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:24:06.365303 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:24:09.923991 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:24:09.923894 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-ae3cd0-9bb5478db-92qxb"] Oct 31 00:24:10.037601 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:24:10.037560 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-ae3cd0-7ffbd9dc87-l54tr"] Oct 31 00:24:10.302403 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:24:10.302277 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:24:10.484353 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:24:10.479128 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:24:10.920149 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:24:10.920107 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:24:11.862191 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:24:11.862159 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:24:14.924734 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:24:14.924690 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-ae3cd0-9bb5478db-92qxb" podUID="1f3db095-44dc-4645-bc72-32bee635facf" containerName="payload-processing" containerID="cri-o://4b78c94d4f323ee60628d81df4364bf5440cd6daa2f3acea24537d781ac02e72" gracePeriod=25 Oct 31 00:24:15.038234 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:24:15.038186 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ae3cd0-7ffbd9dc87-l54tr" podUID="e77c7f6d-5bda-4b89-b7f1-2f647f774d03" containerName="payload-pre-processing" containerID="cri-o://c33fccba96501aec0f1a94c5ae1806295128eca56f9db13f60276151d97f2b2c" gracePeriod=25 Oct 31 00:24:15.201143 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:24:15.201097 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-ae3cd0-9bb5478db-92qxb" Oct 31 00:24:15.258643 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:24:15.258614 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f3db095-44dc-4645-bc72-32bee635facf-kube-api-access-l5rz7\" (UniqueName: \"kubernetes.io/projected/1f3db095-44dc-4645-bc72-32bee635facf-kube-api-access-l5rz7\") pod \"1f3db095-44dc-4645-bc72-32bee635facf\" (UID: \"1f3db095-44dc-4645-bc72-32bee635facf\") " Oct 31 00:24:15.258888 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:24:15.258861 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f3db095-44dc-4645-bc72-32bee635facf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f3db095-44dc-4645-bc72-32bee635facf-plugins-config-volume\") pod \"1f3db095-44dc-4645-bc72-32bee635facf\" (UID: \"1f3db095-44dc-4645-bc72-32bee635facf\") " Oct 31 00:24:15.259222 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:24:15.259200 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f3db095-44dc-4645-bc72-32bee635facf-plugins-config-volume" pod "1f3db095-44dc-4645-bc72-32bee635facf" (UID: "1f3db095-44dc-4645-bc72-32bee635facf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:24:15.259485 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:24:15.259465 4927 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f3db095-44dc-4645-bc72-32bee635facf-plugins-config-volume\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:24:15.261283 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:24:15.261249 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f3db095-44dc-4645-bc72-32bee635facf-kube-api-access-l5rz7" pod "1f3db095-44dc-4645-bc72-32bee635facf" (UID: "1f3db095-44dc-4645-bc72-32bee635facf"). InnerVolumeSpecName "kube-api-access-l5rz7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:24:15.342801 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:24:15.342767 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ae3cd0-7ffbd9dc87-l54tr" Oct 31 00:24:15.361171 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:24:15.361132 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l5rz7\" (UniqueName: \"kubernetes.io/projected/1f3db095-44dc-4645-bc72-32bee635facf-kube-api-access-l5rz7\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:24:15.461901 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:24:15.461824 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e77c7f6d-5bda-4b89-b7f1-2f647f774d03-kube-api-access-9dh27\" (UniqueName: \"kubernetes.io/projected/e77c7f6d-5bda-4b89-b7f1-2f647f774d03-kube-api-access-9dh27\") pod \"e77c7f6d-5bda-4b89-b7f1-2f647f774d03\" (UID: \"e77c7f6d-5bda-4b89-b7f1-2f647f774d03\") " Oct 31 00:24:15.462076 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:24:15.461935 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e77c7f6d-5bda-4b89-b7f1-2f647f774d03-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e77c7f6d-5bda-4b89-b7f1-2f647f774d03-plugins-config-volume\") pod \"e77c7f6d-5bda-4b89-b7f1-2f647f774d03\" (UID: \"e77c7f6d-5bda-4b89-b7f1-2f647f774d03\") " Oct 31 00:24:15.462868 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:24:15.462316 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e77c7f6d-5bda-4b89-b7f1-2f647f774d03-plugins-config-volume" pod "e77c7f6d-5bda-4b89-b7f1-2f647f774d03" (UID: "e77c7f6d-5bda-4b89-b7f1-2f647f774d03"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:24:15.462868 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:24:15.462617 4927 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e77c7f6d-5bda-4b89-b7f1-2f647f774d03-plugins-config-volume\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:24:15.464659 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:24:15.464629 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e77c7f6d-5bda-4b89-b7f1-2f647f774d03-kube-api-access-9dh27" pod "e77c7f6d-5bda-4b89-b7f1-2f647f774d03" (UID: "e77c7f6d-5bda-4b89-b7f1-2f647f774d03"). InnerVolumeSpecName "kube-api-access-9dh27". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:24:15.492877 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:24:15.492842 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:24:15.565357 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:24:15.565162 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9dh27\" (UniqueName: \"kubernetes.io/projected/e77c7f6d-5bda-4b89-b7f1-2f647f774d03-kube-api-access-9dh27\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:24:15.972531 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:24:15.972490 4927 generic.go:356] "Generic (PLEG): container finished" podID="1f3db095-44dc-4645-bc72-32bee635facf" containerID="4b78c94d4f323ee60628d81df4364bf5440cd6daa2f3acea24537d781ac02e72" exitCode=0 Oct 31 00:24:15.972941 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:24:15.972542 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-ae3cd0-9bb5478db-92qxb" Oct 31 00:24:15.972941 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:24:15.972572 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-ae3cd0-9bb5478db-92qxb" event={"ID":"1f3db095-44dc-4645-bc72-32bee635facf","Type":"ContainerDied","Data":"4b78c94d4f323ee60628d81df4364bf5440cd6daa2f3acea24537d781ac02e72"} Oct 31 00:24:15.972941 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:24:15.972652 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-ae3cd0-9bb5478db-92qxb" event={"ID":"1f3db095-44dc-4645-bc72-32bee635facf","Type":"ContainerDied","Data":"25c9f6a96ccd1204299fba0ee938e9ff0a28ce19806f6408cd61c03a2411dd4d"} Oct 31 00:24:15.972941 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:24:15.972681 4927 scope.go:122] "RemoveContainer" containerID="4b78c94d4f323ee60628d81df4364bf5440cd6daa2f3acea24537d781ac02e72" Oct 31 00:24:15.974441 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:24:15.974293 4927 generic.go:356] "Generic (PLEG): container finished" podID="e77c7f6d-5bda-4b89-b7f1-2f647f774d03" containerID="c33fccba96501aec0f1a94c5ae1806295128eca56f9db13f60276151d97f2b2c" exitCode=0 Oct 31 00:24:15.974441 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:24:15.974395 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ae3cd0-7ffbd9dc87-l54tr" Oct 31 00:24:15.974441 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:24:15.974392 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ae3cd0-7ffbd9dc87-l54tr" event={"ID":"e77c7f6d-5bda-4b89-b7f1-2f647f774d03","Type":"ContainerDied","Data":"c33fccba96501aec0f1a94c5ae1806295128eca56f9db13f60276151d97f2b2c"} Oct 31 00:24:15.974713 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:24:15.974486 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ae3cd0-7ffbd9dc87-l54tr" event={"ID":"e77c7f6d-5bda-4b89-b7f1-2f647f774d03","Type":"ContainerDied","Data":"9f5424e54450425c93cc07ff06416bf11b0d6abe8ea3d763de715311208797e0"} Oct 31 00:24:15.985810 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:24:15.985794 4927 scope.go:122] "RemoveContainer" containerID="4b78c94d4f323ee60628d81df4364bf5440cd6daa2f3acea24537d781ac02e72" Oct 31 00:24:15.986102 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:24:15.986082 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b78c94d4f323ee60628d81df4364bf5440cd6daa2f3acea24537d781ac02e72\": container with ID starting with 4b78c94d4f323ee60628d81df4364bf5440cd6daa2f3acea24537d781ac02e72 not found: ID does not exist" containerID="4b78c94d4f323ee60628d81df4364bf5440cd6daa2f3acea24537d781ac02e72" Oct 31 00:24:15.986171 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:24:15.986110 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b78c94d4f323ee60628d81df4364bf5440cd6daa2f3acea24537d781ac02e72"} err="failed to get container status \"4b78c94d4f323ee60628d81df4364bf5440cd6daa2f3acea24537d781ac02e72\": rpc error: code = NotFound desc = could not find container \"4b78c94d4f323ee60628d81df4364bf5440cd6daa2f3acea24537d781ac02e72\": container with ID starting with 4b78c94d4f323ee60628d81df4364bf5440cd6daa2f3acea24537d781ac02e72 not found: ID does not exist" Oct 31 00:24:15.986171 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:24:15.986126 4927 scope.go:122] "RemoveContainer" containerID="c33fccba96501aec0f1a94c5ae1806295128eca56f9db13f60276151d97f2b2c" Oct 31 00:24:16.000788 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:24:16.000765 4927 scope.go:122] "RemoveContainer" containerID="c33fccba96501aec0f1a94c5ae1806295128eca56f9db13f60276151d97f2b2c" Oct 31 00:24:16.001363 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:24:16.001183 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c33fccba96501aec0f1a94c5ae1806295128eca56f9db13f60276151d97f2b2c\": container with ID starting with c33fccba96501aec0f1a94c5ae1806295128eca56f9db13f60276151d97f2b2c not found: ID does not exist" containerID="c33fccba96501aec0f1a94c5ae1806295128eca56f9db13f60276151d97f2b2c" Oct 31 00:24:16.001363 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:24:16.001217 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c33fccba96501aec0f1a94c5ae1806295128eca56f9db13f60276151d97f2b2c"} err="failed to get container status \"c33fccba96501aec0f1a94c5ae1806295128eca56f9db13f60276151d97f2b2c\": rpc error: code = NotFound desc = could not find container \"c33fccba96501aec0f1a94c5ae1806295128eca56f9db13f60276151d97f2b2c\": container with ID starting with c33fccba96501aec0f1a94c5ae1806295128eca56f9db13f60276151d97f2b2c not found: ID does not exist" Oct 31 00:24:16.144364 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:24:16.144261 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-ae3cd0-7ffbd9dc87-l54tr"] Oct 31 00:24:16.195147 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:24:16.195077 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-ae3cd0-7ffbd9dc87-l54tr"] Oct 31 00:24:16.260514 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:24:16.260483 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-ae3cd0-9bb5478db-92qxb"] Oct 31 00:24:16.265665 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:24:16.265634 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-ae3cd0-9bb5478db-92qxb"] Oct 31 00:24:17.088132 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:24:17.088096 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f3db095-44dc-4645-bc72-32bee635facf" path="/var/lib/kubelet/pods/1f3db095-44dc-4645-bc72-32bee635facf/volumes" Oct 31 00:24:17.088522 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:24:17.088513 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e77c7f6d-5bda-4b89-b7f1-2f647f774d03" path="/var/lib/kubelet/pods/e77c7f6d-5bda-4b89-b7f1-2f647f774d03/volumes" Oct 31 00:24:19.679490 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:24:19.679451 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sqv69" containerName="extractor" Oct 31 00:24:20.091535 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:24:20.091497 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:24:21.070277 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:24:21.070245 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:24:24.687519 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:24:24.687481 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:24:29.260291 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:24:29.260254 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:24:32.736691 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:24:32.736655 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:24:33.214374 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:24:33.213290 4927 scope.go:122] "RemoveContainer" containerID="3ea6f5675993b7a169c4f62149c867d3511b2542fc1ca288a4f707580a8296a2" Oct 31 00:24:33.788088 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:24:33.788052 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:24:38.163391 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:24:38.163323 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:24:38.683579 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:24:38.683539 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:24:41.863595 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:24:41.863555 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:24:43.290730 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:24:43.290365 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:24:47.800752 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:24:47.800712 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:24:49.678528 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:24:49.678489 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sqv69" containerName="extractor" Oct 31 00:24:52.386661 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:24:52.386624 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:24:56.325170 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:24:56.325123 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-92e375-6c7f5568fd-s99n8"] Oct 31 00:24:56.325759 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:24:56.325398 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-92e375-6c7f5568fd-s99n8" podUID="2c32e6d3-ca5b-459f-b393-9ba509a1247c" containerName="maas-api" containerID="cri-o://bec463672838581119115800a07feec7fe3e73824910eaf1a9ad41a58eeaa70b" gracePeriod=30 Oct 31 00:24:56.432258 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:24:56.432209 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:24:56.689718 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:24:56.689686 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-92e375-6c7f5568fd-s99n8" Oct 31 00:24:56.789897 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:24:56.789853 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c32e6d3-ca5b-459f-b393-9ba509a1247c-kube-api-access-dlpvs\" (UniqueName: \"kubernetes.io/projected/2c32e6d3-ca5b-459f-b393-9ba509a1247c-kube-api-access-dlpvs\") pod \"2c32e6d3-ca5b-459f-b393-9ba509a1247c\" (UID: \"2c32e6d3-ca5b-459f-b393-9ba509a1247c\") " Oct 31 00:24:56.790170 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:24:56.790144 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2c32e6d3-ca5b-459f-b393-9ba509a1247c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2c32e6d3-ca5b-459f-b393-9ba509a1247c-metrics-certs\") pod \"2c32e6d3-ca5b-459f-b393-9ba509a1247c\" (UID: \"2c32e6d3-ca5b-459f-b393-9ba509a1247c\") " Oct 31 00:24:56.790271 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:24:56.790255 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2c32e6d3-ca5b-459f-b393-9ba509a1247c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2c32e6d3-ca5b-459f-b393-9ba509a1247c-maas-api-tls\") pod \"2c32e6d3-ca5b-459f-b393-9ba509a1247c\" (UID: \"2c32e6d3-ca5b-459f-b393-9ba509a1247c\") " Oct 31 00:24:56.790351 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:24:56.790293 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c32e6d3-ca5b-459f-b393-9ba509a1247c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c32e6d3-ca5b-459f-b393-9ba509a1247c-openshift-service-ca\") pod \"2c32e6d3-ca5b-459f-b393-9ba509a1247c\" (UID: \"2c32e6d3-ca5b-459f-b393-9ba509a1247c\") " Oct 31 00:24:56.791547 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:24:56.791520 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c32e6d3-ca5b-459f-b393-9ba509a1247c-openshift-service-ca" pod "2c32e6d3-ca5b-459f-b393-9ba509a1247c" (UID: "2c32e6d3-ca5b-459f-b393-9ba509a1247c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:24:56.794212 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:24:56.794163 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c32e6d3-ca5b-459f-b393-9ba509a1247c-kube-api-access-dlpvs" pod "2c32e6d3-ca5b-459f-b393-9ba509a1247c" (UID: "2c32e6d3-ca5b-459f-b393-9ba509a1247c"). InnerVolumeSpecName "kube-api-access-dlpvs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:24:56.794422 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:24:56.794395 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c32e6d3-ca5b-459f-b393-9ba509a1247c-maas-api-tls" pod "2c32e6d3-ca5b-459f-b393-9ba509a1247c" (UID: "2c32e6d3-ca5b-459f-b393-9ba509a1247c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:24:56.797500 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:24:56.797468 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c32e6d3-ca5b-459f-b393-9ba509a1247c-metrics-certs" pod "2c32e6d3-ca5b-459f-b393-9ba509a1247c" (UID: "2c32e6d3-ca5b-459f-b393-9ba509a1247c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:24:56.892057 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:24:56.892022 4927 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2c32e6d3-ca5b-459f-b393-9ba509a1247c-maas-api-tls\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:24:56.892057 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:24:56.892052 4927 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c32e6d3-ca5b-459f-b393-9ba509a1247c-openshift-service-ca\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:24:56.892057 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:24:56.892064 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dlpvs\" (UniqueName: \"kubernetes.io/projected/2c32e6d3-ca5b-459f-b393-9ba509a1247c-kube-api-access-dlpvs\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:24:56.892301 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:24:56.892073 4927 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2c32e6d3-ca5b-459f-b393-9ba509a1247c-metrics-certs\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:24:57.333159 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:24:57.332261 4927 generic.go:356] "Generic (PLEG): container finished" podID="2c32e6d3-ca5b-459f-b393-9ba509a1247c" containerID="bec463672838581119115800a07feec7fe3e73824910eaf1a9ad41a58eeaa70b" exitCode=0 Oct 31 00:24:57.333159 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:24:57.332529 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-92e375-6c7f5568fd-s99n8" event={"ID":"2c32e6d3-ca5b-459f-b393-9ba509a1247c","Type":"ContainerDied","Data":"bec463672838581119115800a07feec7fe3e73824910eaf1a9ad41a58eeaa70b"} Oct 31 00:24:57.333159 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:24:57.332582 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-92e375-6c7f5568fd-s99n8" event={"ID":"2c32e6d3-ca5b-459f-b393-9ba509a1247c","Type":"ContainerDied","Data":"b18accd9383843f2fcd9bc7e22f174a25fe4f2a2bec9e21219f3d4b30086e83d"} Oct 31 00:24:57.333159 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:24:57.332613 4927 scope.go:122] "RemoveContainer" containerID="bec463672838581119115800a07feec7fe3e73824910eaf1a9ad41a58eeaa70b" Oct 31 00:24:57.333159 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:24:57.332769 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-92e375-6c7f5568fd-s99n8" Oct 31 00:24:57.359311 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:24:57.359263 4927 scope.go:122] "RemoveContainer" containerID="bec463672838581119115800a07feec7fe3e73824910eaf1a9ad41a58eeaa70b" Oct 31 00:24:57.359747 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:24:57.359705 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bec463672838581119115800a07feec7fe3e73824910eaf1a9ad41a58eeaa70b\": container with ID starting with bec463672838581119115800a07feec7fe3e73824910eaf1a9ad41a58eeaa70b not found: ID does not exist" containerID="bec463672838581119115800a07feec7fe3e73824910eaf1a9ad41a58eeaa70b" Oct 31 00:24:57.359857 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:24:57.359750 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bec463672838581119115800a07feec7fe3e73824910eaf1a9ad41a58eeaa70b"} err="failed to get container status \"bec463672838581119115800a07feec7fe3e73824910eaf1a9ad41a58eeaa70b\": rpc error: code = NotFound desc = could not find container \"bec463672838581119115800a07feec7fe3e73824910eaf1a9ad41a58eeaa70b\": container with ID starting with bec463672838581119115800a07feec7fe3e73824910eaf1a9ad41a58eeaa70b not found: ID does not exist" Oct 31 00:24:57.507179 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:24:57.507142 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-92e375-6c7f5568fd-s99n8"] Oct 31 00:24:57.559984 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:24:57.559949 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-92e375-6c7f5568fd-s99n8"] Oct 31 00:24:57.900154 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:24:57.900097 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:24:58.088128 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:24:58.088086 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:24:58.256402 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:24:58.251621 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:24:58.915714 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:24:58.915678 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:24:59.092057 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:24:59.092017 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c32e6d3-ca5b-459f-b393-9ba509a1247c" path="/var/lib/kubelet/pods/2c32e6d3-ca5b-459f-b393-9ba509a1247c/volumes" Oct 31 00:25:03.005757 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:25:03.005721 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:25:04.791814 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:25:04.791775 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:25:05.259221 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:25:05.259174 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:25:05.419039 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:25:05.419001 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:25:05.628596 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:25:05.628501 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-92e375-openshift-default-6987df58bc-jzxns"] Oct 31 00:25:05.628797 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:25:05.628753 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-92e375-openshift-default-6987df58bc-jzxns" podUID="0802e5e7-e2ef-4a19-8663-6ba9b29ec0c2" containerName="istio-proxy" containerID="cri-o://fffc5fcd5f608f9b8fffa309ac1f3d53a4c358b183655f1b894e9f796a69deb8" gracePeriod=30 Oct 31 00:25:05.663472 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:25:05.663437 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:25:06.519164 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:25:06.519122 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d687ff-hxrmq"] Oct 31 00:25:06.519812 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:25:06.519796 4927 cpu_manager.go:414] "containerMap: removing container" podUID="429ee092-42d8-475e-b0c6-b5a9930766fc" containerName="main" Oct 31 00:25:06.519859 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:25:06.519814 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="429ee092-42d8-475e-b0c6-b5a9930766fc" containerName="main" Oct 31 00:25:06.519859 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:25:06.519825 4927 cpu_manager.go:414] "containerMap: removing container" podUID="e77c7f6d-5bda-4b89-b7f1-2f647f774d03" containerName="payload-pre-processing" Oct 31 00:25:06.519859 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:25:06.519832 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e77c7f6d-5bda-4b89-b7f1-2f647f774d03" containerName="payload-pre-processing" Oct 31 00:25:06.519859 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:25:06.519846 4927 cpu_manager.go:414] "containerMap: removing container" podUID="0be20326-c5ad-48ff-8126-f4a4e067a098" containerName="payload-pre-processing" Oct 31 00:25:06.519859 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:25:06.519851 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0be20326-c5ad-48ff-8126-f4a4e067a098" containerName="payload-pre-processing" Oct 31 00:25:06.520030 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:25:06.519865 4927 cpu_manager.go:414] "containerMap: removing container" podUID="89af18c8-54a0-4dc6-a07b-07bd814ad172" containerName="payload-processing" Oct 31 00:25:06.520030 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:25:06.519872 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89af18c8-54a0-4dc6-a07b-07bd814ad172" containerName="payload-processing" Oct 31 00:25:06.520030 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:25:06.519886 4927 cpu_manager.go:414] "containerMap: removing container" podUID="1f3db095-44dc-4645-bc72-32bee635facf" containerName="payload-processing" Oct 31 00:25:06.520030 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:25:06.519890 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f3db095-44dc-4645-bc72-32bee635facf" containerName="payload-processing" Oct 31 00:25:06.520030 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:25:06.519901 4927 cpu_manager.go:414] "containerMap: removing container" podUID="2b69ba41-3647-4827-9acc-8f00b05d299a" containerName="payload-pre-processing" Oct 31 00:25:06.520030 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:25:06.519906 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b69ba41-3647-4827-9acc-8f00b05d299a" containerName="payload-pre-processing" Oct 31 00:25:06.520030 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:25:06.519912 4927 cpu_manager.go:414] "containerMap: removing container" podUID="2c32e6d3-ca5b-459f-b393-9ba509a1247c" containerName="maas-api" Oct 31 00:25:06.520030 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:25:06.519917 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c32e6d3-ca5b-459f-b393-9ba509a1247c" containerName="maas-api" Oct 31 00:25:06.520030 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:25:06.519929 4927 cpu_manager.go:414] "containerMap: removing container" podUID="f08dd946-14b1-4546-8fae-01564f3c6908" containerName="istio-proxy" Oct 31 00:25:06.520030 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:25:06.519934 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f08dd946-14b1-4546-8fae-01564f3c6908" containerName="istio-proxy" Oct 31 00:25:06.520030 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:25:06.520016 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="0be20326-c5ad-48ff-8126-f4a4e067a098" containerName="payload-pre-processing" Oct 31 00:25:06.520030 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:25:06.520026 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="89af18c8-54a0-4dc6-a07b-07bd814ad172" containerName="payload-processing" Oct 31 00:25:06.520415 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:25:06.520039 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f3db095-44dc-4645-bc72-32bee635facf" containerName="payload-processing" Oct 31 00:25:06.520415 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:25:06.520052 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b69ba41-3647-4827-9acc-8f00b05d299a" containerName="payload-pre-processing" Oct 31 00:25:06.520415 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:25:06.520060 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c32e6d3-ca5b-459f-b393-9ba509a1247c" containerName="maas-api" Oct 31 00:25:06.520415 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:25:06.520068 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="f08dd946-14b1-4546-8fae-01564f3c6908" containerName="istio-proxy" Oct 31 00:25:06.520415 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:25:06.520076 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="429ee092-42d8-475e-b0c6-b5a9930766fc" containerName="main" Oct 31 00:25:06.520415 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:25:06.520084 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="e77c7f6d-5bda-4b89-b7f1-2f647f774d03" containerName="payload-pre-processing" Oct 31 00:25:06.525213 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:25:06.525193 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d687ff-hxrmq" Oct 31 00:25:06.581719 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:25:06.581681 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:25:06.581719 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:25:06.581707 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4fdzs\"" Oct 31 00:25:06.638584 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:25:06.638542 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d687ff-hxrmq"] Oct 31 00:25:06.648424 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:25:06.648385 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b79791f-2aa2-4daf-8946-d99a837250d3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-d687ff-hxrmq\" (UID: \"8b79791f-2aa2-4daf-8946-d99a837250d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d687ff-hxrmq" Oct 31 00:25:06.648621 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:25:06.648433 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mjkq2\" (UniqueName: \"kubernetes.io/projected/8b79791f-2aa2-4daf-8946-d99a837250d3-kube-api-access-mjkq2\") pod \"maas-api-revoke-keys-e2e-api-b-d687ff-hxrmq\" (UID: \"8b79791f-2aa2-4daf-8946-d99a837250d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d687ff-hxrmq" Oct 31 00:25:06.648685 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:25:06.648628 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b79791f-2aa2-4daf-8946-d99a837250d3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-d687ff-hxrmq\" (UID: \"8b79791f-2aa2-4daf-8946-d99a837250d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d687ff-hxrmq" Oct 31 00:25:06.750460 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:25:06.750422 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b79791f-2aa2-4daf-8946-d99a837250d3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-d687ff-hxrmq\" (UID: \"8b79791f-2aa2-4daf-8946-d99a837250d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d687ff-hxrmq" Oct 31 00:25:06.750460 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:25:06.750463 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjkq2\" (UniqueName: \"kubernetes.io/projected/8b79791f-2aa2-4daf-8946-d99a837250d3-kube-api-access-mjkq2\") pod \"maas-api-revoke-keys-e2e-api-b-d687ff-hxrmq\" (UID: \"8b79791f-2aa2-4daf-8946-d99a837250d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d687ff-hxrmq" Oct 31 00:25:06.750732 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:25:06.750561 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b79791f-2aa2-4daf-8946-d99a837250d3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-d687ff-hxrmq\" (UID: \"8b79791f-2aa2-4daf-8946-d99a837250d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d687ff-hxrmq" Oct 31 00:25:06.751311 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:25:06.751285 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b79791f-2aa2-4daf-8946-d99a837250d3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-d687ff-hxrmq\" (UID: \"8b79791f-2aa2-4daf-8946-d99a837250d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d687ff-hxrmq" Oct 31 00:25:06.753027 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:25:06.752997 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b79791f-2aa2-4daf-8946-d99a837250d3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-d687ff-hxrmq\" (UID: \"8b79791f-2aa2-4daf-8946-d99a837250d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d687ff-hxrmq" Oct 31 00:25:06.813937 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:25:06.813857 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mjkq2\" (UniqueName: \"kubernetes.io/projected/8b79791f-2aa2-4daf-8946-d99a837250d3-kube-api-access-mjkq2\") pod \"maas-api-revoke-keys-e2e-api-b-d687ff-hxrmq\" (UID: \"8b79791f-2aa2-4daf-8946-d99a837250d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d687ff-hxrmq" Oct 31 00:25:06.839754 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:25:06.839721 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d687ff-hxrmq" Oct 31 00:25:06.969256 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:25:06.969219 4927 patch_prober.go:29] interesting pod/e2e-api-a-92e375-openshift-default-6987df58bc-jzxns container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 00:25:06.969461 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:25:06.969282 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-92e375-openshift-default-6987df58bc-jzxns" podUID="0802e5e7-e2ef-4a19-8663-6ba9b29ec0c2" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:25:07.225029 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:25:07.224991 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d687ff-hxrmq"] Oct 31 00:25:07.225716 ip-10-0-141-180 kubenswrapper[4927]: W1031 00:25:07.225690 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b79791f_2aa2_4daf_8946_d99a837250d3.slice/crio-e48c73f0f025a1455c4ff1e53bc7a2319e60f2a45db5ed76fade36bb5512f8f1 WatchSource:0}: Error finding container e48c73f0f025a1455c4ff1e53bc7a2319e60f2a45db5ed76fade36bb5512f8f1: Status 404 returned error can't find the container with id e48c73f0f025a1455c4ff1e53bc7a2319e60f2a45db5ed76fade36bb5512f8f1 Oct 31 00:25:07.419575 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:25:07.419536 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d687ff-hxrmq" event={"ID":"8b79791f-2aa2-4daf-8946-d99a837250d3","Type":"ContainerStarted","Data":"e48c73f0f025a1455c4ff1e53bc7a2319e60f2a45db5ed76fade36bb5512f8f1"} Oct 31 00:25:07.759059 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:25:07.759022 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:25:08.430509 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:25:08.430469 4927 generic.go:356] "Generic (PLEG): container finished" podID="8b79791f-2aa2-4daf-8946-d99a837250d3" containerID="e5e8167b7df2e3a35b2b4a67942ca163ec632caafc80cfb20f9422771ce42371" exitCode=0 Oct 31 00:25:08.430713 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:25:08.430566 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d687ff-hxrmq" event={"ID":"8b79791f-2aa2-4daf-8946-d99a837250d3","Type":"ContainerDied","Data":"e5e8167b7df2e3a35b2b4a67942ca163ec632caafc80cfb20f9422771ce42371"} Oct 31 00:25:09.594873 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:25:09.594846 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d687ff-hxrmq" Oct 31 00:25:09.685626 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:25:09.685591 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8b79791f-2aa2-4daf-8946-d99a837250d3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b79791f-2aa2-4daf-8946-d99a837250d3-maas-api-xks-ca\") pod \"8b79791f-2aa2-4daf-8946-d99a837250d3\" (UID: \"8b79791f-2aa2-4daf-8946-d99a837250d3\") " Oct 31 00:25:09.685799 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:25:09.685654 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b79791f-2aa2-4daf-8946-d99a837250d3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b79791f-2aa2-4daf-8946-d99a837250d3-maas-api-openshift-ca\") pod \"8b79791f-2aa2-4daf-8946-d99a837250d3\" (UID: \"8b79791f-2aa2-4daf-8946-d99a837250d3\") " Oct 31 00:25:09.685896 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:25:09.685800 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b79791f-2aa2-4daf-8946-d99a837250d3-kube-api-access-mjkq2\" (UniqueName: \"kubernetes.io/projected/8b79791f-2aa2-4daf-8946-d99a837250d3-kube-api-access-mjkq2\") pod \"8b79791f-2aa2-4daf-8946-d99a837250d3\" (UID: \"8b79791f-2aa2-4daf-8946-d99a837250d3\") " Oct 31 00:25:09.686015 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:25:09.685989 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b79791f-2aa2-4daf-8946-d99a837250d3-maas-api-openshift-ca" pod "8b79791f-2aa2-4daf-8946-d99a837250d3" (UID: "8b79791f-2aa2-4daf-8946-d99a837250d3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:25:09.686193 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:25:09.686171 4927 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b79791f-2aa2-4daf-8946-d99a837250d3-maas-api-openshift-ca\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:25:09.687795 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:25:09.687766 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b79791f-2aa2-4daf-8946-d99a837250d3-maas-api-xks-ca" pod "8b79791f-2aa2-4daf-8946-d99a837250d3" (UID: "8b79791f-2aa2-4daf-8946-d99a837250d3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:25:09.687892 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:25:09.687808 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b79791f-2aa2-4daf-8946-d99a837250d3-kube-api-access-mjkq2" pod "8b79791f-2aa2-4daf-8946-d99a837250d3" (UID: "8b79791f-2aa2-4daf-8946-d99a837250d3"). InnerVolumeSpecName "kube-api-access-mjkq2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:25:09.787040 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:25:09.787002 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mjkq2\" (UniqueName: \"kubernetes.io/projected/8b79791f-2aa2-4daf-8946-d99a837250d3-kube-api-access-mjkq2\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:25:09.787040 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:25:09.787035 4927 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b79791f-2aa2-4daf-8946-d99a837250d3-maas-api-xks-ca\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:25:10.448172 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:25:10.448130 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d687ff-hxrmq" event={"ID":"8b79791f-2aa2-4daf-8946-d99a837250d3","Type":"ContainerDied","Data":"e48c73f0f025a1455c4ff1e53bc7a2319e60f2a45db5ed76fade36bb5512f8f1"} Oct 31 00:25:10.448172 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:25:10.448153 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d687ff-hxrmq" Oct 31 00:25:10.448172 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:25:10.448180 4927 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e48c73f0f025a1455c4ff1e53bc7a2319e60f2a45db5ed76fade36bb5512f8f1" Oct 31 00:25:10.897472 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:25:10.897444 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-92e375-openshift-default-6987df58bc-jzxns" Oct 31 00:25:11.000912 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:25:11.000832 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0802e5e7-e2ef-4a19-8663-6ba9b29ec0c2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0802e5e7-e2ef-4a19-8663-6ba9b29ec0c2-workload-socket\") pod \"0802e5e7-e2ef-4a19-8663-6ba9b29ec0c2\" (UID: \"0802e5e7-e2ef-4a19-8663-6ba9b29ec0c2\") " Oct 31 00:25:11.000912 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:25:11.000873 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0802e5e7-e2ef-4a19-8663-6ba9b29ec0c2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0802e5e7-e2ef-4a19-8663-6ba9b29ec0c2-istio-envoy\") pod \"0802e5e7-e2ef-4a19-8663-6ba9b29ec0c2\" (UID: \"0802e5e7-e2ef-4a19-8663-6ba9b29ec0c2\") " Oct 31 00:25:11.001171 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:25:11.000948 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0802e5e7-e2ef-4a19-8663-6ba9b29ec0c2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0802e5e7-e2ef-4a19-8663-6ba9b29ec0c2-istiod-ca-cert\") pod \"0802e5e7-e2ef-4a19-8663-6ba9b29ec0c2\" (UID: \"0802e5e7-e2ef-4a19-8663-6ba9b29ec0c2\") " Oct 31 00:25:11.001171 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:25:11.000971 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0802e5e7-e2ef-4a19-8663-6ba9b29ec0c2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0802e5e7-e2ef-4a19-8663-6ba9b29ec0c2-istio-podinfo\") pod \"0802e5e7-e2ef-4a19-8663-6ba9b29ec0c2\" (UID: \"0802e5e7-e2ef-4a19-8663-6ba9b29ec0c2\") " Oct 31 00:25:11.001171 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:25:11.001001 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0802e5e7-e2ef-4a19-8663-6ba9b29ec0c2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0802e5e7-e2ef-4a19-8663-6ba9b29ec0c2-credential-socket\") pod \"0802e5e7-e2ef-4a19-8663-6ba9b29ec0c2\" (UID: \"0802e5e7-e2ef-4a19-8663-6ba9b29ec0c2\") " Oct 31 00:25:11.001171 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:25:11.001022 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0802e5e7-e2ef-4a19-8663-6ba9b29ec0c2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0802e5e7-e2ef-4a19-8663-6ba9b29ec0c2-workload-certs\") pod \"0802e5e7-e2ef-4a19-8663-6ba9b29ec0c2\" (UID: \"0802e5e7-e2ef-4a19-8663-6ba9b29ec0c2\") " Oct 31 00:25:11.001171 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:25:11.001039 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0802e5e7-e2ef-4a19-8663-6ba9b29ec0c2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0802e5e7-e2ef-4a19-8663-6ba9b29ec0c2-istio-data\") pod \"0802e5e7-e2ef-4a19-8663-6ba9b29ec0c2\" (UID: \"0802e5e7-e2ef-4a19-8663-6ba9b29ec0c2\") " Oct 31 00:25:11.001171 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:25:11.001129 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0802e5e7-e2ef-4a19-8663-6ba9b29ec0c2-workload-socket" pod "0802e5e7-e2ef-4a19-8663-6ba9b29ec0c2" (UID: "0802e5e7-e2ef-4a19-8663-6ba9b29ec0c2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:25:11.001549 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:25:11.001250 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0802e5e7-e2ef-4a19-8663-6ba9b29ec0c2-istio-token\" (UniqueName: \"kubernetes.io/projected/0802e5e7-e2ef-4a19-8663-6ba9b29ec0c2-istio-token\") pod \"0802e5e7-e2ef-4a19-8663-6ba9b29ec0c2\" (UID: \"0802e5e7-e2ef-4a19-8663-6ba9b29ec0c2\") " Oct 31 00:25:11.001549 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:25:11.001313 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0802e5e7-e2ef-4a19-8663-6ba9b29ec0c2-workload-certs" pod "0802e5e7-e2ef-4a19-8663-6ba9b29ec0c2" (UID: "0802e5e7-e2ef-4a19-8663-6ba9b29ec0c2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:25:11.001549 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:25:11.001346 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0802e5e7-e2ef-4a19-8663-6ba9b29ec0c2-credential-socket" pod "0802e5e7-e2ef-4a19-8663-6ba9b29ec0c2" (UID: "0802e5e7-e2ef-4a19-8663-6ba9b29ec0c2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:25:11.001549 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:25:11.001376 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0802e5e7-e2ef-4a19-8663-6ba9b29ec0c2-istio-data" pod "0802e5e7-e2ef-4a19-8663-6ba9b29ec0c2" (UID: "0802e5e7-e2ef-4a19-8663-6ba9b29ec0c2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:25:11.001549 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:25:11.001410 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0802e5e7-e2ef-4a19-8663-6ba9b29ec0c2-kube-api-access-6pxmc\" (UniqueName: \"kubernetes.io/projected/0802e5e7-e2ef-4a19-8663-6ba9b29ec0c2-kube-api-access-6pxmc\") pod \"0802e5e7-e2ef-4a19-8663-6ba9b29ec0c2\" (UID: \"0802e5e7-e2ef-4a19-8663-6ba9b29ec0c2\") " Oct 31 00:25:11.001549 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:25:11.001455 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0802e5e7-e2ef-4a19-8663-6ba9b29ec0c2-istiod-ca-cert" pod "0802e5e7-e2ef-4a19-8663-6ba9b29ec0c2" (UID: "0802e5e7-e2ef-4a19-8663-6ba9b29ec0c2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:25:11.001935 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:25:11.001903 4927 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0802e5e7-e2ef-4a19-8663-6ba9b29ec0c2-workload-socket\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:25:11.001935 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:25:11.001920 4927 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0802e5e7-e2ef-4a19-8663-6ba9b29ec0c2-istiod-ca-cert\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:25:11.001935 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:25:11.001930 4927 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0802e5e7-e2ef-4a19-8663-6ba9b29ec0c2-credential-socket\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:25:11.002114 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:25:11.001939 4927 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0802e5e7-e2ef-4a19-8663-6ba9b29ec0c2-workload-certs\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:25:11.002114 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:25:11.001948 4927 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0802e5e7-e2ef-4a19-8663-6ba9b29ec0c2-istio-data\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:25:11.003371 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:25:11.003320 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0802e5e7-e2ef-4a19-8663-6ba9b29ec0c2-istio-envoy" pod "0802e5e7-e2ef-4a19-8663-6ba9b29ec0c2" (UID: "0802e5e7-e2ef-4a19-8663-6ba9b29ec0c2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:25:11.003480 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:25:11.003461 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0802e5e7-e2ef-4a19-8663-6ba9b29ec0c2-kube-api-access-6pxmc" pod "0802e5e7-e2ef-4a19-8663-6ba9b29ec0c2" (UID: "0802e5e7-e2ef-4a19-8663-6ba9b29ec0c2"). InnerVolumeSpecName "kube-api-access-6pxmc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:25:11.003640 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:25:11.003617 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0802e5e7-e2ef-4a19-8663-6ba9b29ec0c2-istio-podinfo" pod "0802e5e7-e2ef-4a19-8663-6ba9b29ec0c2" (UID: "0802e5e7-e2ef-4a19-8663-6ba9b29ec0c2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:25:11.003827 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:25:11.003803 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0802e5e7-e2ef-4a19-8663-6ba9b29ec0c2-istio-token" pod "0802e5e7-e2ef-4a19-8663-6ba9b29ec0c2" (UID: "0802e5e7-e2ef-4a19-8663-6ba9b29ec0c2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:25:11.103743 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:25:11.103702 4927 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0802e5e7-e2ef-4a19-8663-6ba9b29ec0c2-istio-envoy\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:25:11.103743 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:25:11.103736 4927 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0802e5e7-e2ef-4a19-8663-6ba9b29ec0c2-istio-podinfo\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:25:11.103980 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:25:11.103754 4927 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0802e5e7-e2ef-4a19-8663-6ba9b29ec0c2-istio-token\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:25:11.103980 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:25:11.103771 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6pxmc\" (UniqueName: \"kubernetes.io/projected/0802e5e7-e2ef-4a19-8663-6ba9b29ec0c2-kube-api-access-6pxmc\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:25:11.457182 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:25:11.457147 4927 generic.go:356] "Generic (PLEG): container finished" podID="0802e5e7-e2ef-4a19-8663-6ba9b29ec0c2" containerID="fffc5fcd5f608f9b8fffa309ac1f3d53a4c358b183655f1b894e9f796a69deb8" exitCode=0 Oct 31 00:25:11.457370 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:25:11.457208 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-92e375-openshift-default-6987df58bc-jzxns" Oct 31 00:25:11.457370 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:25:11.457217 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-92e375-openshift-default-6987df58bc-jzxns" event={"ID":"0802e5e7-e2ef-4a19-8663-6ba9b29ec0c2","Type":"ContainerDied","Data":"fffc5fcd5f608f9b8fffa309ac1f3d53a4c358b183655f1b894e9f796a69deb8"} Oct 31 00:25:11.457370 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:25:11.457288 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-92e375-openshift-default-6987df58bc-jzxns" event={"ID":"0802e5e7-e2ef-4a19-8663-6ba9b29ec0c2","Type":"ContainerDied","Data":"a47efad15df9bcb79f44211b1ddf40865d3a22e9712944dbb0dfbbbc77106e8a"} Oct 31 00:25:11.457370 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:25:11.457320 4927 scope.go:122] "RemoveContainer" containerID="fffc5fcd5f608f9b8fffa309ac1f3d53a4c358b183655f1b894e9f796a69deb8" Oct 31 00:25:11.468041 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:25:11.468006 4927 scope.go:122] "RemoveContainer" containerID="678ab560f58ffa4cbda83d6e146dea152a309dabcc7fbecda4d40d0b9cf12dc7" Oct 31 00:25:11.477820 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:25:11.477792 4927 scope.go:122] "RemoveContainer" containerID="fffc5fcd5f608f9b8fffa309ac1f3d53a4c358b183655f1b894e9f796a69deb8" Oct 31 00:25:11.478090 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:25:11.478068 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fffc5fcd5f608f9b8fffa309ac1f3d53a4c358b183655f1b894e9f796a69deb8\": container with ID starting with fffc5fcd5f608f9b8fffa309ac1f3d53a4c358b183655f1b894e9f796a69deb8 not found: ID does not exist" containerID="fffc5fcd5f608f9b8fffa309ac1f3d53a4c358b183655f1b894e9f796a69deb8" Oct 31 00:25:11.478183 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:25:11.478097 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fffc5fcd5f608f9b8fffa309ac1f3d53a4c358b183655f1b894e9f796a69deb8"} err="failed to get container status \"fffc5fcd5f608f9b8fffa309ac1f3d53a4c358b183655f1b894e9f796a69deb8\": rpc error: code = NotFound desc = could not find container \"fffc5fcd5f608f9b8fffa309ac1f3d53a4c358b183655f1b894e9f796a69deb8\": container with ID starting with fffc5fcd5f608f9b8fffa309ac1f3d53a4c358b183655f1b894e9f796a69deb8 not found: ID does not exist" Oct 31 00:25:11.478183 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:25:11.478116 4927 scope.go:122] "RemoveContainer" containerID="678ab560f58ffa4cbda83d6e146dea152a309dabcc7fbecda4d40d0b9cf12dc7" Oct 31 00:25:11.478364 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:25:11.478344 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"678ab560f58ffa4cbda83d6e146dea152a309dabcc7fbecda4d40d0b9cf12dc7\": container with ID starting with 678ab560f58ffa4cbda83d6e146dea152a309dabcc7fbecda4d40d0b9cf12dc7 not found: ID does not exist" containerID="678ab560f58ffa4cbda83d6e146dea152a309dabcc7fbecda4d40d0b9cf12dc7" Oct 31 00:25:11.478415 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:25:11.478370 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"678ab560f58ffa4cbda83d6e146dea152a309dabcc7fbecda4d40d0b9cf12dc7"} err="failed to get container status \"678ab560f58ffa4cbda83d6e146dea152a309dabcc7fbecda4d40d0b9cf12dc7\": rpc error: code = NotFound desc = could not find container \"678ab560f58ffa4cbda83d6e146dea152a309dabcc7fbecda4d40d0b9cf12dc7\": container with ID starting with 678ab560f58ffa4cbda83d6e146dea152a309dabcc7fbecda4d40d0b9cf12dc7 not found: ID does not exist" Oct 31 00:25:11.630523 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:25:11.630491 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-92e375-openshift-default-6987df58bc-jzxns"] Oct 31 00:25:11.679216 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:25:11.679182 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-92e375-openshift-default-6987df58bc-jzxns"] Oct 31 00:25:12.577466 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:25:12.577430 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:25:13.087428 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:25:13.087395 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0802e5e7-e2ef-4a19-8663-6ba9b29ec0c2" path="/var/lib/kubelet/pods/0802e5e7-e2ef-4a19-8663-6ba9b29ec0c2/volumes" Oct 31 00:25:17.440355 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:25:17.437185 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:25:18.297969 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:25:18.297934 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:25:18.472364 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:25:18.468014 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:25:19.695292 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:25:19.695253 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sqv69" containerName="extractor" Oct 31 00:25:20.535104 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:25:20.535068 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:25:22.064086 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:25:22.064040 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:25:26.498887 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:25:26.498845 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:25:31.063729 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:25:31.063684 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:25:33.075199 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:25:33.075165 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:25:33.460625 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:25:33.460585 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:25:35.605303 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:25:35.605265 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:25:40.229306 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:25:40.229208 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:25:40.877632 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:25:40.877583 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-37abc0/test-model-37abc0-kserve-5f4c8bbbb7-5959k"] Oct 31 00:25:40.878185 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:25:40.878172 4927 cpu_manager.go:414] "containerMap: removing container" podUID="8b79791f-2aa2-4daf-8946-d99a837250d3" containerName="revoke-keys" Oct 31 00:25:40.878244 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:25:40.878186 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b79791f-2aa2-4daf-8946-d99a837250d3" containerName="revoke-keys" Oct 31 00:25:40.878244 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:25:40.878227 4927 cpu_manager.go:414] "containerMap: removing container" podUID="0802e5e7-e2ef-4a19-8663-6ba9b29ec0c2" containerName="istio-proxy" Oct 31 00:25:40.878244 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:25:40.878232 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0802e5e7-e2ef-4a19-8663-6ba9b29ec0c2" containerName="istio-proxy" Oct 31 00:25:40.878244 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:25:40.878245 4927 cpu_manager.go:414] "containerMap: removing container" podUID="0802e5e7-e2ef-4a19-8663-6ba9b29ec0c2" containerName="istio-proxy" Oct 31 00:25:40.878417 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:25:40.878250 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0802e5e7-e2ef-4a19-8663-6ba9b29ec0c2" containerName="istio-proxy" Oct 31 00:25:40.878417 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:25:40.878350 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="0802e5e7-e2ef-4a19-8663-6ba9b29ec0c2" containerName="istio-proxy" Oct 31 00:25:40.878417 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:25:40.878359 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="0802e5e7-e2ef-4a19-8663-6ba9b29ec0c2" containerName="istio-proxy" Oct 31 00:25:40.878417 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:25:40.878368 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b79791f-2aa2-4daf-8946-d99a837250d3" containerName="revoke-keys" Oct 31 00:25:40.883237 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:25:40.883220 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-37abc0/test-model-37abc0-kserve-5f4c8bbbb7-5959k" Oct 31 00:25:40.906816 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:25:40.906779 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8zh5\" (UniqueName: \"kubernetes.io/projected/46532fb5-6817-4bf2-bb01-7b03eda6810a-kube-api-access-q8zh5\") pod \"test-model-37abc0-kserve-5f4c8bbbb7-5959k\" (UID: \"46532fb5-6817-4bf2-bb01-7b03eda6810a\") " pod="ai-tenant-e2e-inf-b-37abc0/test-model-37abc0-kserve-5f4c8bbbb7-5959k" Oct 31 00:25:40.906969 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:25:40.906950 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/46532fb5-6817-4bf2-bb01-7b03eda6810a-dshm\") pod \"test-model-37abc0-kserve-5f4c8bbbb7-5959k\" (UID: \"46532fb5-6817-4bf2-bb01-7b03eda6810a\") " pod="ai-tenant-e2e-inf-b-37abc0/test-model-37abc0-kserve-5f4c8bbbb7-5959k" Oct 31 00:25:40.907021 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:25:40.906990 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/46532fb5-6817-4bf2-bb01-7b03eda6810a-model-cache\") pod \"test-model-37abc0-kserve-5f4c8bbbb7-5959k\" (UID: \"46532fb5-6817-4bf2-bb01-7b03eda6810a\") " pod="ai-tenant-e2e-inf-b-37abc0/test-model-37abc0-kserve-5f4c8bbbb7-5959k" Oct 31 00:25:40.907062 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:25:40.907047 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/46532fb5-6817-4bf2-bb01-7b03eda6810a-home\") pod \"test-model-37abc0-kserve-5f4c8bbbb7-5959k\" (UID: \"46532fb5-6817-4bf2-bb01-7b03eda6810a\") " pod="ai-tenant-e2e-inf-b-37abc0/test-model-37abc0-kserve-5f4c8bbbb7-5959k" Oct 31 00:25:40.907108 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:25:40.907088 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/46532fb5-6817-4bf2-bb01-7b03eda6810a-tls-certs\") pod \"test-model-37abc0-kserve-5f4c8bbbb7-5959k\" (UID: \"46532fb5-6817-4bf2-bb01-7b03eda6810a\") " pod="ai-tenant-e2e-inf-b-37abc0/test-model-37abc0-kserve-5f4c8bbbb7-5959k" Oct 31 00:25:40.907244 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:25:40.907224 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/46532fb5-6817-4bf2-bb01-7b03eda6810a-tmp-dir\") pod \"test-model-37abc0-kserve-5f4c8bbbb7-5959k\" (UID: \"46532fb5-6817-4bf2-bb01-7b03eda6810a\") " pod="ai-tenant-e2e-inf-b-37abc0/test-model-37abc0-kserve-5f4c8bbbb7-5959k" Oct 31 00:25:40.942794 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:25:40.942192 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-37abc0/test-model-37abc0-kserve-5f4c8bbbb7-5959k"] Oct 31 00:25:40.973929 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:25:40.973896 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:25:40.987269 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:25:40.987238 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-37abc0\"/\"openshift-service-ca.crt\"" Oct 31 00:25:40.987467 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:25:40.987243 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-37abc0\"/\"kube-root-ca.crt\"" Oct 31 00:25:40.990990 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:25:40.990970 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-37abc0\"/\"default-dockercfg-wp64h\"" Oct 31 00:25:40.991149 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:25:40.991003 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-37abc0\"/\"test-model-37abc0-kserve-self-signed-certs\"" Oct 31 00:25:41.008929 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:25:41.008897 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/46532fb5-6817-4bf2-bb01-7b03eda6810a-tmp-dir\") pod \"test-model-37abc0-kserve-5f4c8bbbb7-5959k\" (UID: \"46532fb5-6817-4bf2-bb01-7b03eda6810a\") " pod="ai-tenant-e2e-inf-b-37abc0/test-model-37abc0-kserve-5f4c8bbbb7-5959k" Oct 31 00:25:41.009054 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:25:41.009016 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8zh5\" (UniqueName: \"kubernetes.io/projected/46532fb5-6817-4bf2-bb01-7b03eda6810a-kube-api-access-q8zh5\") pod \"test-model-37abc0-kserve-5f4c8bbbb7-5959k\" (UID: \"46532fb5-6817-4bf2-bb01-7b03eda6810a\") " pod="ai-tenant-e2e-inf-b-37abc0/test-model-37abc0-kserve-5f4c8bbbb7-5959k" Oct 31 00:25:41.009122 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:25:41.009110 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/46532fb5-6817-4bf2-bb01-7b03eda6810a-dshm\") pod \"test-model-37abc0-kserve-5f4c8bbbb7-5959k\" (UID: \"46532fb5-6817-4bf2-bb01-7b03eda6810a\") " pod="ai-tenant-e2e-inf-b-37abc0/test-model-37abc0-kserve-5f4c8bbbb7-5959k" Oct 31 00:25:41.009194 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:25:41.009154 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/46532fb5-6817-4bf2-bb01-7b03eda6810a-model-cache\") pod \"test-model-37abc0-kserve-5f4c8bbbb7-5959k\" (UID: \"46532fb5-6817-4bf2-bb01-7b03eda6810a\") " pod="ai-tenant-e2e-inf-b-37abc0/test-model-37abc0-kserve-5f4c8bbbb7-5959k" Oct 31 00:25:41.009276 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:25:41.009191 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/46532fb5-6817-4bf2-bb01-7b03eda6810a-home\") pod \"test-model-37abc0-kserve-5f4c8bbbb7-5959k\" (UID: \"46532fb5-6817-4bf2-bb01-7b03eda6810a\") " pod="ai-tenant-e2e-inf-b-37abc0/test-model-37abc0-kserve-5f4c8bbbb7-5959k" Oct 31 00:25:41.009354 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:25:41.009298 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/46532fb5-6817-4bf2-bb01-7b03eda6810a-tls-certs\") pod \"test-model-37abc0-kserve-5f4c8bbbb7-5959k\" (UID: \"46532fb5-6817-4bf2-bb01-7b03eda6810a\") " pod="ai-tenant-e2e-inf-b-37abc0/test-model-37abc0-kserve-5f4c8bbbb7-5959k" Oct 31 00:25:41.009422 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:25:41.009394 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/46532fb5-6817-4bf2-bb01-7b03eda6810a-tmp-dir\") pod \"test-model-37abc0-kserve-5f4c8bbbb7-5959k\" (UID: \"46532fb5-6817-4bf2-bb01-7b03eda6810a\") " pod="ai-tenant-e2e-inf-b-37abc0/test-model-37abc0-kserve-5f4c8bbbb7-5959k" Oct 31 00:25:41.009815 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:25:41.009759 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/46532fb5-6817-4bf2-bb01-7b03eda6810a-model-cache\") pod \"test-model-37abc0-kserve-5f4c8bbbb7-5959k\" (UID: \"46532fb5-6817-4bf2-bb01-7b03eda6810a\") " pod="ai-tenant-e2e-inf-b-37abc0/test-model-37abc0-kserve-5f4c8bbbb7-5959k" Oct 31 00:25:41.009815 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:25:41.009767 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/46532fb5-6817-4bf2-bb01-7b03eda6810a-home\") pod \"test-model-37abc0-kserve-5f4c8bbbb7-5959k\" (UID: \"46532fb5-6817-4bf2-bb01-7b03eda6810a\") " pod="ai-tenant-e2e-inf-b-37abc0/test-model-37abc0-kserve-5f4c8bbbb7-5959k" Oct 31 00:25:41.011938 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:25:41.011911 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/46532fb5-6817-4bf2-bb01-7b03eda6810a-dshm\") pod \"test-model-37abc0-kserve-5f4c8bbbb7-5959k\" (UID: \"46532fb5-6817-4bf2-bb01-7b03eda6810a\") " pod="ai-tenant-e2e-inf-b-37abc0/test-model-37abc0-kserve-5f4c8bbbb7-5959k" Oct 31 00:25:41.012133 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:25:41.012115 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/46532fb5-6817-4bf2-bb01-7b03eda6810a-tls-certs\") pod \"test-model-37abc0-kserve-5f4c8bbbb7-5959k\" (UID: \"46532fb5-6817-4bf2-bb01-7b03eda6810a\") " pod="ai-tenant-e2e-inf-b-37abc0/test-model-37abc0-kserve-5f4c8bbbb7-5959k" Oct 31 00:25:41.100697 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:25:41.100667 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8zh5\" (UniqueName: \"kubernetes.io/projected/46532fb5-6817-4bf2-bb01-7b03eda6810a-kube-api-access-q8zh5\") pod \"test-model-37abc0-kserve-5f4c8bbbb7-5959k\" (UID: \"46532fb5-6817-4bf2-bb01-7b03eda6810a\") " pod="ai-tenant-e2e-inf-b-37abc0/test-model-37abc0-kserve-5f4c8bbbb7-5959k" Oct 31 00:25:41.196485 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:25:41.196451 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-37abc0/test-model-37abc0-kserve-5f4c8bbbb7-5959k" Oct 31 00:25:41.585419 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:25:41.585389 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-37abc0/test-model-37abc0-kserve-5f4c8bbbb7-5959k"] Oct 31 00:25:41.586012 ip-10-0-141-180 kubenswrapper[4927]: W1031 00:25:41.585979 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod46532fb5_6817_4bf2_bb01_7b03eda6810a.slice/crio-5803f62a54736e3b1fe5877987cc61f0332691e9000bb8e5e99d150b474841f5 WatchSource:0}: Error finding container 5803f62a54736e3b1fe5877987cc61f0332691e9000bb8e5e99d150b474841f5: Status 404 returned error can't find the container with id 5803f62a54736e3b1fe5877987cc61f0332691e9000bb8e5e99d150b474841f5 Oct 31 00:25:41.749413 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:25:41.749370 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-37abc0/test-model-37abc0-kserve-5f4c8bbbb7-5959k" event={"ID":"46532fb5-6817-4bf2-bb01-7b03eda6810a","Type":"ContainerStarted","Data":"a2ad8efaa4c1b787fd64ba1901edc4ccb646b28661f27bcd59e466becfe4be88"} Oct 31 00:25:41.749598 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:25:41.749427 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-37abc0/test-model-37abc0-kserve-5f4c8bbbb7-5959k" event={"ID":"46532fb5-6817-4bf2-bb01-7b03eda6810a","Type":"ContainerStarted","Data":"5803f62a54736e3b1fe5877987cc61f0332691e9000bb8e5e99d150b474841f5"} Oct 31 00:25:41.869876 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:25:41.869751 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-37abc0/test-model-37abc0-kserve-5f4c8bbbb7-5959k" podStartSLOduration=1.869738206 podStartE2EDuration="1.869738206s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:25:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:25:41.868863134 +0000 UTC m=+2654.905193985" watchObservedRunningTime="2026-10-31 00:25:41.869738206 +0000 UTC m=+2654.906069028" Oct 31 00:25:42.196979 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:25:42.196945 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-37abc0/test-model-37abc0-kserve-5f4c8bbbb7-5959k" Oct 31 00:25:44.157645 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:25:44.157609 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:25:45.662989 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:25:45.662953 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:25:45.892212 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:25:45.892172 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:25:46.147093 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:25:46.147005 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:25:46.598817 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:25:46.598772 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:25:49.678711 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:25:49.678668 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sqv69" containerName="extractor" Oct 31 00:25:50.804383 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:25:50.804319 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:25:51.197278 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:25:51.197244 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-37abc0/test-model-37abc0-kserve-5f4c8bbbb7-5959k" Oct 31 00:25:51.210459 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:25:51.210430 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-37abc0/test-model-37abc0-kserve-5f4c8bbbb7-5959k" Oct 31 00:25:51.835581 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:25:51.835549 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-37abc0/test-model-37abc0-kserve-5f4c8bbbb7-5959k" Oct 31 00:25:52.949085 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:25:52.949055 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:25:53.169557 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:25:53.169519 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:25:53.332554 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:25:53.332462 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-ae3cd0-openshift-default-5dbcd797d5-hvcv4"] Oct 31 00:25:53.332741 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:25:53.332684 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-ae3cd0-openshift-default-5dbcd797d5-hvcv4" podUID="31803f91-4f1e-407f-b82e-386fd8a80be8" containerName="istio-proxy" containerID="cri-o://5c6b29b49ef7ac09a23b9a5c1c6e55666fcfac059a961b651d90a5f50945e119" gracePeriod=30 Oct 31 00:25:53.454051 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:25:53.454015 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:25:54.177234 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:25:54.177192 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1e5c33-z7f7x"] Oct 31 00:25:54.191188 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:25:54.191159 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1e5c33-z7f7x" Oct 31 00:25:54.240768 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:25:54.240733 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1e5c33-z7f7x"] Oct 31 00:25:54.245019 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:25:54.244985 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:25:54.245180 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:25:54.245105 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4fdzs\"" Oct 31 00:25:54.322006 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:25:54.321972 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e39eb8b7-aaa4-470e-a63d-2b05c14ba5c7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-1e5c33-z7f7x\" (UID: \"e39eb8b7-aaa4-470e-a63d-2b05c14ba5c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1e5c33-z7f7x" Oct 31 00:25:54.322203 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:25:54.322183 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t8grz\" (UniqueName: \"kubernetes.io/projected/e39eb8b7-aaa4-470e-a63d-2b05c14ba5c7-kube-api-access-t8grz\") pod \"maas-api-revoke-keys-e2e-api-b-1e5c33-z7f7x\" (UID: \"e39eb8b7-aaa4-470e-a63d-2b05c14ba5c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1e5c33-z7f7x" Oct 31 00:25:54.322269 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:25:54.322232 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e39eb8b7-aaa4-470e-a63d-2b05c14ba5c7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-1e5c33-z7f7x\" (UID: \"e39eb8b7-aaa4-470e-a63d-2b05c14ba5c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1e5c33-z7f7x" Oct 31 00:25:54.423948 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:25:54.423899 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e39eb8b7-aaa4-470e-a63d-2b05c14ba5c7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-1e5c33-z7f7x\" (UID: \"e39eb8b7-aaa4-470e-a63d-2b05c14ba5c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1e5c33-z7f7x" Oct 31 00:25:54.424171 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:25:54.424046 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8grz\" (UniqueName: \"kubernetes.io/projected/e39eb8b7-aaa4-470e-a63d-2b05c14ba5c7-kube-api-access-t8grz\") pod \"maas-api-revoke-keys-e2e-api-b-1e5c33-z7f7x\" (UID: \"e39eb8b7-aaa4-470e-a63d-2b05c14ba5c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1e5c33-z7f7x" Oct 31 00:25:54.424171 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:25:54.424081 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e39eb8b7-aaa4-470e-a63d-2b05c14ba5c7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-1e5c33-z7f7x\" (UID: \"e39eb8b7-aaa4-470e-a63d-2b05c14ba5c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1e5c33-z7f7x" Oct 31 00:25:54.424958 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:25:54.424929 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e39eb8b7-aaa4-470e-a63d-2b05c14ba5c7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-1e5c33-z7f7x\" (UID: \"e39eb8b7-aaa4-470e-a63d-2b05c14ba5c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1e5c33-z7f7x" Oct 31 00:25:54.426561 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:25:54.426536 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e39eb8b7-aaa4-470e-a63d-2b05c14ba5c7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-1e5c33-z7f7x\" (UID: \"e39eb8b7-aaa4-470e-a63d-2b05c14ba5c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1e5c33-z7f7x" Oct 31 00:25:54.486973 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:25:54.486909 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8grz\" (UniqueName: \"kubernetes.io/projected/e39eb8b7-aaa4-470e-a63d-2b05c14ba5c7-kube-api-access-t8grz\") pod \"maas-api-revoke-keys-e2e-api-b-1e5c33-z7f7x\" (UID: \"e39eb8b7-aaa4-470e-a63d-2b05c14ba5c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1e5c33-z7f7x" Oct 31 00:25:54.505736 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:25:54.505707 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1e5c33-z7f7x" Oct 31 00:25:54.892729 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:25:54.892700 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1e5c33-z7f7x"] Oct 31 00:25:54.893644 ip-10-0-141-180 kubenswrapper[4927]: W1031 00:25:54.893621 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode39eb8b7_aaa4_470e_a63d_2b05c14ba5c7.slice/crio-b03e396df3e77206de4678cba695ad6e37b1dc98a9bc7033089eb7976e3632cf WatchSource:0}: Error finding container b03e396df3e77206de4678cba695ad6e37b1dc98a9bc7033089eb7976e3632cf: Status 404 returned error can't find the container with id b03e396df3e77206de4678cba695ad6e37b1dc98a9bc7033089eb7976e3632cf Oct 31 00:25:55.668534 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:25:55.668503 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:25:55.856660 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:25:55.856629 4927 generic.go:356] "Generic (PLEG): container finished" podID="e39eb8b7-aaa4-470e-a63d-2b05c14ba5c7" containerID="2a9a1dc22d5842c3027722852dbf2f09a8c03e50fa6a17b20e14d13a4f43a983" exitCode=0 Oct 31 00:25:55.856825 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:25:55.856712 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1e5c33-z7f7x" event={"ID":"e39eb8b7-aaa4-470e-a63d-2b05c14ba5c7","Type":"ContainerDied","Data":"2a9a1dc22d5842c3027722852dbf2f09a8c03e50fa6a17b20e14d13a4f43a983"} Oct 31 00:25:55.856825 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:25:55.856772 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1e5c33-z7f7x" event={"ID":"e39eb8b7-aaa4-470e-a63d-2b05c14ba5c7","Type":"ContainerStarted","Data":"b03e396df3e77206de4678cba695ad6e37b1dc98a9bc7033089eb7976e3632cf"} Oct 31 00:25:56.853258 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:25:56.853216 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-678tp"] Oct 31 00:25:56.853788 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:25:56.853498 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-678tp" podUID="8844df03-4076-4f22-94bd-49a4514c21ff" containerName="istio-proxy" containerID="cri-o://54d3605313529f3a8d87d2eeeb44e571a9060e411a04b7802df8676f9d20bac3" gracePeriod=30 Oct 31 00:25:57.048463 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:25:57.048417 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1e5c33-z7f7x" Oct 31 00:25:57.161176 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:25:57.161143 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e39eb8b7-aaa4-470e-a63d-2b05c14ba5c7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e39eb8b7-aaa4-470e-a63d-2b05c14ba5c7-maas-api-openshift-ca\") pod \"e39eb8b7-aaa4-470e-a63d-2b05c14ba5c7\" (UID: \"e39eb8b7-aaa4-470e-a63d-2b05c14ba5c7\") " Oct 31 00:25:57.161402 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:25:57.161190 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e39eb8b7-aaa4-470e-a63d-2b05c14ba5c7-kube-api-access-t8grz\" (UniqueName: \"kubernetes.io/projected/e39eb8b7-aaa4-470e-a63d-2b05c14ba5c7-kube-api-access-t8grz\") pod \"e39eb8b7-aaa4-470e-a63d-2b05c14ba5c7\" (UID: \"e39eb8b7-aaa4-470e-a63d-2b05c14ba5c7\") " Oct 31 00:25:57.161402 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:25:57.161308 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e39eb8b7-aaa4-470e-a63d-2b05c14ba5c7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e39eb8b7-aaa4-470e-a63d-2b05c14ba5c7-maas-api-xks-ca\") pod \"e39eb8b7-aaa4-470e-a63d-2b05c14ba5c7\" (UID: \"e39eb8b7-aaa4-470e-a63d-2b05c14ba5c7\") " Oct 31 00:25:57.161596 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:25:57.161560 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e39eb8b7-aaa4-470e-a63d-2b05c14ba5c7-maas-api-openshift-ca" pod "e39eb8b7-aaa4-470e-a63d-2b05c14ba5c7" (UID: "e39eb8b7-aaa4-470e-a63d-2b05c14ba5c7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:25:57.161803 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:25:57.161783 4927 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e39eb8b7-aaa4-470e-a63d-2b05c14ba5c7-maas-api-openshift-ca\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:25:57.163461 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:25:57.163437 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e39eb8b7-aaa4-470e-a63d-2b05c14ba5c7-maas-api-xks-ca" pod "e39eb8b7-aaa4-470e-a63d-2b05c14ba5c7" (UID: "e39eb8b7-aaa4-470e-a63d-2b05c14ba5c7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:25:57.163631 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:25:57.163610 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e39eb8b7-aaa4-470e-a63d-2b05c14ba5c7-kube-api-access-t8grz" pod "e39eb8b7-aaa4-470e-a63d-2b05c14ba5c7" (UID: "e39eb8b7-aaa4-470e-a63d-2b05c14ba5c7"). InnerVolumeSpecName "kube-api-access-t8grz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:25:57.262724 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:25:57.262685 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t8grz\" (UniqueName: \"kubernetes.io/projected/e39eb8b7-aaa4-470e-a63d-2b05c14ba5c7-kube-api-access-t8grz\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:25:57.262724 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:25:57.262719 4927 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e39eb8b7-aaa4-470e-a63d-2b05c14ba5c7-maas-api-xks-ca\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:25:57.871713 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:25:57.871678 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1e5c33-z7f7x" event={"ID":"e39eb8b7-aaa4-470e-a63d-2b05c14ba5c7","Type":"ContainerDied","Data":"b03e396df3e77206de4678cba695ad6e37b1dc98a9bc7033089eb7976e3632cf"} Oct 31 00:25:57.872099 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:25:57.871737 4927 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b03e396df3e77206de4678cba695ad6e37b1dc98a9bc7033089eb7976e3632cf" Oct 31 00:25:57.872099 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:25:57.871690 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1e5c33-z7f7x" Oct 31 00:25:58.608176 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:25:58.608111 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-ae3cd0-openshift-default-5dbcd797d5-hvcv4" Oct 31 00:25:58.687143 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:25:58.687106 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31803f91-4f1e-407f-b82e-386fd8a80be8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31803f91-4f1e-407f-b82e-386fd8a80be8-istio-envoy\") pod \"31803f91-4f1e-407f-b82e-386fd8a80be8\" (UID: \"31803f91-4f1e-407f-b82e-386fd8a80be8\") " Oct 31 00:25:58.687312 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:25:58.687154 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31803f91-4f1e-407f-b82e-386fd8a80be8-kube-api-access-l2f45\" (UniqueName: \"kubernetes.io/projected/31803f91-4f1e-407f-b82e-386fd8a80be8-kube-api-access-l2f45\") pod \"31803f91-4f1e-407f-b82e-386fd8a80be8\" (UID: \"31803f91-4f1e-407f-b82e-386fd8a80be8\") " Oct 31 00:25:58.687312 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:25:58.687175 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31803f91-4f1e-407f-b82e-386fd8a80be8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31803f91-4f1e-407f-b82e-386fd8a80be8-workload-certs\") pod \"31803f91-4f1e-407f-b82e-386fd8a80be8\" (UID: \"31803f91-4f1e-407f-b82e-386fd8a80be8\") " Oct 31 00:25:58.687312 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:25:58.687231 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31803f91-4f1e-407f-b82e-386fd8a80be8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31803f91-4f1e-407f-b82e-386fd8a80be8-credential-socket\") pod \"31803f91-4f1e-407f-b82e-386fd8a80be8\" (UID: \"31803f91-4f1e-407f-b82e-386fd8a80be8\") " Oct 31 00:25:58.687312 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:25:58.687265 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/31803f91-4f1e-407f-b82e-386fd8a80be8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31803f91-4f1e-407f-b82e-386fd8a80be8-istiod-ca-cert\") pod \"31803f91-4f1e-407f-b82e-386fd8a80be8\" (UID: \"31803f91-4f1e-407f-b82e-386fd8a80be8\") " Oct 31 00:25:58.687312 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:25:58.687309 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31803f91-4f1e-407f-b82e-386fd8a80be8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31803f91-4f1e-407f-b82e-386fd8a80be8-workload-socket\") pod \"31803f91-4f1e-407f-b82e-386fd8a80be8\" (UID: \"31803f91-4f1e-407f-b82e-386fd8a80be8\") " Oct 31 00:25:58.687577 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:25:58.687377 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/31803f91-4f1e-407f-b82e-386fd8a80be8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31803f91-4f1e-407f-b82e-386fd8a80be8-istio-podinfo\") pod \"31803f91-4f1e-407f-b82e-386fd8a80be8\" (UID: \"31803f91-4f1e-407f-b82e-386fd8a80be8\") " Oct 31 00:25:58.687577 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:25:58.687429 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31803f91-4f1e-407f-b82e-386fd8a80be8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31803f91-4f1e-407f-b82e-386fd8a80be8-istio-data\") pod \"31803f91-4f1e-407f-b82e-386fd8a80be8\" (UID: \"31803f91-4f1e-407f-b82e-386fd8a80be8\") " Oct 31 00:25:58.687577 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:25:58.687493 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31803f91-4f1e-407f-b82e-386fd8a80be8-istio-token\" (UniqueName: \"kubernetes.io/projected/31803f91-4f1e-407f-b82e-386fd8a80be8-istio-token\") pod \"31803f91-4f1e-407f-b82e-386fd8a80be8\" (UID: \"31803f91-4f1e-407f-b82e-386fd8a80be8\") " Oct 31 00:25:58.687577 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:25:58.687527 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31803f91-4f1e-407f-b82e-386fd8a80be8-credential-socket" pod "31803f91-4f1e-407f-b82e-386fd8a80be8" (UID: "31803f91-4f1e-407f-b82e-386fd8a80be8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:25:58.687577 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:25:58.687566 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31803f91-4f1e-407f-b82e-386fd8a80be8-workload-certs" pod "31803f91-4f1e-407f-b82e-386fd8a80be8" (UID: "31803f91-4f1e-407f-b82e-386fd8a80be8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:25:58.687845 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:25:58.687686 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31803f91-4f1e-407f-b82e-386fd8a80be8-workload-socket" pod "31803f91-4f1e-407f-b82e-386fd8a80be8" (UID: "31803f91-4f1e-407f-b82e-386fd8a80be8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:25:58.687845 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:25:58.687711 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31803f91-4f1e-407f-b82e-386fd8a80be8-istio-data" pod "31803f91-4f1e-407f-b82e-386fd8a80be8" (UID: "31803f91-4f1e-407f-b82e-386fd8a80be8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:25:58.687845 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:25:58.687799 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31803f91-4f1e-407f-b82e-386fd8a80be8-istiod-ca-cert" pod "31803f91-4f1e-407f-b82e-386fd8a80be8" (UID: "31803f91-4f1e-407f-b82e-386fd8a80be8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:25:58.688030 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:25:58.688009 4927 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31803f91-4f1e-407f-b82e-386fd8a80be8-workload-certs\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:25:58.688097 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:25:58.688032 4927 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31803f91-4f1e-407f-b82e-386fd8a80be8-credential-socket\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:25:58.688097 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:25:58.688046 4927 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31803f91-4f1e-407f-b82e-386fd8a80be8-istiod-ca-cert\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:25:58.688097 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:25:58.688058 4927 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31803f91-4f1e-407f-b82e-386fd8a80be8-workload-socket\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:25:58.688097 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:25:58.688066 4927 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31803f91-4f1e-407f-b82e-386fd8a80be8-istio-data\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:25:58.689485 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:25:58.689459 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31803f91-4f1e-407f-b82e-386fd8a80be8-kube-api-access-l2f45" pod "31803f91-4f1e-407f-b82e-386fd8a80be8" (UID: "31803f91-4f1e-407f-b82e-386fd8a80be8"). InnerVolumeSpecName "kube-api-access-l2f45". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:25:58.689767 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:25:58.689745 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/31803f91-4f1e-407f-b82e-386fd8a80be8-istio-podinfo" pod "31803f91-4f1e-407f-b82e-386fd8a80be8" (UID: "31803f91-4f1e-407f-b82e-386fd8a80be8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:25:58.690021 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:25:58.689995 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31803f91-4f1e-407f-b82e-386fd8a80be8-istio-envoy" pod "31803f91-4f1e-407f-b82e-386fd8a80be8" (UID: "31803f91-4f1e-407f-b82e-386fd8a80be8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:25:58.690021 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:25:58.690001 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31803f91-4f1e-407f-b82e-386fd8a80be8-istio-token" pod "31803f91-4f1e-407f-b82e-386fd8a80be8" (UID: "31803f91-4f1e-407f-b82e-386fd8a80be8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:25:58.788908 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:25:58.788877 4927 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31803f91-4f1e-407f-b82e-386fd8a80be8-istio-envoy\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:25:58.788908 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:25:58.788906 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l2f45\" (UniqueName: \"kubernetes.io/projected/31803f91-4f1e-407f-b82e-386fd8a80be8-kube-api-access-l2f45\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:25:58.789110 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:25:58.788921 4927 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31803f91-4f1e-407f-b82e-386fd8a80be8-istio-podinfo\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:25:58.789110 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:25:58.788933 4927 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/31803f91-4f1e-407f-b82e-386fd8a80be8-istio-token\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:25:58.880953 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:25:58.880862 4927 generic.go:356] "Generic (PLEG): container finished" podID="31803f91-4f1e-407f-b82e-386fd8a80be8" containerID="5c6b29b49ef7ac09a23b9a5c1c6e55666fcfac059a961b651d90a5f50945e119" exitCode=0 Oct 31 00:25:58.880953 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:25:58.880923 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-ae3cd0-openshift-default-5dbcd797d5-hvcv4" Oct 31 00:25:58.880953 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:25:58.880940 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ae3cd0-openshift-default-5dbcd797d5-hvcv4" event={"ID":"31803f91-4f1e-407f-b82e-386fd8a80be8","Type":"ContainerDied","Data":"5c6b29b49ef7ac09a23b9a5c1c6e55666fcfac059a961b651d90a5f50945e119"} Oct 31 00:25:58.881667 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:25:58.881004 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ae3cd0-openshift-default-5dbcd797d5-hvcv4" event={"ID":"31803f91-4f1e-407f-b82e-386fd8a80be8","Type":"ContainerDied","Data":"681fe38e39b6fcce80cc7eaca86251337c0aaf6e3b408355fa84b3af0285124b"} Oct 31 00:25:58.881667 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:25:58.881019 4927 scope.go:122] "RemoveContainer" containerID="5c6b29b49ef7ac09a23b9a5c1c6e55666fcfac059a961b651d90a5f50945e119" Oct 31 00:25:58.892341 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:25:58.892313 4927 scope.go:122] "RemoveContainer" containerID="dac22421d14078433923c10beb17d05dee975d3befd8509fa2341aa6c6bfe2ae" Oct 31 00:25:58.901824 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:25:58.901807 4927 scope.go:122] "RemoveContainer" containerID="5c6b29b49ef7ac09a23b9a5c1c6e55666fcfac059a961b651d90a5f50945e119" Oct 31 00:25:58.902072 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:25:58.902052 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c6b29b49ef7ac09a23b9a5c1c6e55666fcfac059a961b651d90a5f50945e119\": container with ID starting with 5c6b29b49ef7ac09a23b9a5c1c6e55666fcfac059a961b651d90a5f50945e119 not found: ID does not exist" containerID="5c6b29b49ef7ac09a23b9a5c1c6e55666fcfac059a961b651d90a5f50945e119" Oct 31 00:25:58.902133 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:25:58.902081 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c6b29b49ef7ac09a23b9a5c1c6e55666fcfac059a961b651d90a5f50945e119"} err="failed to get container status \"5c6b29b49ef7ac09a23b9a5c1c6e55666fcfac059a961b651d90a5f50945e119\": rpc error: code = NotFound desc = could not find container \"5c6b29b49ef7ac09a23b9a5c1c6e55666fcfac059a961b651d90a5f50945e119\": container with ID starting with 5c6b29b49ef7ac09a23b9a5c1c6e55666fcfac059a961b651d90a5f50945e119 not found: ID does not exist" Oct 31 00:25:58.902133 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:25:58.902098 4927 scope.go:122] "RemoveContainer" containerID="dac22421d14078433923c10beb17d05dee975d3befd8509fa2341aa6c6bfe2ae" Oct 31 00:25:58.902361 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:25:58.902325 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dac22421d14078433923c10beb17d05dee975d3befd8509fa2341aa6c6bfe2ae\": container with ID starting with dac22421d14078433923c10beb17d05dee975d3befd8509fa2341aa6c6bfe2ae not found: ID does not exist" containerID="dac22421d14078433923c10beb17d05dee975d3befd8509fa2341aa6c6bfe2ae" Oct 31 00:25:58.902425 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:25:58.902366 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dac22421d14078433923c10beb17d05dee975d3befd8509fa2341aa6c6bfe2ae"} err="failed to get container status \"dac22421d14078433923c10beb17d05dee975d3befd8509fa2341aa6c6bfe2ae\": rpc error: code = NotFound desc = could not find container \"dac22421d14078433923c10beb17d05dee975d3befd8509fa2341aa6c6bfe2ae\": container with ID starting with dac22421d14078433923c10beb17d05dee975d3befd8509fa2341aa6c6bfe2ae not found: ID does not exist" Oct 31 00:25:59.055375 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:25:59.053449 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-ae3cd0-openshift-default-5dbcd797d5-hvcv4"] Oct 31 00:25:59.057533 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:25:59.057510 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-ae3cd0-openshift-default-5dbcd797d5-hvcv4"] Oct 31 00:25:59.087600 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:25:59.087575 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="31803f91-4f1e-407f-b82e-386fd8a80be8" path="/var/lib/kubelet/pods/31803f91-4f1e-407f-b82e-386fd8a80be8/volumes" Oct 31 00:26:00.482204 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:26:00.482171 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:26:02.122708 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:26:02.122628 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-678tp" Oct 31 00:26:02.136976 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:26:02.136946 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8844df03-4076-4f22-94bd-49a4514c21ff-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8844df03-4076-4f22-94bd-49a4514c21ff-istio-data\") pod \"8844df03-4076-4f22-94bd-49a4514c21ff\" (UID: \"8844df03-4076-4f22-94bd-49a4514c21ff\") " Oct 31 00:26:02.137147 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:26:02.136982 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8844df03-4076-4f22-94bd-49a4514c21ff-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8844df03-4076-4f22-94bd-49a4514c21ff-istio-envoy\") pod \"8844df03-4076-4f22-94bd-49a4514c21ff\" (UID: \"8844df03-4076-4f22-94bd-49a4514c21ff\") " Oct 31 00:26:02.137147 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:26:02.137105 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8844df03-4076-4f22-94bd-49a4514c21ff-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8844df03-4076-4f22-94bd-49a4514c21ff-credential-socket\") pod \"8844df03-4076-4f22-94bd-49a4514c21ff\" (UID: \"8844df03-4076-4f22-94bd-49a4514c21ff\") " Oct 31 00:26:02.137304 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:26:02.137153 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8844df03-4076-4f22-94bd-49a4514c21ff-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8844df03-4076-4f22-94bd-49a4514c21ff-istiod-ca-cert\") pod \"8844df03-4076-4f22-94bd-49a4514c21ff\" (UID: \"8844df03-4076-4f22-94bd-49a4514c21ff\") " Oct 31 00:26:02.137304 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:26:02.137181 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8844df03-4076-4f22-94bd-49a4514c21ff-istio-token\" (UniqueName: \"kubernetes.io/projected/8844df03-4076-4f22-94bd-49a4514c21ff-istio-token\") pod \"8844df03-4076-4f22-94bd-49a4514c21ff\" (UID: \"8844df03-4076-4f22-94bd-49a4514c21ff\") " Oct 31 00:26:02.137304 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:26:02.137205 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8844df03-4076-4f22-94bd-49a4514c21ff-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8844df03-4076-4f22-94bd-49a4514c21ff-workload-certs\") pod \"8844df03-4076-4f22-94bd-49a4514c21ff\" (UID: \"8844df03-4076-4f22-94bd-49a4514c21ff\") " Oct 31 00:26:02.137304 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:26:02.137247 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8844df03-4076-4f22-94bd-49a4514c21ff-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8844df03-4076-4f22-94bd-49a4514c21ff-istio-podinfo\") pod \"8844df03-4076-4f22-94bd-49a4514c21ff\" (UID: \"8844df03-4076-4f22-94bd-49a4514c21ff\") " Oct 31 00:26:02.137556 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:26:02.137358 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8844df03-4076-4f22-94bd-49a4514c21ff-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8844df03-4076-4f22-94bd-49a4514c21ff-workload-socket\") pod \"8844df03-4076-4f22-94bd-49a4514c21ff\" (UID: \"8844df03-4076-4f22-94bd-49a4514c21ff\") " Oct 31 00:26:02.137556 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:26:02.137422 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8844df03-4076-4f22-94bd-49a4514c21ff-kube-api-access-z5lvp\" (UniqueName: \"kubernetes.io/projected/8844df03-4076-4f22-94bd-49a4514c21ff-kube-api-access-z5lvp\") pod \"8844df03-4076-4f22-94bd-49a4514c21ff\" (UID: \"8844df03-4076-4f22-94bd-49a4514c21ff\") " Oct 31 00:26:02.137556 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:26:02.137245 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8844df03-4076-4f22-94bd-49a4514c21ff-istio-data" pod "8844df03-4076-4f22-94bd-49a4514c21ff" (UID: "8844df03-4076-4f22-94bd-49a4514c21ff"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:26:02.137556 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:26:02.137526 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8844df03-4076-4f22-94bd-49a4514c21ff-credential-socket" pod "8844df03-4076-4f22-94bd-49a4514c21ff" (UID: "8844df03-4076-4f22-94bd-49a4514c21ff"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:26:02.137769 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:26:02.137606 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8844df03-4076-4f22-94bd-49a4514c21ff-workload-certs" pod "8844df03-4076-4f22-94bd-49a4514c21ff" (UID: "8844df03-4076-4f22-94bd-49a4514c21ff"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:26:02.137769 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:26:02.137713 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8844df03-4076-4f22-94bd-49a4514c21ff-istiod-ca-cert" pod "8844df03-4076-4f22-94bd-49a4514c21ff" (UID: "8844df03-4076-4f22-94bd-49a4514c21ff"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:26:02.137884 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:26:02.137780 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8844df03-4076-4f22-94bd-49a4514c21ff-workload-socket" pod "8844df03-4076-4f22-94bd-49a4514c21ff" (UID: "8844df03-4076-4f22-94bd-49a4514c21ff"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:26:02.138094 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:26:02.138072 4927 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8844df03-4076-4f22-94bd-49a4514c21ff-credential-socket\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:26:02.138179 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:26:02.138099 4927 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8844df03-4076-4f22-94bd-49a4514c21ff-istiod-ca-cert\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:26:02.138179 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:26:02.138114 4927 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8844df03-4076-4f22-94bd-49a4514c21ff-workload-certs\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:26:02.138179 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:26:02.138128 4927 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8844df03-4076-4f22-94bd-49a4514c21ff-workload-socket\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:26:02.138179 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:26:02.138142 4927 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8844df03-4076-4f22-94bd-49a4514c21ff-istio-data\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:26:02.139511 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:26:02.139487 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8844df03-4076-4f22-94bd-49a4514c21ff-istio-podinfo" pod "8844df03-4076-4f22-94bd-49a4514c21ff" (UID: "8844df03-4076-4f22-94bd-49a4514c21ff"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:26:02.139623 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:26:02.139511 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8844df03-4076-4f22-94bd-49a4514c21ff-istio-envoy" pod "8844df03-4076-4f22-94bd-49a4514c21ff" (UID: "8844df03-4076-4f22-94bd-49a4514c21ff"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:26:02.139623 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:26:02.139615 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8844df03-4076-4f22-94bd-49a4514c21ff-kube-api-access-z5lvp" pod "8844df03-4076-4f22-94bd-49a4514c21ff" (UID: "8844df03-4076-4f22-94bd-49a4514c21ff"). InnerVolumeSpecName "kube-api-access-z5lvp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:26:02.139714 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:26:02.139695 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8844df03-4076-4f22-94bd-49a4514c21ff-istio-token" pod "8844df03-4076-4f22-94bd-49a4514c21ff" (UID: "8844df03-4076-4f22-94bd-49a4514c21ff"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:26:02.239265 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:26:02.239228 4927 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8844df03-4076-4f22-94bd-49a4514c21ff-istio-token\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:26:02.239265 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:26:02.239260 4927 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8844df03-4076-4f22-94bd-49a4514c21ff-istio-podinfo\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:26:02.239265 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:26:02.239272 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z5lvp\" (UniqueName: \"kubernetes.io/projected/8844df03-4076-4f22-94bd-49a4514c21ff-kube-api-access-z5lvp\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:26:02.239532 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:26:02.239283 4927 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8844df03-4076-4f22-94bd-49a4514c21ff-istio-envoy\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:26:02.916750 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:26:02.916713 4927 generic.go:356] "Generic (PLEG): container finished" podID="8844df03-4076-4f22-94bd-49a4514c21ff" containerID="54d3605313529f3a8d87d2eeeb44e571a9060e411a04b7802df8676f9d20bac3" exitCode=0 Oct 31 00:26:02.916952 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:26:02.916780 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-678tp" Oct 31 00:26:02.916952 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:26:02.916795 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-678tp" event={"ID":"8844df03-4076-4f22-94bd-49a4514c21ff","Type":"ContainerDied","Data":"54d3605313529f3a8d87d2eeeb44e571a9060e411a04b7802df8676f9d20bac3"} Oct 31 00:26:02.916952 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:26:02.916874 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-678tp" event={"ID":"8844df03-4076-4f22-94bd-49a4514c21ff","Type":"ContainerDied","Data":"970f4bdd6ccf5cc6f2ec3c3b9f89f3eefb48a2f1211f35a2592e860ba0ad7ce1"} Oct 31 00:26:02.916952 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:26:02.916897 4927 scope.go:122] "RemoveContainer" containerID="54d3605313529f3a8d87d2eeeb44e571a9060e411a04b7802df8676f9d20bac3" Oct 31 00:26:02.928644 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:26:02.928624 4927 scope.go:122] "RemoveContainer" containerID="2eb5077930c6b1e1cddac4a30c1a767126179b165ee2d731d11d0be2e41468fe" Oct 31 00:26:02.940097 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:26:02.940076 4927 scope.go:122] "RemoveContainer" containerID="54d3605313529f3a8d87d2eeeb44e571a9060e411a04b7802df8676f9d20bac3" Oct 31 00:26:02.941574 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:26:02.941541 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54d3605313529f3a8d87d2eeeb44e571a9060e411a04b7802df8676f9d20bac3\": container with ID starting with 54d3605313529f3a8d87d2eeeb44e571a9060e411a04b7802df8676f9d20bac3 not found: ID does not exist" containerID="54d3605313529f3a8d87d2eeeb44e571a9060e411a04b7802df8676f9d20bac3" Oct 31 00:26:02.941675 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:26:02.941583 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54d3605313529f3a8d87d2eeeb44e571a9060e411a04b7802df8676f9d20bac3"} err="failed to get container status \"54d3605313529f3a8d87d2eeeb44e571a9060e411a04b7802df8676f9d20bac3\": rpc error: code = NotFound desc = could not find container \"54d3605313529f3a8d87d2eeeb44e571a9060e411a04b7802df8676f9d20bac3\": container with ID starting with 54d3605313529f3a8d87d2eeeb44e571a9060e411a04b7802df8676f9d20bac3 not found: ID does not exist" Oct 31 00:26:02.941675 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:26:02.941608 4927 scope.go:122] "RemoveContainer" containerID="2eb5077930c6b1e1cddac4a30c1a767126179b165ee2d731d11d0be2e41468fe" Oct 31 00:26:02.942001 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:26:02.941975 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2eb5077930c6b1e1cddac4a30c1a767126179b165ee2d731d11d0be2e41468fe\": container with ID starting with 2eb5077930c6b1e1cddac4a30c1a767126179b165ee2d731d11d0be2e41468fe not found: ID does not exist" containerID="2eb5077930c6b1e1cddac4a30c1a767126179b165ee2d731d11d0be2e41468fe" Oct 31 00:26:02.942118 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:26:02.942003 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2eb5077930c6b1e1cddac4a30c1a767126179b165ee2d731d11d0be2e41468fe"} err="failed to get container status \"2eb5077930c6b1e1cddac4a30c1a767126179b165ee2d731d11d0be2e41468fe\": rpc error: code = NotFound desc = could not find container \"2eb5077930c6b1e1cddac4a30c1a767126179b165ee2d731d11d0be2e41468fe\": container with ID starting with 2eb5077930c6b1e1cddac4a30c1a767126179b165ee2d731d11d0be2e41468fe not found: ID does not exist" Oct 31 00:26:03.089797 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:26:03.089762 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-678tp"] Oct 31 00:26:03.140100 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:26:03.140066 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-678tp"] Oct 31 00:26:03.898148 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:26:03.898109 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:26:05.022202 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:26:05.022166 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:26:05.087625 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:26:05.087585 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8844df03-4076-4f22-94bd-49a4514c21ff" path="/var/lib/kubelet/pods/8844df03-4076-4f22-94bd-49a4514c21ff/volumes" Oct 31 00:26:06.051999 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:26:06.043814 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:26:06.203722 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:26:06.203685 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:26:09.736250 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:26:09.736211 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:26:10.466978 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:26:10.466944 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-jqc5p"] Oct 31 00:26:10.467279 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:26:10.467237 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-jqc5p" podUID="a23bdae1-1e2a-4e26-8e2b-43d7b50a800e" containerName="istio-proxy" containerID="cri-o://604282d2cb273a5b341a0126f2894ded0f1f72e73c53d2b17248a900ced00ea5" gracePeriod=30 Oct 31 00:26:14.285185 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:26:14.285144 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:26:15.742239 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:26:15.742182 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-jqc5p" Oct 31 00:26:15.811687 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:26:15.811659 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a23bdae1-1e2a-4e26-8e2b-43d7b50a800e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a23bdae1-1e2a-4e26-8e2b-43d7b50a800e-workload-certs\") pod \"a23bdae1-1e2a-4e26-8e2b-43d7b50a800e\" (UID: \"a23bdae1-1e2a-4e26-8e2b-43d7b50a800e\") " Oct 31 00:26:15.811846 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:26:15.811710 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a23bdae1-1e2a-4e26-8e2b-43d7b50a800e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a23bdae1-1e2a-4e26-8e2b-43d7b50a800e-credential-socket\") pod \"a23bdae1-1e2a-4e26-8e2b-43d7b50a800e\" (UID: \"a23bdae1-1e2a-4e26-8e2b-43d7b50a800e\") " Oct 31 00:26:15.811846 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:26:15.811770 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a23bdae1-1e2a-4e26-8e2b-43d7b50a800e-istio-token\" (UniqueName: \"kubernetes.io/projected/a23bdae1-1e2a-4e26-8e2b-43d7b50a800e-istio-token\") pod \"a23bdae1-1e2a-4e26-8e2b-43d7b50a800e\" (UID: \"a23bdae1-1e2a-4e26-8e2b-43d7b50a800e\") " Oct 31 00:26:15.811846 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:26:15.811814 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a23bdae1-1e2a-4e26-8e2b-43d7b50a800e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a23bdae1-1e2a-4e26-8e2b-43d7b50a800e-istio-data\") pod \"a23bdae1-1e2a-4e26-8e2b-43d7b50a800e\" (UID: \"a23bdae1-1e2a-4e26-8e2b-43d7b50a800e\") " Oct 31 00:26:15.812036 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:26:15.811847 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a23bdae1-1e2a-4e26-8e2b-43d7b50a800e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a23bdae1-1e2a-4e26-8e2b-43d7b50a800e-istio-podinfo\") pod \"a23bdae1-1e2a-4e26-8e2b-43d7b50a800e\" (UID: \"a23bdae1-1e2a-4e26-8e2b-43d7b50a800e\") " Oct 31 00:26:15.812036 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:26:15.811943 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a23bdae1-1e2a-4e26-8e2b-43d7b50a800e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a23bdae1-1e2a-4e26-8e2b-43d7b50a800e-workload-socket\") pod \"a23bdae1-1e2a-4e26-8e2b-43d7b50a800e\" (UID: \"a23bdae1-1e2a-4e26-8e2b-43d7b50a800e\") " Oct 31 00:26:15.812036 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:26:15.811968 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a23bdae1-1e2a-4e26-8e2b-43d7b50a800e-workload-certs" pod "a23bdae1-1e2a-4e26-8e2b-43d7b50a800e" (UID: "a23bdae1-1e2a-4e26-8e2b-43d7b50a800e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:26:15.812036 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:26:15.811977 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a23bdae1-1e2a-4e26-8e2b-43d7b50a800e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a23bdae1-1e2a-4e26-8e2b-43d7b50a800e-istiod-ca-cert\") pod \"a23bdae1-1e2a-4e26-8e2b-43d7b50a800e\" (UID: \"a23bdae1-1e2a-4e26-8e2b-43d7b50a800e\") " Oct 31 00:26:15.812262 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:26:15.812064 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a23bdae1-1e2a-4e26-8e2b-43d7b50a800e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a23bdae1-1e2a-4e26-8e2b-43d7b50a800e-istio-envoy\") pod \"a23bdae1-1e2a-4e26-8e2b-43d7b50a800e\" (UID: \"a23bdae1-1e2a-4e26-8e2b-43d7b50a800e\") " Oct 31 00:26:15.812262 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:26:15.812086 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a23bdae1-1e2a-4e26-8e2b-43d7b50a800e-credential-socket" pod "a23bdae1-1e2a-4e26-8e2b-43d7b50a800e" (UID: "a23bdae1-1e2a-4e26-8e2b-43d7b50a800e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:26:15.812262 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:26:15.812145 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a23bdae1-1e2a-4e26-8e2b-43d7b50a800e-kube-api-access-86d72\" (UniqueName: \"kubernetes.io/projected/a23bdae1-1e2a-4e26-8e2b-43d7b50a800e-kube-api-access-86d72\") pod \"a23bdae1-1e2a-4e26-8e2b-43d7b50a800e\" (UID: \"a23bdae1-1e2a-4e26-8e2b-43d7b50a800e\") " Oct 31 00:26:15.812262 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:26:15.812206 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a23bdae1-1e2a-4e26-8e2b-43d7b50a800e-istio-data" pod "a23bdae1-1e2a-4e26-8e2b-43d7b50a800e" (UID: "a23bdae1-1e2a-4e26-8e2b-43d7b50a800e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:26:15.812527 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:26:15.812305 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a23bdae1-1e2a-4e26-8e2b-43d7b50a800e-workload-socket" pod "a23bdae1-1e2a-4e26-8e2b-43d7b50a800e" (UID: "a23bdae1-1e2a-4e26-8e2b-43d7b50a800e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:26:15.812527 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:26:15.812385 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a23bdae1-1e2a-4e26-8e2b-43d7b50a800e-istiod-ca-cert" pod "a23bdae1-1e2a-4e26-8e2b-43d7b50a800e" (UID: "a23bdae1-1e2a-4e26-8e2b-43d7b50a800e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:26:15.813050 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:26:15.813026 4927 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a23bdae1-1e2a-4e26-8e2b-43d7b50a800e-workload-certs\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:26:15.813187 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:26:15.813052 4927 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a23bdae1-1e2a-4e26-8e2b-43d7b50a800e-credential-socket\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:26:15.813187 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:26:15.813067 4927 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a23bdae1-1e2a-4e26-8e2b-43d7b50a800e-istio-data\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:26:15.813187 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:26:15.813082 4927 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a23bdae1-1e2a-4e26-8e2b-43d7b50a800e-workload-socket\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:26:15.813187 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:26:15.813096 4927 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a23bdae1-1e2a-4e26-8e2b-43d7b50a800e-istiod-ca-cert\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:26:15.814678 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:26:15.814649 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a23bdae1-1e2a-4e26-8e2b-43d7b50a800e-kube-api-access-86d72" pod "a23bdae1-1e2a-4e26-8e2b-43d7b50a800e" (UID: "a23bdae1-1e2a-4e26-8e2b-43d7b50a800e"). InnerVolumeSpecName "kube-api-access-86d72". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:26:15.814907 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:26:15.814873 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a23bdae1-1e2a-4e26-8e2b-43d7b50a800e-istio-token" pod "a23bdae1-1e2a-4e26-8e2b-43d7b50a800e" (UID: "a23bdae1-1e2a-4e26-8e2b-43d7b50a800e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:26:15.814907 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:26:15.814894 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a23bdae1-1e2a-4e26-8e2b-43d7b50a800e-istio-envoy" pod "a23bdae1-1e2a-4e26-8e2b-43d7b50a800e" (UID: "a23bdae1-1e2a-4e26-8e2b-43d7b50a800e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:26:15.814907 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:26:15.814901 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a23bdae1-1e2a-4e26-8e2b-43d7b50a800e-istio-podinfo" pod "a23bdae1-1e2a-4e26-8e2b-43d7b50a800e" (UID: "a23bdae1-1e2a-4e26-8e2b-43d7b50a800e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:26:15.914259 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:26:15.914222 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-86d72\" (UniqueName: \"kubernetes.io/projected/a23bdae1-1e2a-4e26-8e2b-43d7b50a800e-kube-api-access-86d72\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:26:15.914259 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:26:15.914261 4927 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a23bdae1-1e2a-4e26-8e2b-43d7b50a800e-istio-token\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:26:15.914468 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:26:15.914278 4927 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a23bdae1-1e2a-4e26-8e2b-43d7b50a800e-istio-podinfo\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:26:15.914468 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:26:15.914292 4927 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a23bdae1-1e2a-4e26-8e2b-43d7b50a800e-istio-envoy\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:26:16.034798 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:26:16.034701 4927 generic.go:356] "Generic (PLEG): container finished" podID="a23bdae1-1e2a-4e26-8e2b-43d7b50a800e" containerID="604282d2cb273a5b341a0126f2894ded0f1f72e73c53d2b17248a900ced00ea5" exitCode=0 Oct 31 00:26:16.034798 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:26:16.034753 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-jqc5p" Oct 31 00:26:16.034798 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:26:16.034766 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-jqc5p" event={"ID":"a23bdae1-1e2a-4e26-8e2b-43d7b50a800e","Type":"ContainerDied","Data":"604282d2cb273a5b341a0126f2894ded0f1f72e73c53d2b17248a900ced00ea5"} Oct 31 00:26:16.035097 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:26:16.034821 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-jqc5p" event={"ID":"a23bdae1-1e2a-4e26-8e2b-43d7b50a800e","Type":"ContainerDied","Data":"6926825dfbee21fd260113e9c0aa1c7f14d821b4d30af33e5ee86d8050ffc4f0"} Oct 31 00:26:16.035097 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:26:16.034851 4927 scope.go:122] "RemoveContainer" containerID="604282d2cb273a5b341a0126f2894ded0f1f72e73c53d2b17248a900ced00ea5" Oct 31 00:26:16.047514 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:26:16.046771 4927 scope.go:122] "RemoveContainer" containerID="66fe27523fed3875d511f1effa6f7cdd362683d0e88c130474cb5f5fb93ff4fd" Oct 31 00:26:16.058319 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:26:16.058298 4927 scope.go:122] "RemoveContainer" containerID="604282d2cb273a5b341a0126f2894ded0f1f72e73c53d2b17248a900ced00ea5" Oct 31 00:26:16.058586 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:26:16.058563 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"604282d2cb273a5b341a0126f2894ded0f1f72e73c53d2b17248a900ced00ea5\": container with ID starting with 604282d2cb273a5b341a0126f2894ded0f1f72e73c53d2b17248a900ced00ea5 not found: ID does not exist" containerID="604282d2cb273a5b341a0126f2894ded0f1f72e73c53d2b17248a900ced00ea5" Oct 31 00:26:16.058675 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:26:16.058597 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"604282d2cb273a5b341a0126f2894ded0f1f72e73c53d2b17248a900ced00ea5"} err="failed to get container status \"604282d2cb273a5b341a0126f2894ded0f1f72e73c53d2b17248a900ced00ea5\": rpc error: code = NotFound desc = could not find container \"604282d2cb273a5b341a0126f2894ded0f1f72e73c53d2b17248a900ced00ea5\": container with ID starting with 604282d2cb273a5b341a0126f2894ded0f1f72e73c53d2b17248a900ced00ea5 not found: ID does not exist" Oct 31 00:26:16.058675 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:26:16.058622 4927 scope.go:122] "RemoveContainer" containerID="66fe27523fed3875d511f1effa6f7cdd362683d0e88c130474cb5f5fb93ff4fd" Oct 31 00:26:16.058890 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:26:16.058870 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66fe27523fed3875d511f1effa6f7cdd362683d0e88c130474cb5f5fb93ff4fd\": container with ID starting with 66fe27523fed3875d511f1effa6f7cdd362683d0e88c130474cb5f5fb93ff4fd not found: ID does not exist" containerID="66fe27523fed3875d511f1effa6f7cdd362683d0e88c130474cb5f5fb93ff4fd" Oct 31 00:26:16.058931 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:26:16.058896 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66fe27523fed3875d511f1effa6f7cdd362683d0e88c130474cb5f5fb93ff4fd"} err="failed to get container status \"66fe27523fed3875d511f1effa6f7cdd362683d0e88c130474cb5f5fb93ff4fd\": rpc error: code = NotFound desc = could not find container \"66fe27523fed3875d511f1effa6f7cdd362683d0e88c130474cb5f5fb93ff4fd\": container with ID starting with 66fe27523fed3875d511f1effa6f7cdd362683d0e88c130474cb5f5fb93ff4fd not found: ID does not exist" Oct 31 00:26:16.207058 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:26:16.207025 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-jqc5p"] Oct 31 00:26:16.211524 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:26:16.211496 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-jqc5p"] Oct 31 00:26:17.086984 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:26:17.086948 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a23bdae1-1e2a-4e26-8e2b-43d7b50a800e" path="/var/lib/kubelet/pods/a23bdae1-1e2a-4e26-8e2b-43d7b50a800e/volumes" Oct 31 00:26:18.894531 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:26:18.894491 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:26:19.680026 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:26:19.679982 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sqv69" containerName="extractor" Oct 31 00:26:23.505990 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:26:23.505952 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:26:27.984721 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:26:27.984688 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:26:28.701425 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:26:28.701397 4927 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-pqt2c_6382dfb5-a363-47a2-bc79-0b96f7bec0c3/console-operator/3.log" Oct 31 00:26:29.715138 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:26:29.715107 4927 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-pqt2c_6382dfb5-a363-47a2-bc79-0b96f7bec0c3/console-operator/3.log" Oct 31 00:26:32.508667 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:26:32.508633 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:26:33.435973 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:26:33.435935 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:26:37.060782 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:26:37.060744 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:26:41.649713 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:26:41.649674 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:26:46.133846 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:26:46.133802 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:26:49.679654 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:26:49.679615 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sqv69" containerName="extractor" Oct 31 00:26:50.646854 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:26:50.646795 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:26:55.201040 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:26:55.200999 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:26:59.795148 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:26:59.795112 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:27:03.736382 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:03.736344 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-d687ff-6f87cb4d9c-m557f"] Oct 31 00:27:03.736895 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:03.736609 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d687ff-6f87cb4d9c-m557f" podUID="29148bdd-fd25-4c41-bf85-bfa6dc2489d8" containerName="maas-api" containerID="cri-o://0a417e90b14f56265c300803c8bd93e02dd45a54af137d382980caf0cd25b9be" gracePeriod=30 Oct 31 00:27:03.907392 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:03.907321 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:27:04.042245 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:04.042217 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d687ff-6f87cb4d9c-m557f" Oct 31 00:27:04.114351 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:04.110789 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29148bdd-fd25-4c41-bf85-bfa6dc2489d8-kube-api-access-86nlx\" (UniqueName: \"kubernetes.io/projected/29148bdd-fd25-4c41-bf85-bfa6dc2489d8-kube-api-access-86nlx\") pod \"29148bdd-fd25-4c41-bf85-bfa6dc2489d8\" (UID: \"29148bdd-fd25-4c41-bf85-bfa6dc2489d8\") " Oct 31 00:27:04.114351 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:04.110858 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/29148bdd-fd25-4c41-bf85-bfa6dc2489d8-metrics-certs\" (UniqueName: \"kubernetes.io/secret/29148bdd-fd25-4c41-bf85-bfa6dc2489d8-metrics-certs\") pod \"29148bdd-fd25-4c41-bf85-bfa6dc2489d8\" (UID: \"29148bdd-fd25-4c41-bf85-bfa6dc2489d8\") " Oct 31 00:27:04.114351 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:04.110996 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/29148bdd-fd25-4c41-bf85-bfa6dc2489d8-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/29148bdd-fd25-4c41-bf85-bfa6dc2489d8-maas-api-tls\") pod \"29148bdd-fd25-4c41-bf85-bfa6dc2489d8\" (UID: \"29148bdd-fd25-4c41-bf85-bfa6dc2489d8\") " Oct 31 00:27:04.114351 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:04.111054 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/29148bdd-fd25-4c41-bf85-bfa6dc2489d8-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/29148bdd-fd25-4c41-bf85-bfa6dc2489d8-openshift-service-ca\") pod \"29148bdd-fd25-4c41-bf85-bfa6dc2489d8\" (UID: \"29148bdd-fd25-4c41-bf85-bfa6dc2489d8\") " Oct 31 00:27:04.123907 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:04.123847 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29148bdd-fd25-4c41-bf85-bfa6dc2489d8-kube-api-access-86nlx" pod "29148bdd-fd25-4c41-bf85-bfa6dc2489d8" (UID: "29148bdd-fd25-4c41-bf85-bfa6dc2489d8"). InnerVolumeSpecName "kube-api-access-86nlx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:27:04.136669 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:04.136555 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29148bdd-fd25-4c41-bf85-bfa6dc2489d8-openshift-service-ca" pod "29148bdd-fd25-4c41-bf85-bfa6dc2489d8" (UID: "29148bdd-fd25-4c41-bf85-bfa6dc2489d8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:27:04.160643 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:04.160494 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29148bdd-fd25-4c41-bf85-bfa6dc2489d8-maas-api-tls" pod "29148bdd-fd25-4c41-bf85-bfa6dc2489d8" (UID: "29148bdd-fd25-4c41-bf85-bfa6dc2489d8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:27:04.160643 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:04.160603 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29148bdd-fd25-4c41-bf85-bfa6dc2489d8-metrics-certs" pod "29148bdd-fd25-4c41-bf85-bfa6dc2489d8" (UID: "29148bdd-fd25-4c41-bf85-bfa6dc2489d8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:27:04.213674 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:04.213527 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-86nlx\" (UniqueName: \"kubernetes.io/projected/29148bdd-fd25-4c41-bf85-bfa6dc2489d8-kube-api-access-86nlx\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:27:04.213674 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:04.213565 4927 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/29148bdd-fd25-4c41-bf85-bfa6dc2489d8-metrics-certs\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:27:04.213674 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:04.213580 4927 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/29148bdd-fd25-4c41-bf85-bfa6dc2489d8-maas-api-tls\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:27:04.213674 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:04.213595 4927 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/29148bdd-fd25-4c41-bf85-bfa6dc2489d8-openshift-service-ca\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:27:04.417759 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:04.417726 4927 generic.go:356] "Generic (PLEG): container finished" podID="29148bdd-fd25-4c41-bf85-bfa6dc2489d8" containerID="0a417e90b14f56265c300803c8bd93e02dd45a54af137d382980caf0cd25b9be" exitCode=0 Oct 31 00:27:04.417932 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:04.417785 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d687ff-6f87cb4d9c-m557f" Oct 31 00:27:04.417932 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:04.417810 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d687ff-6f87cb4d9c-m557f" event={"ID":"29148bdd-fd25-4c41-bf85-bfa6dc2489d8","Type":"ContainerDied","Data":"0a417e90b14f56265c300803c8bd93e02dd45a54af137d382980caf0cd25b9be"} Oct 31 00:27:04.417932 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:04.417869 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d687ff-6f87cb4d9c-m557f" event={"ID":"29148bdd-fd25-4c41-bf85-bfa6dc2489d8","Type":"ContainerDied","Data":"e58421fc668a27cc8498b2031025ea611be0b794221940d36045a6fbfaef4a23"} Oct 31 00:27:04.417932 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:04.417901 4927 scope.go:122] "RemoveContainer" containerID="0a417e90b14f56265c300803c8bd93e02dd45a54af137d382980caf0cd25b9be" Oct 31 00:27:04.429178 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:04.429151 4927 scope.go:122] "RemoveContainer" containerID="0a417e90b14f56265c300803c8bd93e02dd45a54af137d382980caf0cd25b9be" Oct 31 00:27:04.429572 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:27:04.429552 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a417e90b14f56265c300803c8bd93e02dd45a54af137d382980caf0cd25b9be\": container with ID starting with 0a417e90b14f56265c300803c8bd93e02dd45a54af137d382980caf0cd25b9be not found: ID does not exist" containerID="0a417e90b14f56265c300803c8bd93e02dd45a54af137d382980caf0cd25b9be" Oct 31 00:27:04.429629 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:04.429580 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a417e90b14f56265c300803c8bd93e02dd45a54af137d382980caf0cd25b9be"} err="failed to get container status \"0a417e90b14f56265c300803c8bd93e02dd45a54af137d382980caf0cd25b9be\": rpc error: code = NotFound desc = could not find container \"0a417e90b14f56265c300803c8bd93e02dd45a54af137d382980caf0cd25b9be\": container with ID starting with 0a417e90b14f56265c300803c8bd93e02dd45a54af137d382980caf0cd25b9be not found: ID does not exist" Oct 31 00:27:04.588722 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:04.588683 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-d687ff-6f87cb4d9c-m557f"] Oct 31 00:27:04.593062 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:04.593032 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-d687ff-6f87cb4d9c-m557f"] Oct 31 00:27:05.086666 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:05.086634 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="29148bdd-fd25-4c41-bf85-bfa6dc2489d8" path="/var/lib/kubelet/pods/29148bdd-fd25-4c41-bf85-bfa6dc2489d8/volumes" Oct 31 00:27:05.266835 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:05.266790 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:27:05.498041 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:05.498003 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:27:05.678954 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:05.678920 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:27:05.759315 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:05.759235 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d687ff-hxrmq"] Oct 31 00:27:05.810323 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:05.810288 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d687ff-hxrmq"] Oct 31 00:27:06.199062 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:06.199033 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:27:07.088605 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:07.088526 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b79791f-2aa2-4daf-8946-d99a837250d3" path="/var/lib/kubelet/pods/8b79791f-2aa2-4daf-8946-d99a837250d3/volumes" Oct 31 00:27:10.306969 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:10.306931 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:27:12.201129 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:12.201087 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:27:12.385254 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:12.385213 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:27:12.590450 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:12.590367 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:27:12.594104 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:12.594080 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-d687ff-openshift-default-5c75fcb6d5-25thf"] Oct 31 00:27:12.594295 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:12.594274 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-d687ff-openshift-default-5c75fcb6d5-25thf" podUID="360cd796-06da-49c1-9bd4-f2f41f04efeb" containerName="istio-proxy" containerID="cri-o://4d2982d9bc716ac015cb59bdf6cfcd1346e09345f5cb2be03e87a4815f52b7da" gracePeriod=30 Oct 31 00:27:13.104340 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:13.104299 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:27:13.560515 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:13.560480 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-34f1ce-tlltn"] Oct 31 00:27:13.561071 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:13.561057 4927 cpu_manager.go:414] "containerMap: removing container" podUID="8844df03-4076-4f22-94bd-49a4514c21ff" containerName="istio-proxy" Oct 31 00:27:13.561071 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:13.561071 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8844df03-4076-4f22-94bd-49a4514c21ff" containerName="istio-proxy" Oct 31 00:27:13.561182 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:13.561088 4927 cpu_manager.go:414] "containerMap: removing container" podUID="31803f91-4f1e-407f-b82e-386fd8a80be8" containerName="istio-proxy" Oct 31 00:27:13.561182 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:13.561094 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31803f91-4f1e-407f-b82e-386fd8a80be8" containerName="istio-proxy" Oct 31 00:27:13.561182 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:13.561112 4927 cpu_manager.go:414] "containerMap: removing container" podUID="8844df03-4076-4f22-94bd-49a4514c21ff" containerName="istio-proxy" Oct 31 00:27:13.561182 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:13.561118 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8844df03-4076-4f22-94bd-49a4514c21ff" containerName="istio-proxy" Oct 31 00:27:13.561182 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:13.561132 4927 cpu_manager.go:414] "containerMap: removing container" podUID="29148bdd-fd25-4c41-bf85-bfa6dc2489d8" containerName="maas-api" Oct 31 00:27:13.561182 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:13.561137 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29148bdd-fd25-4c41-bf85-bfa6dc2489d8" containerName="maas-api" Oct 31 00:27:13.561182 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:13.561149 4927 cpu_manager.go:414] "containerMap: removing container" podUID="e39eb8b7-aaa4-470e-a63d-2b05c14ba5c7" containerName="revoke-keys" Oct 31 00:27:13.561182 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:13.561154 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e39eb8b7-aaa4-470e-a63d-2b05c14ba5c7" containerName="revoke-keys" Oct 31 00:27:13.561182 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:13.561164 4927 cpu_manager.go:414] "containerMap: removing container" podUID="a23bdae1-1e2a-4e26-8e2b-43d7b50a800e" containerName="istio-proxy" Oct 31 00:27:13.561182 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:13.561169 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a23bdae1-1e2a-4e26-8e2b-43d7b50a800e" containerName="istio-proxy" Oct 31 00:27:13.561182 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:13.561180 4927 cpu_manager.go:414] "containerMap: removing container" podUID="a23bdae1-1e2a-4e26-8e2b-43d7b50a800e" containerName="istio-proxy" Oct 31 00:27:13.561182 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:13.561185 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a23bdae1-1e2a-4e26-8e2b-43d7b50a800e" containerName="istio-proxy" Oct 31 00:27:13.561609 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:13.561192 4927 cpu_manager.go:414] "containerMap: removing container" podUID="31803f91-4f1e-407f-b82e-386fd8a80be8" containerName="istio-proxy" Oct 31 00:27:13.561609 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:13.561197 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31803f91-4f1e-407f-b82e-386fd8a80be8" containerName="istio-proxy" Oct 31 00:27:13.561609 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:13.561289 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="8844df03-4076-4f22-94bd-49a4514c21ff" containerName="istio-proxy" Oct 31 00:27:13.561609 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:13.561299 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="31803f91-4f1e-407f-b82e-386fd8a80be8" containerName="istio-proxy" Oct 31 00:27:13.561609 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:13.561307 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="29148bdd-fd25-4c41-bf85-bfa6dc2489d8" containerName="maas-api" Oct 31 00:27:13.561609 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:13.561313 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="e39eb8b7-aaa4-470e-a63d-2b05c14ba5c7" containerName="revoke-keys" Oct 31 00:27:13.561609 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:13.561320 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="a23bdae1-1e2a-4e26-8e2b-43d7b50a800e" containerName="istio-proxy" Oct 31 00:27:13.561609 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:13.561343 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="a23bdae1-1e2a-4e26-8e2b-43d7b50a800e" containerName="istio-proxy" Oct 31 00:27:13.561609 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:13.561353 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="31803f91-4f1e-407f-b82e-386fd8a80be8" containerName="istio-proxy" Oct 31 00:27:13.565186 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:13.565162 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-34f1ce-tlltn" Oct 31 00:27:13.622561 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:13.622525 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-34f1ce-tlltn"] Oct 31 00:27:13.667866 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:13.667834 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:27:13.668036 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:13.667937 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4fdzs\"" Oct 31 00:27:13.745348 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:13.745294 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/505b964b-07ad-4e68-bbf9-4ce388e91aa0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-34f1ce-tlltn\" (UID: \"505b964b-07ad-4e68-bbf9-4ce388e91aa0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-34f1ce-tlltn" Oct 31 00:27:13.745563 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:13.745473 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/505b964b-07ad-4e68-bbf9-4ce388e91aa0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-34f1ce-tlltn\" (UID: \"505b964b-07ad-4e68-bbf9-4ce388e91aa0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-34f1ce-tlltn" Oct 31 00:27:13.745563 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:13.745513 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwbds\" (UniqueName: \"kubernetes.io/projected/505b964b-07ad-4e68-bbf9-4ce388e91aa0-kube-api-access-dwbds\") pod \"maas-api-revoke-keys-e2e-worker-w4-34f1ce-tlltn\" (UID: \"505b964b-07ad-4e68-bbf9-4ce388e91aa0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-34f1ce-tlltn" Oct 31 00:27:13.847001 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:13.846908 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/505b964b-07ad-4e68-bbf9-4ce388e91aa0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-34f1ce-tlltn\" (UID: \"505b964b-07ad-4e68-bbf9-4ce388e91aa0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-34f1ce-tlltn" Oct 31 00:27:13.847177 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:13.847042 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/505b964b-07ad-4e68-bbf9-4ce388e91aa0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-34f1ce-tlltn\" (UID: \"505b964b-07ad-4e68-bbf9-4ce388e91aa0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-34f1ce-tlltn" Oct 31 00:27:13.847177 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:13.847092 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwbds\" (UniqueName: \"kubernetes.io/projected/505b964b-07ad-4e68-bbf9-4ce388e91aa0-kube-api-access-dwbds\") pod \"maas-api-revoke-keys-e2e-worker-w4-34f1ce-tlltn\" (UID: \"505b964b-07ad-4e68-bbf9-4ce388e91aa0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-34f1ce-tlltn" Oct 31 00:27:13.847765 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:13.847741 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/505b964b-07ad-4e68-bbf9-4ce388e91aa0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-34f1ce-tlltn\" (UID: \"505b964b-07ad-4e68-bbf9-4ce388e91aa0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-34f1ce-tlltn" Oct 31 00:27:13.849463 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:13.849439 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/505b964b-07ad-4e68-bbf9-4ce388e91aa0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-34f1ce-tlltn\" (UID: \"505b964b-07ad-4e68-bbf9-4ce388e91aa0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-34f1ce-tlltn" Oct 31 00:27:13.910099 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:13.910065 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwbds\" (UniqueName: \"kubernetes.io/projected/505b964b-07ad-4e68-bbf9-4ce388e91aa0-kube-api-access-dwbds\") pod \"maas-api-revoke-keys-e2e-worker-w4-34f1ce-tlltn\" (UID: \"505b964b-07ad-4e68-bbf9-4ce388e91aa0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-34f1ce-tlltn" Oct 31 00:27:14.180499 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:14.180466 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-34f1ce-tlltn" Oct 31 00:27:14.570905 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:14.570868 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-34f1ce-tlltn"] Oct 31 00:27:14.571213 ip-10-0-141-180 kubenswrapper[4927]: W1031 00:27:14.570900 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod505b964b_07ad_4e68_bbf9_4ce388e91aa0.slice/crio-1aeeb4456b2920d0b45aca2620d185563414712195d266064341edf246dc8510 WatchSource:0}: Error finding container 1aeeb4456b2920d0b45aca2620d185563414712195d266064341edf246dc8510: Status 404 returned error can't find the container with id 1aeeb4456b2920d0b45aca2620d185563414712195d266064341edf246dc8510 Oct 31 00:27:15.069733 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:15.069651 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:27:15.508714 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:15.508677 4927 generic.go:356] "Generic (PLEG): container finished" podID="505b964b-07ad-4e68-bbf9-4ce388e91aa0" containerID="902c792a2f697e696ce631afd9a8da776f53ebff8a73e2fb62b7ae1bfe90148a" exitCode=0 Oct 31 00:27:15.508860 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:15.508818 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-34f1ce-tlltn" event={"ID":"505b964b-07ad-4e68-bbf9-4ce388e91aa0","Type":"ContainerDied","Data":"902c792a2f697e696ce631afd9a8da776f53ebff8a73e2fb62b7ae1bfe90148a"} Oct 31 00:27:15.508910 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:15.508869 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-34f1ce-tlltn" event={"ID":"505b964b-07ad-4e68-bbf9-4ce388e91aa0","Type":"ContainerStarted","Data":"1aeeb4456b2920d0b45aca2620d185563414712195d266064341edf246dc8510"} Oct 31 00:27:16.688198 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:16.688177 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-34f1ce-tlltn" Oct 31 00:27:16.781282 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:16.781192 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/505b964b-07ad-4e68-bbf9-4ce388e91aa0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/505b964b-07ad-4e68-bbf9-4ce388e91aa0-maas-api-xks-ca\") pod \"505b964b-07ad-4e68-bbf9-4ce388e91aa0\" (UID: \"505b964b-07ad-4e68-bbf9-4ce388e91aa0\") " Oct 31 00:27:16.781282 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:16.781262 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/505b964b-07ad-4e68-bbf9-4ce388e91aa0-kube-api-access-dwbds\" (UniqueName: \"kubernetes.io/projected/505b964b-07ad-4e68-bbf9-4ce388e91aa0-kube-api-access-dwbds\") pod \"505b964b-07ad-4e68-bbf9-4ce388e91aa0\" (UID: \"505b964b-07ad-4e68-bbf9-4ce388e91aa0\") " Oct 31 00:27:16.781496 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:16.781326 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/505b964b-07ad-4e68-bbf9-4ce388e91aa0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/505b964b-07ad-4e68-bbf9-4ce388e91aa0-maas-api-openshift-ca\") pod \"505b964b-07ad-4e68-bbf9-4ce388e91aa0\" (UID: \"505b964b-07ad-4e68-bbf9-4ce388e91aa0\") " Oct 31 00:27:16.781734 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:16.781706 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/505b964b-07ad-4e68-bbf9-4ce388e91aa0-maas-api-openshift-ca" pod "505b964b-07ad-4e68-bbf9-4ce388e91aa0" (UID: "505b964b-07ad-4e68-bbf9-4ce388e91aa0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:27:16.783977 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:16.783945 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/505b964b-07ad-4e68-bbf9-4ce388e91aa0-maas-api-xks-ca" pod "505b964b-07ad-4e68-bbf9-4ce388e91aa0" (UID: "505b964b-07ad-4e68-bbf9-4ce388e91aa0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:27:16.784076 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:16.783955 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/505b964b-07ad-4e68-bbf9-4ce388e91aa0-kube-api-access-dwbds" pod "505b964b-07ad-4e68-bbf9-4ce388e91aa0" (UID: "505b964b-07ad-4e68-bbf9-4ce388e91aa0"). InnerVolumeSpecName "kube-api-access-dwbds". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:27:16.882744 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:16.882709 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dwbds\" (UniqueName: \"kubernetes.io/projected/505b964b-07ad-4e68-bbf9-4ce388e91aa0-kube-api-access-dwbds\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:27:16.882744 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:16.882741 4927 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/505b964b-07ad-4e68-bbf9-4ce388e91aa0-maas-api-openshift-ca\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:27:16.882953 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:16.882755 4927 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/505b964b-07ad-4e68-bbf9-4ce388e91aa0-maas-api-xks-ca\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:27:17.525981 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:17.525949 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-34f1ce-tlltn" Oct 31 00:27:17.526177 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:17.525987 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-34f1ce-tlltn" event={"ID":"505b964b-07ad-4e68-bbf9-4ce388e91aa0","Type":"ContainerDied","Data":"1aeeb4456b2920d0b45aca2620d185563414712195d266064341edf246dc8510"} Oct 31 00:27:17.526177 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:17.526035 4927 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1aeeb4456b2920d0b45aca2620d185563414712195d266064341edf246dc8510" Oct 31 00:27:17.901699 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:17.901660 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-d687ff-openshift-default-5c75fcb6d5-25thf" Oct 31 00:27:17.994733 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:17.994700 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/360cd796-06da-49c1-9bd4-f2f41f04efeb-kube-api-access-pfz6l\" (UniqueName: \"kubernetes.io/projected/360cd796-06da-49c1-9bd4-f2f41f04efeb-kube-api-access-pfz6l\") pod \"360cd796-06da-49c1-9bd4-f2f41f04efeb\" (UID: \"360cd796-06da-49c1-9bd4-f2f41f04efeb\") " Oct 31 00:27:17.994910 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:17.994745 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/360cd796-06da-49c1-9bd4-f2f41f04efeb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/360cd796-06da-49c1-9bd4-f2f41f04efeb-workload-socket\") pod \"360cd796-06da-49c1-9bd4-f2f41f04efeb\" (UID: \"360cd796-06da-49c1-9bd4-f2f41f04efeb\") " Oct 31 00:27:17.994910 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:17.994774 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/360cd796-06da-49c1-9bd4-f2f41f04efeb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/360cd796-06da-49c1-9bd4-f2f41f04efeb-istiod-ca-cert\") pod \"360cd796-06da-49c1-9bd4-f2f41f04efeb\" (UID: \"360cd796-06da-49c1-9bd4-f2f41f04efeb\") " Oct 31 00:27:17.994910 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:17.994805 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/360cd796-06da-49c1-9bd4-f2f41f04efeb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/360cd796-06da-49c1-9bd4-f2f41f04efeb-workload-certs\") pod \"360cd796-06da-49c1-9bd4-f2f41f04efeb\" (UID: \"360cd796-06da-49c1-9bd4-f2f41f04efeb\") " Oct 31 00:27:17.994910 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:17.994889 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/360cd796-06da-49c1-9bd4-f2f41f04efeb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/360cd796-06da-49c1-9bd4-f2f41f04efeb-istio-envoy\") pod \"360cd796-06da-49c1-9bd4-f2f41f04efeb\" (UID: \"360cd796-06da-49c1-9bd4-f2f41f04efeb\") " Oct 31 00:27:17.994910 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:17.994909 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/360cd796-06da-49c1-9bd4-f2f41f04efeb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/360cd796-06da-49c1-9bd4-f2f41f04efeb-istio-data\") pod \"360cd796-06da-49c1-9bd4-f2f41f04efeb\" (UID: \"360cd796-06da-49c1-9bd4-f2f41f04efeb\") " Oct 31 00:27:17.995204 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:17.994956 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/360cd796-06da-49c1-9bd4-f2f41f04efeb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/360cd796-06da-49c1-9bd4-f2f41f04efeb-istio-podinfo\") pod \"360cd796-06da-49c1-9bd4-f2f41f04efeb\" (UID: \"360cd796-06da-49c1-9bd4-f2f41f04efeb\") " Oct 31 00:27:17.995204 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:17.995023 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/360cd796-06da-49c1-9bd4-f2f41f04efeb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/360cd796-06da-49c1-9bd4-f2f41f04efeb-credential-socket\") pod \"360cd796-06da-49c1-9bd4-f2f41f04efeb\" (UID: \"360cd796-06da-49c1-9bd4-f2f41f04efeb\") " Oct 31 00:27:17.995204 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:17.995044 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/360cd796-06da-49c1-9bd4-f2f41f04efeb-workload-socket" pod "360cd796-06da-49c1-9bd4-f2f41f04efeb" (UID: "360cd796-06da-49c1-9bd4-f2f41f04efeb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:27:17.995204 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:17.995132 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/360cd796-06da-49c1-9bd4-f2f41f04efeb-istio-token\" (UniqueName: \"kubernetes.io/projected/360cd796-06da-49c1-9bd4-f2f41f04efeb-istio-token\") pod \"360cd796-06da-49c1-9bd4-f2f41f04efeb\" (UID: \"360cd796-06da-49c1-9bd4-f2f41f04efeb\") " Oct 31 00:27:17.995204 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:17.995195 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/360cd796-06da-49c1-9bd4-f2f41f04efeb-istiod-ca-cert" pod "360cd796-06da-49c1-9bd4-f2f41f04efeb" (UID: "360cd796-06da-49c1-9bd4-f2f41f04efeb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:27:17.995532 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:17.995378 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/360cd796-06da-49c1-9bd4-f2f41f04efeb-istio-data" pod "360cd796-06da-49c1-9bd4-f2f41f04efeb" (UID: "360cd796-06da-49c1-9bd4-f2f41f04efeb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:27:17.995532 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:17.995397 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/360cd796-06da-49c1-9bd4-f2f41f04efeb-credential-socket" pod "360cd796-06da-49c1-9bd4-f2f41f04efeb" (UID: "360cd796-06da-49c1-9bd4-f2f41f04efeb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:27:17.995643 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:17.995528 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/360cd796-06da-49c1-9bd4-f2f41f04efeb-workload-certs" pod "360cd796-06da-49c1-9bd4-f2f41f04efeb" (UID: "360cd796-06da-49c1-9bd4-f2f41f04efeb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:27:17.995773 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:17.995747 4927 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/360cd796-06da-49c1-9bd4-f2f41f04efeb-workload-socket\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:27:17.995773 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:17.995775 4927 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/360cd796-06da-49c1-9bd4-f2f41f04efeb-istiod-ca-cert\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:27:17.995987 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:17.995786 4927 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/360cd796-06da-49c1-9bd4-f2f41f04efeb-workload-certs\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:27:17.995987 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:17.995794 4927 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/360cd796-06da-49c1-9bd4-f2f41f04efeb-istio-data\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:27:17.995987 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:17.995803 4927 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/360cd796-06da-49c1-9bd4-f2f41f04efeb-credential-socket\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:27:17.997390 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:17.997362 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/360cd796-06da-49c1-9bd4-f2f41f04efeb-istio-podinfo" pod "360cd796-06da-49c1-9bd4-f2f41f04efeb" (UID: "360cd796-06da-49c1-9bd4-f2f41f04efeb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:27:17.997503 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:17.997458 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/360cd796-06da-49c1-9bd4-f2f41f04efeb-istio-token" pod "360cd796-06da-49c1-9bd4-f2f41f04efeb" (UID: "360cd796-06da-49c1-9bd4-f2f41f04efeb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:27:17.997574 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:17.997548 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/360cd796-06da-49c1-9bd4-f2f41f04efeb-kube-api-access-pfz6l" pod "360cd796-06da-49c1-9bd4-f2f41f04efeb" (UID: "360cd796-06da-49c1-9bd4-f2f41f04efeb"). InnerVolumeSpecName "kube-api-access-pfz6l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:27:17.997631 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:17.997586 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/360cd796-06da-49c1-9bd4-f2f41f04efeb-istio-envoy" pod "360cd796-06da-49c1-9bd4-f2f41f04efeb" (UID: "360cd796-06da-49c1-9bd4-f2f41f04efeb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:27:18.096731 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:18.096695 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pfz6l\" (UniqueName: \"kubernetes.io/projected/360cd796-06da-49c1-9bd4-f2f41f04efeb-kube-api-access-pfz6l\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:27:18.096731 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:18.096728 4927 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/360cd796-06da-49c1-9bd4-f2f41f04efeb-istio-envoy\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:27:18.096954 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:18.096743 4927 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/360cd796-06da-49c1-9bd4-f2f41f04efeb-istio-podinfo\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:27:18.096954 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:18.096758 4927 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/360cd796-06da-49c1-9bd4-f2f41f04efeb-istio-token\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:27:18.157593 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:18.157563 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:27:18.536276 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:18.536179 4927 generic.go:356] "Generic (PLEG): container finished" podID="360cd796-06da-49c1-9bd4-f2f41f04efeb" containerID="4d2982d9bc716ac015cb59bdf6cfcd1346e09345f5cb2be03e87a4815f52b7da" exitCode=0 Oct 31 00:27:18.536276 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:18.536233 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-d687ff-openshift-default-5c75fcb6d5-25thf" Oct 31 00:27:18.536276 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:18.536247 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-d687ff-openshift-default-5c75fcb6d5-25thf" event={"ID":"360cd796-06da-49c1-9bd4-f2f41f04efeb","Type":"ContainerDied","Data":"4d2982d9bc716ac015cb59bdf6cfcd1346e09345f5cb2be03e87a4815f52b7da"} Oct 31 00:27:18.536601 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:18.536290 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-d687ff-openshift-default-5c75fcb6d5-25thf" event={"ID":"360cd796-06da-49c1-9bd4-f2f41f04efeb","Type":"ContainerDied","Data":"907068bcecaf2ef9dff3d90652492e2d0faf0e0877159994588f132d21d9099a"} Oct 31 00:27:18.536601 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:18.536311 4927 scope.go:122] "RemoveContainer" containerID="4d2982d9bc716ac015cb59bdf6cfcd1346e09345f5cb2be03e87a4815f52b7da" Oct 31 00:27:18.548739 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:18.548719 4927 scope.go:122] "RemoveContainer" containerID="402a284d6077e106734bc3fd7369626a1d28550adc4df5e45092d3862efc2cf8" Oct 31 00:27:18.559151 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:18.559129 4927 scope.go:122] "RemoveContainer" containerID="4d2982d9bc716ac015cb59bdf6cfcd1346e09345f5cb2be03e87a4815f52b7da" Oct 31 00:27:18.559476 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:27:18.559451 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d2982d9bc716ac015cb59bdf6cfcd1346e09345f5cb2be03e87a4815f52b7da\": container with ID starting with 4d2982d9bc716ac015cb59bdf6cfcd1346e09345f5cb2be03e87a4815f52b7da not found: ID does not exist" containerID="4d2982d9bc716ac015cb59bdf6cfcd1346e09345f5cb2be03e87a4815f52b7da" Oct 31 00:27:18.559568 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:18.559483 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d2982d9bc716ac015cb59bdf6cfcd1346e09345f5cb2be03e87a4815f52b7da"} err="failed to get container status \"4d2982d9bc716ac015cb59bdf6cfcd1346e09345f5cb2be03e87a4815f52b7da\": rpc error: code = NotFound desc = could not find container \"4d2982d9bc716ac015cb59bdf6cfcd1346e09345f5cb2be03e87a4815f52b7da\": container with ID starting with 4d2982d9bc716ac015cb59bdf6cfcd1346e09345f5cb2be03e87a4815f52b7da not found: ID does not exist" Oct 31 00:27:18.559568 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:18.559501 4927 scope.go:122] "RemoveContainer" containerID="402a284d6077e106734bc3fd7369626a1d28550adc4df5e45092d3862efc2cf8" Oct 31 00:27:18.559782 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:27:18.559762 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"402a284d6077e106734bc3fd7369626a1d28550adc4df5e45092d3862efc2cf8\": container with ID starting with 402a284d6077e106734bc3fd7369626a1d28550adc4df5e45092d3862efc2cf8 not found: ID does not exist" containerID="402a284d6077e106734bc3fd7369626a1d28550adc4df5e45092d3862efc2cf8" Oct 31 00:27:18.559828 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:18.559788 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"402a284d6077e106734bc3fd7369626a1d28550adc4df5e45092d3862efc2cf8"} err="failed to get container status \"402a284d6077e106734bc3fd7369626a1d28550adc4df5e45092d3862efc2cf8\": rpc error: code = NotFound desc = could not find container \"402a284d6077e106734bc3fd7369626a1d28550adc4df5e45092d3862efc2cf8\": container with ID starting with 402a284d6077e106734bc3fd7369626a1d28550adc4df5e45092d3862efc2cf8 not found: ID does not exist" Oct 31 00:27:18.707567 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:18.707532 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-d687ff-openshift-default-5c75fcb6d5-25thf"] Oct 31 00:27:18.759160 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:18.759123 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-d687ff-openshift-default-5c75fcb6d5-25thf"] Oct 31 00:27:19.087009 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:19.086973 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="360cd796-06da-49c1-9bd4-f2f41f04efeb" path="/var/lib/kubelet/pods/360cd796-06da-49c1-9bd4-f2f41f04efeb/volumes" Oct 31 00:27:19.679627 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:27:19.679590 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sqv69" containerName="extractor" Oct 31 00:27:19.988562 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:19.988514 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:27:24.689101 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:24.689068 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:27:25.324150 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:25.324102 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:27:25.524921 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:25.524880 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:27:27.653420 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:27.653382 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-jpjkl"] Oct 31 00:27:27.653936 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:27.653656 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-jpjkl" podUID="334942ae-4532-436c-b274-532b56c035ec" containerName="istio-proxy" containerID="cri-o://f9740f71a450a8d486b1459a6121a8eb91adab0a8dd81fe391c0f9d113468e7b" gracePeriod=30 Oct 31 00:27:27.869991 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:27.869956 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:27:28.799567 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:28.799528 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-517d808e-6ltzk"] Oct 31 00:27:28.800104 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:28.800091 4927 cpu_manager.go:414] "containerMap: removing container" podUID="360cd796-06da-49c1-9bd4-f2f41f04efeb" containerName="istio-proxy" Oct 31 00:27:28.800156 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:28.800105 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="360cd796-06da-49c1-9bd4-f2f41f04efeb" containerName="istio-proxy" Oct 31 00:27:28.800156 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:28.800118 4927 cpu_manager.go:414] "containerMap: removing container" podUID="505b964b-07ad-4e68-bbf9-4ce388e91aa0" containerName="revoke-keys" Oct 31 00:27:28.800156 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:28.800123 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="505b964b-07ad-4e68-bbf9-4ce388e91aa0" containerName="revoke-keys" Oct 31 00:27:28.800279 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:28.800268 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="8844df03-4076-4f22-94bd-49a4514c21ff" containerName="istio-proxy" Oct 31 00:27:28.800320 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:28.800283 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="360cd796-06da-49c1-9bd4-f2f41f04efeb" containerName="istio-proxy" Oct 31 00:27:28.800320 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:28.800293 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="360cd796-06da-49c1-9bd4-f2f41f04efeb" containerName="istio-proxy" Oct 31 00:27:28.800320 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:28.800301 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="505b964b-07ad-4e68-bbf9-4ce388e91aa0" containerName="revoke-keys" Oct 31 00:27:28.803912 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:28.803893 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-517d808e-6ltzk" Oct 31 00:27:28.845075 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:28.845045 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9r8j\" (UniqueName: \"kubernetes.io/projected/e5f5046d-b709-4aed-9cd8-0d480dc2960e-kube-api-access-j9r8j\") pod \"maas-api-revoke-sub-517d808e-6ltzk\" (UID: \"e5f5046d-b709-4aed-9cd8-0d480dc2960e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-517d808e-6ltzk" Oct 31 00:27:28.845205 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:28.845103 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5f5046d-b709-4aed-9cd8-0d480dc2960e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-517d808e-6ltzk\" (UID: \"e5f5046d-b709-4aed-9cd8-0d480dc2960e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-517d808e-6ltzk" Oct 31 00:27:28.845205 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:28.845157 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5f5046d-b709-4aed-9cd8-0d480dc2960e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-517d808e-6ltzk\" (UID: \"e5f5046d-b709-4aed-9cd8-0d480dc2960e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-517d808e-6ltzk" Oct 31 00:27:28.857947 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:28.857614 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:27:28.908769 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:28.908729 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4fdzs\"" Oct 31 00:27:28.912429 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:28.912406 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-517d808e-6ltzk"] Oct 31 00:27:28.946022 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:28.945994 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9r8j\" (UniqueName: \"kubernetes.io/projected/e5f5046d-b709-4aed-9cd8-0d480dc2960e-kube-api-access-j9r8j\") pod \"maas-api-revoke-sub-517d808e-6ltzk\" (UID: \"e5f5046d-b709-4aed-9cd8-0d480dc2960e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-517d808e-6ltzk" Oct 31 00:27:28.946174 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:28.946050 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5f5046d-b709-4aed-9cd8-0d480dc2960e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-517d808e-6ltzk\" (UID: \"e5f5046d-b709-4aed-9cd8-0d480dc2960e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-517d808e-6ltzk" Oct 31 00:27:28.946174 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:28.946069 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5f5046d-b709-4aed-9cd8-0d480dc2960e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-517d808e-6ltzk\" (UID: \"e5f5046d-b709-4aed-9cd8-0d480dc2960e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-517d808e-6ltzk" Oct 31 00:27:28.946787 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:28.946767 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5f5046d-b709-4aed-9cd8-0d480dc2960e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-517d808e-6ltzk\" (UID: \"e5f5046d-b709-4aed-9cd8-0d480dc2960e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-517d808e-6ltzk" Oct 31 00:27:28.948538 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:28.948515 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5f5046d-b709-4aed-9cd8-0d480dc2960e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-517d808e-6ltzk\" (UID: \"e5f5046d-b709-4aed-9cd8-0d480dc2960e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-517d808e-6ltzk" Oct 31 00:27:29.008672 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:29.008631 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9r8j\" (UniqueName: \"kubernetes.io/projected/e5f5046d-b709-4aed-9cd8-0d480dc2960e-kube-api-access-j9r8j\") pod \"maas-api-revoke-sub-517d808e-6ltzk\" (UID: \"e5f5046d-b709-4aed-9cd8-0d480dc2960e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-517d808e-6ltzk" Oct 31 00:27:29.118093 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:29.118004 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-517d808e-6ltzk" Oct 31 00:27:29.399080 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:29.398998 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:27:29.498058 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:29.498018 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-517d808e-6ltzk"] Oct 31 00:27:29.499534 ip-10-0-141-180 kubenswrapper[4927]: W1031 00:27:29.499501 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5f5046d_b709_4aed_9cd8_0d480dc2960e.slice/crio-3408aa08bd796a5a3351a369ab7f32f2a5fa52c869cc3773b4e88c4b35eee2a9 WatchSource:0}: Error finding container 3408aa08bd796a5a3351a369ab7f32f2a5fa52c869cc3773b4e88c4b35eee2a9: Status 404 returned error can't find the container with id 3408aa08bd796a5a3351a369ab7f32f2a5fa52c869cc3773b4e88c4b35eee2a9 Oct 31 00:27:29.626204 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:29.626169 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-517d808e-6ltzk" event={"ID":"e5f5046d-b709-4aed-9cd8-0d480dc2960e","Type":"ContainerStarted","Data":"3408aa08bd796a5a3351a369ab7f32f2a5fa52c869cc3773b4e88c4b35eee2a9"} Oct 31 00:27:30.636248 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:30.636207 4927 generic.go:356] "Generic (PLEG): container finished" podID="e5f5046d-b709-4aed-9cd8-0d480dc2960e" containerID="5d032e6fb419be7d3db411a038e3db084f8bdfc111084bc9b03c67501118cb17" exitCode=0 Oct 31 00:27:30.636754 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:30.636291 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-517d808e-6ltzk" event={"ID":"e5f5046d-b709-4aed-9cd8-0d480dc2960e","Type":"ContainerDied","Data":"5d032e6fb419be7d3db411a038e3db084f8bdfc111084bc9b03c67501118cb17"} Oct 31 00:27:31.797928 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:31.797901 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-517d808e-6ltzk" Oct 31 00:27:31.884700 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:31.884666 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e5f5046d-b709-4aed-9cd8-0d480dc2960e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5f5046d-b709-4aed-9cd8-0d480dc2960e-maas-api-openshift-ca\") pod \"e5f5046d-b709-4aed-9cd8-0d480dc2960e\" (UID: \"e5f5046d-b709-4aed-9cd8-0d480dc2960e\") " Oct 31 00:27:31.884876 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:31.884724 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e5f5046d-b709-4aed-9cd8-0d480dc2960e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5f5046d-b709-4aed-9cd8-0d480dc2960e-maas-api-xks-ca\") pod \"e5f5046d-b709-4aed-9cd8-0d480dc2960e\" (UID: \"e5f5046d-b709-4aed-9cd8-0d480dc2960e\") " Oct 31 00:27:31.884876 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:31.884813 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5f5046d-b709-4aed-9cd8-0d480dc2960e-kube-api-access-j9r8j\" (UniqueName: \"kubernetes.io/projected/e5f5046d-b709-4aed-9cd8-0d480dc2960e-kube-api-access-j9r8j\") pod \"e5f5046d-b709-4aed-9cd8-0d480dc2960e\" (UID: \"e5f5046d-b709-4aed-9cd8-0d480dc2960e\") " Oct 31 00:27:31.885085 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:31.885051 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5f5046d-b709-4aed-9cd8-0d480dc2960e-maas-api-openshift-ca" pod "e5f5046d-b709-4aed-9cd8-0d480dc2960e" (UID: "e5f5046d-b709-4aed-9cd8-0d480dc2960e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:27:31.885272 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:31.885251 4927 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5f5046d-b709-4aed-9cd8-0d480dc2960e-maas-api-openshift-ca\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:27:31.886928 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:31.886884 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e5f5046d-b709-4aed-9cd8-0d480dc2960e-maas-api-xks-ca" pod "e5f5046d-b709-4aed-9cd8-0d480dc2960e" (UID: "e5f5046d-b709-4aed-9cd8-0d480dc2960e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:27:31.887024 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:31.886996 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5f5046d-b709-4aed-9cd8-0d480dc2960e-kube-api-access-j9r8j" pod "e5f5046d-b709-4aed-9cd8-0d480dc2960e" (UID: "e5f5046d-b709-4aed-9cd8-0d480dc2960e"). InnerVolumeSpecName "kube-api-access-j9r8j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:27:31.986132 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:31.986094 4927 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5f5046d-b709-4aed-9cd8-0d480dc2960e-maas-api-xks-ca\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:27:31.986132 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:31.986130 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j9r8j\" (UniqueName: \"kubernetes.io/projected/e5f5046d-b709-4aed-9cd8-0d480dc2960e-kube-api-access-j9r8j\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:27:32.654868 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:32.654825 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-517d808e-6ltzk" Oct 31 00:27:32.654868 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:32.654851 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-517d808e-6ltzk" event={"ID":"e5f5046d-b709-4aed-9cd8-0d480dc2960e","Type":"ContainerDied","Data":"3408aa08bd796a5a3351a369ab7f32f2a5fa52c869cc3773b4e88c4b35eee2a9"} Oct 31 00:27:32.655155 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:32.654905 4927 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3408aa08bd796a5a3351a369ab7f32f2a5fa52c869cc3773b4e88c4b35eee2a9" Oct 31 00:27:32.926373 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:32.926343 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-jpjkl" Oct 31 00:27:32.998310 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:32.998275 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/334942ae-4532-436c-b274-532b56c035ec-istio-token\" (UniqueName: \"kubernetes.io/projected/334942ae-4532-436c-b274-532b56c035ec-istio-token\") pod \"334942ae-4532-436c-b274-532b56c035ec\" (UID: \"334942ae-4532-436c-b274-532b56c035ec\") " Oct 31 00:27:32.998310 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:32.998322 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/334942ae-4532-436c-b274-532b56c035ec-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/334942ae-4532-436c-b274-532b56c035ec-istio-data\") pod \"334942ae-4532-436c-b274-532b56c035ec\" (UID: \"334942ae-4532-436c-b274-532b56c035ec\") " Oct 31 00:27:32.998617 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:32.998388 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/334942ae-4532-436c-b274-532b56c035ec-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/334942ae-4532-436c-b274-532b56c035ec-credential-socket\") pod \"334942ae-4532-436c-b274-532b56c035ec\" (UID: \"334942ae-4532-436c-b274-532b56c035ec\") " Oct 31 00:27:32.998617 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:32.998423 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/334942ae-4532-436c-b274-532b56c035ec-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/334942ae-4532-436c-b274-532b56c035ec-istio-envoy\") pod \"334942ae-4532-436c-b274-532b56c035ec\" (UID: \"334942ae-4532-436c-b274-532b56c035ec\") " Oct 31 00:27:32.998617 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:32.998499 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/334942ae-4532-436c-b274-532b56c035ec-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/334942ae-4532-436c-b274-532b56c035ec-workload-certs\") pod \"334942ae-4532-436c-b274-532b56c035ec\" (UID: \"334942ae-4532-436c-b274-532b56c035ec\") " Oct 31 00:27:32.998617 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:32.998595 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/334942ae-4532-436c-b274-532b56c035ec-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/334942ae-4532-436c-b274-532b56c035ec-istiod-ca-cert\") pod \"334942ae-4532-436c-b274-532b56c035ec\" (UID: \"334942ae-4532-436c-b274-532b56c035ec\") " Oct 31 00:27:32.998617 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:32.998621 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/334942ae-4532-436c-b274-532b56c035ec-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/334942ae-4532-436c-b274-532b56c035ec-workload-socket\") pod \"334942ae-4532-436c-b274-532b56c035ec\" (UID: \"334942ae-4532-436c-b274-532b56c035ec\") " Oct 31 00:27:32.998617 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:32.998657 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/334942ae-4532-436c-b274-532b56c035ec-kube-api-access-5xc6h\" (UniqueName: \"kubernetes.io/projected/334942ae-4532-436c-b274-532b56c035ec-kube-api-access-5xc6h\") pod \"334942ae-4532-436c-b274-532b56c035ec\" (UID: \"334942ae-4532-436c-b274-532b56c035ec\") " Oct 31 00:27:32.998617 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:32.998677 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/334942ae-4532-436c-b274-532b56c035ec-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/334942ae-4532-436c-b274-532b56c035ec-istio-podinfo\") pod \"334942ae-4532-436c-b274-532b56c035ec\" (UID: \"334942ae-4532-436c-b274-532b56c035ec\") " Oct 31 00:27:32.998617 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:32.998743 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/334942ae-4532-436c-b274-532b56c035ec-istio-data" pod "334942ae-4532-436c-b274-532b56c035ec" (UID: "334942ae-4532-436c-b274-532b56c035ec"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:27:32.998617 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:32.998784 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/334942ae-4532-436c-b274-532b56c035ec-credential-socket" pod "334942ae-4532-436c-b274-532b56c035ec" (UID: "334942ae-4532-436c-b274-532b56c035ec"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:27:32.998617 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:32.998794 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/334942ae-4532-436c-b274-532b56c035ec-workload-certs" pod "334942ae-4532-436c-b274-532b56c035ec" (UID: "334942ae-4532-436c-b274-532b56c035ec"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:27:32.998617 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:32.998987 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/334942ae-4532-436c-b274-532b56c035ec-workload-socket" pod "334942ae-4532-436c-b274-532b56c035ec" (UID: "334942ae-4532-436c-b274-532b56c035ec"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:27:32.998617 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:32.999017 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/334942ae-4532-436c-b274-532b56c035ec-istiod-ca-cert" pod "334942ae-4532-436c-b274-532b56c035ec" (UID: "334942ae-4532-436c-b274-532b56c035ec"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:27:32.999667 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:32.999157 4927 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/334942ae-4532-436c-b274-532b56c035ec-workload-certs\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:27:32.999667 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:32.999180 4927 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/334942ae-4532-436c-b274-532b56c035ec-istiod-ca-cert\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:27:32.999667 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:32.999194 4927 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/334942ae-4532-436c-b274-532b56c035ec-workload-socket\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:27:32.999667 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:32.999206 4927 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/334942ae-4532-436c-b274-532b56c035ec-istio-data\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:27:32.999667 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:32.999219 4927 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/334942ae-4532-436c-b274-532b56c035ec-credential-socket\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:27:33.000717 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:33.000696 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/334942ae-4532-436c-b274-532b56c035ec-istio-envoy" pod "334942ae-4532-436c-b274-532b56c035ec" (UID: "334942ae-4532-436c-b274-532b56c035ec"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:27:33.001047 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:33.001021 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/334942ae-4532-436c-b274-532b56c035ec-kube-api-access-5xc6h" pod "334942ae-4532-436c-b274-532b56c035ec" (UID: "334942ae-4532-436c-b274-532b56c035ec"). InnerVolumeSpecName "kube-api-access-5xc6h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:27:33.001154 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:33.001092 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/334942ae-4532-436c-b274-532b56c035ec-istio-token" pod "334942ae-4532-436c-b274-532b56c035ec" (UID: "334942ae-4532-436c-b274-532b56c035ec"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:27:33.001227 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:33.001160 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/334942ae-4532-436c-b274-532b56c035ec-istio-podinfo" pod "334942ae-4532-436c-b274-532b56c035ec" (UID: "334942ae-4532-436c-b274-532b56c035ec"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:27:33.100455 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:33.100427 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5xc6h\" (UniqueName: \"kubernetes.io/projected/334942ae-4532-436c-b274-532b56c035ec-kube-api-access-5xc6h\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:27:33.100455 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:33.100453 4927 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/334942ae-4532-436c-b274-532b56c035ec-istio-podinfo\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:27:33.100650 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:33.100465 4927 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/334942ae-4532-436c-b274-532b56c035ec-istio-token\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:27:33.100650 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:33.100475 4927 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/334942ae-4532-436c-b274-532b56c035ec-istio-envoy\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:27:33.321903 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:33.321821 4927 scope.go:122] "RemoveContainer" containerID="f9740f71a450a8d486b1459a6121a8eb91adab0a8dd81fe391c0f9d113468e7b" Oct 31 00:27:33.667995 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:33.667949 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-jpjkl" event={"ID":"334942ae-4532-436c-b274-532b56c035ec","Type":"ContainerDied","Data":"f9740f71a450a8d486b1459a6121a8eb91adab0a8dd81fe391c0f9d113468e7b"} Oct 31 00:27:33.668154 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:33.668006 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-jpjkl" event={"ID":"334942ae-4532-436c-b274-532b56c035ec","Type":"ContainerDied","Data":"4b33d1c39456bfabc4b0c5ed6e18a53ea62419d8ae6a6cb82175a243e46aacac"} Oct 31 00:27:33.668154 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:33.668009 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-jpjkl" Oct 31 00:27:33.781576 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:33.781537 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:27:33.959190 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:33.959100 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-jpjkl"] Oct 31 00:27:33.963028 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:33.962999 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-jpjkl"] Oct 31 00:27:34.005892 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:34.005856 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:27:35.086897 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:35.086860 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="334942ae-4532-436c-b274-532b56c035ec" path="/var/lib/kubelet/pods/334942ae-4532-436c-b274-532b56c035ec/volumes" Oct 31 00:27:38.592468 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:38.592431 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:27:42.616729 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:42.616688 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:27:44.096531 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:44.096493 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:27:44.298004 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:44.297962 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:27:44.454746 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:44.454711 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:27:44.548036 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:44.548005 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1e5c33-z7f7x"] Oct 31 00:27:44.599572 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:44.599540 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1e5c33-z7f7x"] Oct 31 00:27:45.006908 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:45.006867 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:27:45.087442 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:45.087407 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e39eb8b7-aaa4-470e-a63d-2b05c14ba5c7" path="/var/lib/kubelet/pods/e39eb8b7-aaa4-470e-a63d-2b05c14ba5c7/volumes" Oct 31 00:27:48.525131 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:48.525086 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-83ae5d88-mmqdp"] Oct 31 00:27:48.526023 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:48.525998 4927 cpu_manager.go:414] "containerMap: removing container" podUID="360cd796-06da-49c1-9bd4-f2f41f04efeb" containerName="istio-proxy" Oct 31 00:27:48.526071 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:48.526027 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="360cd796-06da-49c1-9bd4-f2f41f04efeb" containerName="istio-proxy" Oct 31 00:27:48.526112 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:48.526085 4927 cpu_manager.go:414] "containerMap: removing container" podUID="e5f5046d-b709-4aed-9cd8-0d480dc2960e" containerName="revoke-keys" Oct 31 00:27:48.526112 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:48.526094 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5f5046d-b709-4aed-9cd8-0d480dc2960e" containerName="revoke-keys" Oct 31 00:27:48.526178 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:48.526114 4927 cpu_manager.go:414] "containerMap: removing container" podUID="334942ae-4532-436c-b274-532b56c035ec" containerName="istio-proxy" Oct 31 00:27:48.526178 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:48.526121 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="334942ae-4532-436c-b274-532b56c035ec" containerName="istio-proxy" Oct 31 00:27:48.526287 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:48.526274 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5f5046d-b709-4aed-9cd8-0d480dc2960e" containerName="revoke-keys" Oct 31 00:27:48.526327 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:48.526291 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="334942ae-4532-436c-b274-532b56c035ec" containerName="istio-proxy" Oct 31 00:27:48.532115 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:48.532096 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83ae5d88-mmqdp" Oct 31 00:27:48.586700 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:48.586669 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-83ae5d88-mmqdp"] Oct 31 00:27:48.632732 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:48.632699 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:27:48.632938 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:48.632714 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4fdzs\"" Oct 31 00:27:48.692414 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:48.692378 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/085c6984-6c5b-4346-a05f-c29c82c4816c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-83ae5d88-mmqdp\" (UID: \"085c6984-6c5b-4346-a05f-c29c82c4816c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83ae5d88-mmqdp" Oct 31 00:27:48.692579 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:48.692507 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/085c6984-6c5b-4346-a05f-c29c82c4816c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-83ae5d88-mmqdp\" (UID: \"085c6984-6c5b-4346-a05f-c29c82c4816c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83ae5d88-mmqdp" Oct 31 00:27:48.692623 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:48.692588 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gtn4w\" (UniqueName: \"kubernetes.io/projected/085c6984-6c5b-4346-a05f-c29c82c4816c-kube-api-access-gtn4w\") pod \"maas-api-revoke-sub-83ae5d88-mmqdp\" (UID: \"085c6984-6c5b-4346-a05f-c29c82c4816c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83ae5d88-mmqdp" Oct 31 00:27:48.794233 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:48.794146 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/085c6984-6c5b-4346-a05f-c29c82c4816c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-83ae5d88-mmqdp\" (UID: \"085c6984-6c5b-4346-a05f-c29c82c4816c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83ae5d88-mmqdp" Oct 31 00:27:48.794454 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:48.794231 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gtn4w\" (UniqueName: \"kubernetes.io/projected/085c6984-6c5b-4346-a05f-c29c82c4816c-kube-api-access-gtn4w\") pod \"maas-api-revoke-sub-83ae5d88-mmqdp\" (UID: \"085c6984-6c5b-4346-a05f-c29c82c4816c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83ae5d88-mmqdp" Oct 31 00:27:48.794454 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:48.794325 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/085c6984-6c5b-4346-a05f-c29c82c4816c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-83ae5d88-mmqdp\" (UID: \"085c6984-6c5b-4346-a05f-c29c82c4816c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83ae5d88-mmqdp" Oct 31 00:27:48.795038 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:48.795009 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/085c6984-6c5b-4346-a05f-c29c82c4816c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-83ae5d88-mmqdp\" (UID: \"085c6984-6c5b-4346-a05f-c29c82c4816c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83ae5d88-mmqdp" Oct 31 00:27:48.796671 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:48.796649 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/085c6984-6c5b-4346-a05f-c29c82c4816c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-83ae5d88-mmqdp\" (UID: \"085c6984-6c5b-4346-a05f-c29c82c4816c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83ae5d88-mmqdp" Oct 31 00:27:48.857142 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:48.857104 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gtn4w\" (UniqueName: \"kubernetes.io/projected/085c6984-6c5b-4346-a05f-c29c82c4816c-kube-api-access-gtn4w\") pod \"maas-api-revoke-sub-83ae5d88-mmqdp\" (UID: \"085c6984-6c5b-4346-a05f-c29c82c4816c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83ae5d88-mmqdp" Oct 31 00:27:49.146514 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:49.146433 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83ae5d88-mmqdp" Oct 31 00:27:49.308249 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:49.308214 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:27:49.529599 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:49.529563 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-83ae5d88-mmqdp"] Oct 31 00:27:49.531615 ip-10-0-141-180 kubenswrapper[4927]: W1031 00:27:49.531587 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod085c6984_6c5b_4346_a05f_c29c82c4816c.slice/crio-1144edd09b3de8880bf233b24103492586249bf54bc4cb4fa2e15e093b4aa5d3 WatchSource:0}: Error finding container 1144edd09b3de8880bf233b24103492586249bf54bc4cb4fa2e15e093b4aa5d3: Status 404 returned error can't find the container with id 1144edd09b3de8880bf233b24103492586249bf54bc4cb4fa2e15e093b4aa5d3 Oct 31 00:27:49.679056 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:27:49.679019 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sqv69" containerName="extractor" Oct 31 00:27:49.805751 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:49.805646 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83ae5d88-mmqdp" event={"ID":"085c6984-6c5b-4346-a05f-c29c82c4816c","Type":"ContainerStarted","Data":"1144edd09b3de8880bf233b24103492586249bf54bc4cb4fa2e15e093b4aa5d3"} Oct 31 00:27:50.814488 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:50.814398 4927 generic.go:356] "Generic (PLEG): container finished" podID="085c6984-6c5b-4346-a05f-c29c82c4816c" containerID="4ea0caf9f5c31d337f75b3402c35dabcc1642bbd258b4674a56fd6417b3fce3a" exitCode=0 Oct 31 00:27:50.814488 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:50.814471 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83ae5d88-mmqdp" event={"ID":"085c6984-6c5b-4346-a05f-c29c82c4816c","Type":"ContainerDied","Data":"4ea0caf9f5c31d337f75b3402c35dabcc1642bbd258b4674a56fd6417b3fce3a"} Oct 31 00:27:51.345260 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:51.345225 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:27:51.598749 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:51.598658 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:27:51.793084 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:51.793045 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:27:52.031395 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:52.031368 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83ae5d88-mmqdp" Oct 31 00:27:52.139173 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:52.139096 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/085c6984-6c5b-4346-a05f-c29c82c4816c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/085c6984-6c5b-4346-a05f-c29c82c4816c-maas-api-xks-ca\") pod \"085c6984-6c5b-4346-a05f-c29c82c4816c\" (UID: \"085c6984-6c5b-4346-a05f-c29c82c4816c\") " Oct 31 00:27:52.139173 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:52.139159 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/085c6984-6c5b-4346-a05f-c29c82c4816c-kube-api-access-gtn4w\" (UniqueName: \"kubernetes.io/projected/085c6984-6c5b-4346-a05f-c29c82c4816c-kube-api-access-gtn4w\") pod \"085c6984-6c5b-4346-a05f-c29c82c4816c\" (UID: \"085c6984-6c5b-4346-a05f-c29c82c4816c\") " Oct 31 00:27:52.139397 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:52.139228 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/085c6984-6c5b-4346-a05f-c29c82c4816c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/085c6984-6c5b-4346-a05f-c29c82c4816c-maas-api-openshift-ca\") pod \"085c6984-6c5b-4346-a05f-c29c82c4816c\" (UID: \"085c6984-6c5b-4346-a05f-c29c82c4816c\") " Oct 31 00:27:52.139567 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:52.139542 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/085c6984-6c5b-4346-a05f-c29c82c4816c-maas-api-openshift-ca" pod "085c6984-6c5b-4346-a05f-c29c82c4816c" (UID: "085c6984-6c5b-4346-a05f-c29c82c4816c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:27:52.139749 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:52.139732 4927 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/085c6984-6c5b-4346-a05f-c29c82c4816c-maas-api-openshift-ca\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:27:52.141245 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:52.141217 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/085c6984-6c5b-4346-a05f-c29c82c4816c-maas-api-xks-ca" pod "085c6984-6c5b-4346-a05f-c29c82c4816c" (UID: "085c6984-6c5b-4346-a05f-c29c82c4816c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:27:52.141370 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:52.141269 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/085c6984-6c5b-4346-a05f-c29c82c4816c-kube-api-access-gtn4w" pod "085c6984-6c5b-4346-a05f-c29c82c4816c" (UID: "085c6984-6c5b-4346-a05f-c29c82c4816c"). InnerVolumeSpecName "kube-api-access-gtn4w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:27:52.240991 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:52.240955 4927 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/085c6984-6c5b-4346-a05f-c29c82c4816c-maas-api-xks-ca\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:27:52.240991 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:52.240987 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gtn4w\" (UniqueName: \"kubernetes.io/projected/085c6984-6c5b-4346-a05f-c29c82c4816c-kube-api-access-gtn4w\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:27:52.264881 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:52.264844 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:27:52.744298 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:52.744257 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-986d37-dhjvh"] Oct 31 00:27:52.744881 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:52.744865 4927 cpu_manager.go:414] "containerMap: removing container" podUID="085c6984-6c5b-4346-a05f-c29c82c4816c" containerName="revoke-keys" Oct 31 00:27:52.744938 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:52.744883 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="085c6984-6c5b-4346-a05f-c29c82c4816c" containerName="revoke-keys" Oct 31 00:27:52.745002 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:52.744993 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="085c6984-6c5b-4346-a05f-c29c82c4816c" containerName="revoke-keys" Oct 31 00:27:52.748885 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:52.748863 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-986d37-dhjvh" Oct 31 00:27:52.831900 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:52.831859 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-986d37-dhjvh"] Oct 31 00:27:52.847496 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:52.847462 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6dc275c7-21fb-4b5c-ba2c-21b88ca78efc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-986d37-dhjvh\" (UID: \"6dc275c7-21fb-4b5c-ba2c-21b88ca78efc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-986d37-dhjvh" Oct 31 00:27:52.847675 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:52.847604 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6dc275c7-21fb-4b5c-ba2c-21b88ca78efc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-986d37-dhjvh\" (UID: \"6dc275c7-21fb-4b5c-ba2c-21b88ca78efc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-986d37-dhjvh" Oct 31 00:27:52.847675 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:52.847646 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgm4q\" (UniqueName: \"kubernetes.io/projected/6dc275c7-21fb-4b5c-ba2c-21b88ca78efc-kube-api-access-xgm4q\") pod \"maas-api-revoke-keys-e2e-worker-w3-986d37-dhjvh\" (UID: \"6dc275c7-21fb-4b5c-ba2c-21b88ca78efc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-986d37-dhjvh" Oct 31 00:27:52.886812 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:52.886783 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83ae5d88-mmqdp" Oct 31 00:27:52.886973 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:52.886819 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83ae5d88-mmqdp" event={"ID":"085c6984-6c5b-4346-a05f-c29c82c4816c","Type":"ContainerDied","Data":"1144edd09b3de8880bf233b24103492586249bf54bc4cb4fa2e15e093b4aa5d3"} Oct 31 00:27:52.886973 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:52.886867 4927 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1144edd09b3de8880bf233b24103492586249bf54bc4cb4fa2e15e093b4aa5d3" Oct 31 00:27:52.948287 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:52.948255 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6dc275c7-21fb-4b5c-ba2c-21b88ca78efc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-986d37-dhjvh\" (UID: \"6dc275c7-21fb-4b5c-ba2c-21b88ca78efc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-986d37-dhjvh" Oct 31 00:27:52.948465 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:52.948320 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgm4q\" (UniqueName: \"kubernetes.io/projected/6dc275c7-21fb-4b5c-ba2c-21b88ca78efc-kube-api-access-xgm4q\") pod \"maas-api-revoke-keys-e2e-worker-w3-986d37-dhjvh\" (UID: \"6dc275c7-21fb-4b5c-ba2c-21b88ca78efc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-986d37-dhjvh" Oct 31 00:27:52.948465 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:52.948406 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6dc275c7-21fb-4b5c-ba2c-21b88ca78efc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-986d37-dhjvh\" (UID: \"6dc275c7-21fb-4b5c-ba2c-21b88ca78efc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-986d37-dhjvh" Oct 31 00:27:52.949134 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:52.949102 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6dc275c7-21fb-4b5c-ba2c-21b88ca78efc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-986d37-dhjvh\" (UID: \"6dc275c7-21fb-4b5c-ba2c-21b88ca78efc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-986d37-dhjvh" Oct 31 00:27:52.950829 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:52.950807 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6dc275c7-21fb-4b5c-ba2c-21b88ca78efc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-986d37-dhjvh\" (UID: \"6dc275c7-21fb-4b5c-ba2c-21b88ca78efc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-986d37-dhjvh" Oct 31 00:27:53.011621 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:53.011554 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgm4q\" (UniqueName: \"kubernetes.io/projected/6dc275c7-21fb-4b5c-ba2c-21b88ca78efc-kube-api-access-xgm4q\") pod \"maas-api-revoke-keys-e2e-worker-w3-986d37-dhjvh\" (UID: \"6dc275c7-21fb-4b5c-ba2c-21b88ca78efc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-986d37-dhjvh" Oct 31 00:27:53.062882 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:53.062853 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-986d37-dhjvh" Oct 31 00:27:53.449025 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:53.448992 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-986d37-dhjvh"] Oct 31 00:27:53.450062 ip-10-0-141-180 kubenswrapper[4927]: W1031 00:27:53.450036 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6dc275c7_21fb_4b5c_ba2c_21b88ca78efc.slice/crio-e334c3c985aea395a496645baf59e8324875f5d25e7039e08940b89c32971688 WatchSource:0}: Error finding container e334c3c985aea395a496645baf59e8324875f5d25e7039e08940b89c32971688: Status 404 returned error can't find the container with id e334c3c985aea395a496645baf59e8324875f5d25e7039e08940b89c32971688 Oct 31 00:27:53.895103 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:53.895066 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-986d37-dhjvh" event={"ID":"6dc275c7-21fb-4b5c-ba2c-21b88ca78efc","Type":"ContainerStarted","Data":"972ad31fdb50e0c19c46f8b0620a553cd610bb681037e0ae9322cb27883e7793"} Oct 31 00:27:53.895326 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:53.895123 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-986d37-dhjvh" event={"ID":"6dc275c7-21fb-4b5c-ba2c-21b88ca78efc","Type":"ContainerStarted","Data":"e334c3c985aea395a496645baf59e8324875f5d25e7039e08940b89c32971688"} Oct 31 00:27:54.029151 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:54.029043 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-986d37-dhjvh" podStartSLOduration=2.029029442 podStartE2EDuration="2.029029442s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:27:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:27:54.026832151 +0000 UTC m=+2787.063162993" watchObservedRunningTime="2026-10-31 00:27:54.029029442 +0000 UTC m=+2787.065360264" Oct 31 00:27:54.313265 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:54.313231 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:27:54.903660 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:54.903627 4927 generic.go:356] "Generic (PLEG): container finished" podID="6dc275c7-21fb-4b5c-ba2c-21b88ca78efc" containerID="972ad31fdb50e0c19c46f8b0620a553cd610bb681037e0ae9322cb27883e7793" exitCode=0 Oct 31 00:27:54.903847 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:54.903718 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-986d37-dhjvh" event={"ID":"6dc275c7-21fb-4b5c-ba2c-21b88ca78efc","Type":"ContainerDied","Data":"972ad31fdb50e0c19c46f8b0620a553cd610bb681037e0ae9322cb27883e7793"} Oct 31 00:27:56.100814 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:56.100763 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:27:56.101594 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:56.101566 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-986d37-dhjvh" Oct 31 00:27:56.191405 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:56.190709 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6dc275c7-21fb-4b5c-ba2c-21b88ca78efc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6dc275c7-21fb-4b5c-ba2c-21b88ca78efc-maas-api-openshift-ca\") pod \"6dc275c7-21fb-4b5c-ba2c-21b88ca78efc\" (UID: \"6dc275c7-21fb-4b5c-ba2c-21b88ca78efc\") " Oct 31 00:27:56.191405 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:56.190824 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6dc275c7-21fb-4b5c-ba2c-21b88ca78efc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6dc275c7-21fb-4b5c-ba2c-21b88ca78efc-maas-api-xks-ca\") pod \"6dc275c7-21fb-4b5c-ba2c-21b88ca78efc\" (UID: \"6dc275c7-21fb-4b5c-ba2c-21b88ca78efc\") " Oct 31 00:27:56.191405 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:56.191000 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6dc275c7-21fb-4b5c-ba2c-21b88ca78efc-kube-api-access-xgm4q\" (UniqueName: \"kubernetes.io/projected/6dc275c7-21fb-4b5c-ba2c-21b88ca78efc-kube-api-access-xgm4q\") pod \"6dc275c7-21fb-4b5c-ba2c-21b88ca78efc\" (UID: \"6dc275c7-21fb-4b5c-ba2c-21b88ca78efc\") " Oct 31 00:27:56.196845 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:56.196518 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6dc275c7-21fb-4b5c-ba2c-21b88ca78efc-maas-api-openshift-ca" pod "6dc275c7-21fb-4b5c-ba2c-21b88ca78efc" (UID: "6dc275c7-21fb-4b5c-ba2c-21b88ca78efc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:27:56.204810 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:56.204765 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6dc275c7-21fb-4b5c-ba2c-21b88ca78efc-kube-api-access-xgm4q" pod "6dc275c7-21fb-4b5c-ba2c-21b88ca78efc" (UID: "6dc275c7-21fb-4b5c-ba2c-21b88ca78efc"). InnerVolumeSpecName "kube-api-access-xgm4q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:27:56.214014 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:56.213973 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6dc275c7-21fb-4b5c-ba2c-21b88ca78efc-maas-api-xks-ca" pod "6dc275c7-21fb-4b5c-ba2c-21b88ca78efc" (UID: "6dc275c7-21fb-4b5c-ba2c-21b88ca78efc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:27:56.292795 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:56.292756 4927 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6dc275c7-21fb-4b5c-ba2c-21b88ca78efc-maas-api-openshift-ca\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:27:56.292795 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:56.292792 4927 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6dc275c7-21fb-4b5c-ba2c-21b88ca78efc-maas-api-xks-ca\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:27:56.292985 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:56.292802 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xgm4q\" (UniqueName: \"kubernetes.io/projected/6dc275c7-21fb-4b5c-ba2c-21b88ca78efc-kube-api-access-xgm4q\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:27:56.304867 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:56.304836 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:27:56.917803 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:56.917773 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-986d37-dhjvh" Oct 31 00:27:56.917981 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:56.917779 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-986d37-dhjvh" event={"ID":"6dc275c7-21fb-4b5c-ba2c-21b88ca78efc","Type":"ContainerDied","Data":"e334c3c985aea395a496645baf59e8324875f5d25e7039e08940b89c32971688"} Oct 31 00:27:56.917981 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:56.917928 4927 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e334c3c985aea395a496645baf59e8324875f5d25e7039e08940b89c32971688" Oct 31 00:27:57.220043 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:57.219961 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-986d37/premium-w3-986d37-kserve-5c7c7757fb-r99j7"] Oct 31 00:27:57.318650 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:57.318616 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:27:57.980553 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:57.980507 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:27:58.442129 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:58.442095 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-76b073-7zs2f"] Oct 31 00:27:58.442847 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:58.442828 4927 cpu_manager.go:414] "containerMap: removing container" podUID="6dc275c7-21fb-4b5c-ba2c-21b88ca78efc" containerName="revoke-keys" Oct 31 00:27:58.442911 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:58.442849 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6dc275c7-21fb-4b5c-ba2c-21b88ca78efc" containerName="revoke-keys" Oct 31 00:27:58.442954 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:58.442942 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="6dc275c7-21fb-4b5c-ba2c-21b88ca78efc" containerName="revoke-keys" Oct 31 00:27:58.446597 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:58.446579 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-76b073-7zs2f" Oct 31 00:27:58.502972 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:58.502927 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:27:58.506382 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:58.506319 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-76b073-7zs2f"] Oct 31 00:27:58.519044 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:58.519012 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9cbgx\" (UniqueName: \"kubernetes.io/projected/b22587ca-cec2-41f1-8f67-cfac35cfbcdd-kube-api-access-9cbgx\") pod \"maas-api-revoke-keys-e2e-inf-a-76b073-7zs2f\" (UID: \"b22587ca-cec2-41f1-8f67-cfac35cfbcdd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-76b073-7zs2f" Oct 31 00:27:58.519199 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:58.519057 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b22587ca-cec2-41f1-8f67-cfac35cfbcdd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-76b073-7zs2f\" (UID: \"b22587ca-cec2-41f1-8f67-cfac35cfbcdd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-76b073-7zs2f" Oct 31 00:27:58.519272 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:58.519219 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b22587ca-cec2-41f1-8f67-cfac35cfbcdd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-76b073-7zs2f\" (UID: \"b22587ca-cec2-41f1-8f67-cfac35cfbcdd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-76b073-7zs2f" Oct 31 00:27:58.551291 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:58.551252 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4fdzs\"" Oct 31 00:27:58.620026 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:58.619996 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b22587ca-cec2-41f1-8f67-cfac35cfbcdd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-76b073-7zs2f\" (UID: \"b22587ca-cec2-41f1-8f67-cfac35cfbcdd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-76b073-7zs2f" Oct 31 00:27:58.620187 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:58.620129 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9cbgx\" (UniqueName: \"kubernetes.io/projected/b22587ca-cec2-41f1-8f67-cfac35cfbcdd-kube-api-access-9cbgx\") pod \"maas-api-revoke-keys-e2e-inf-a-76b073-7zs2f\" (UID: \"b22587ca-cec2-41f1-8f67-cfac35cfbcdd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-76b073-7zs2f" Oct 31 00:27:58.620187 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:58.620165 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b22587ca-cec2-41f1-8f67-cfac35cfbcdd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-76b073-7zs2f\" (UID: \"b22587ca-cec2-41f1-8f67-cfac35cfbcdd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-76b073-7zs2f" Oct 31 00:27:58.620818 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:58.620796 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b22587ca-cec2-41f1-8f67-cfac35cfbcdd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-76b073-7zs2f\" (UID: \"b22587ca-cec2-41f1-8f67-cfac35cfbcdd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-76b073-7zs2f" Oct 31 00:27:58.622652 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:58.622630 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b22587ca-cec2-41f1-8f67-cfac35cfbcdd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-76b073-7zs2f\" (UID: \"b22587ca-cec2-41f1-8f67-cfac35cfbcdd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-76b073-7zs2f" Oct 31 00:27:58.682122 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:58.682096 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9cbgx\" (UniqueName: \"kubernetes.io/projected/b22587ca-cec2-41f1-8f67-cfac35cfbcdd-kube-api-access-9cbgx\") pod \"maas-api-revoke-keys-e2e-inf-a-76b073-7zs2f\" (UID: \"b22587ca-cec2-41f1-8f67-cfac35cfbcdd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-76b073-7zs2f" Oct 31 00:27:58.763652 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:58.763578 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-76b073-7zs2f" Oct 31 00:27:59.151974 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:59.151942 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-76b073-7zs2f"] Oct 31 00:27:59.152712 ip-10-0-141-180 kubenswrapper[4927]: W1031 00:27:59.152684 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb22587ca_cec2_41f1_8f67_cfac35cfbcdd.slice/crio-30abe322d6ca41c2286c080e64bb47a980b9ee364117ddd0f5b5066d8b668711 WatchSource:0}: Error finding container 30abe322d6ca41c2286c080e64bb47a980b9ee364117ddd0f5b5066d8b668711: Status 404 returned error can't find the container with id 30abe322d6ca41c2286c080e64bb47a980b9ee364117ddd0f5b5066d8b668711 Oct 31 00:27:59.235224 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:59.235194 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:27:59.944358 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:59.943348 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-76b073-7zs2f" event={"ID":"b22587ca-cec2-41f1-8f67-cfac35cfbcdd","Type":"ContainerStarted","Data":"5ff3010631c35b1b8eeb97fa47d4a597d3c9abf24e922a7d18122ed165c36f1b"} Oct 31 00:27:59.944358 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:27:59.943413 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-76b073-7zs2f" event={"ID":"b22587ca-cec2-41f1-8f67-cfac35cfbcdd","Type":"ContainerStarted","Data":"30abe322d6ca41c2286c080e64bb47a980b9ee364117ddd0f5b5066d8b668711"} Oct 31 00:28:00.063490 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:00.063437 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-76b073-7zs2f" podStartSLOduration=2.063421622 podStartE2EDuration="2.063421622s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:27:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:28:00.061622182 +0000 UTC m=+2793.097953004" watchObservedRunningTime="2026-10-31 00:28:00.063421622 +0000 UTC m=+2793.099752444" Oct 31 00:28:00.951175 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:00.951138 4927 generic.go:356] "Generic (PLEG): container finished" podID="b22587ca-cec2-41f1-8f67-cfac35cfbcdd" containerID="5ff3010631c35b1b8eeb97fa47d4a597d3c9abf24e922a7d18122ed165c36f1b" exitCode=0 Oct 31 00:28:00.951618 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:00.951227 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-76b073-7zs2f" event={"ID":"b22587ca-cec2-41f1-8f67-cfac35cfbcdd","Type":"ContainerDied","Data":"5ff3010631c35b1b8eeb97fa47d4a597d3c9abf24e922a7d18122ed165c36f1b"} Oct 31 00:28:02.119396 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:02.119370 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-76b073-7zs2f" Oct 31 00:28:02.160582 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:02.160523 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b22587ca-cec2-41f1-8f67-cfac35cfbcdd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b22587ca-cec2-41f1-8f67-cfac35cfbcdd-maas-api-xks-ca\") pod \"b22587ca-cec2-41f1-8f67-cfac35cfbcdd\" (UID: \"b22587ca-cec2-41f1-8f67-cfac35cfbcdd\") " Oct 31 00:28:02.160788 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:02.160687 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b22587ca-cec2-41f1-8f67-cfac35cfbcdd-kube-api-access-9cbgx\" (UniqueName: \"kubernetes.io/projected/b22587ca-cec2-41f1-8f67-cfac35cfbcdd-kube-api-access-9cbgx\") pod \"b22587ca-cec2-41f1-8f67-cfac35cfbcdd\" (UID: \"b22587ca-cec2-41f1-8f67-cfac35cfbcdd\") " Oct 31 00:28:02.160860 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:02.160793 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b22587ca-cec2-41f1-8f67-cfac35cfbcdd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b22587ca-cec2-41f1-8f67-cfac35cfbcdd-maas-api-openshift-ca\") pod \"b22587ca-cec2-41f1-8f67-cfac35cfbcdd\" (UID: \"b22587ca-cec2-41f1-8f67-cfac35cfbcdd\") " Oct 31 00:28:02.161140 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:02.161110 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b22587ca-cec2-41f1-8f67-cfac35cfbcdd-maas-api-openshift-ca" pod "b22587ca-cec2-41f1-8f67-cfac35cfbcdd" (UID: "b22587ca-cec2-41f1-8f67-cfac35cfbcdd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:28:02.162712 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:02.162692 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b22587ca-cec2-41f1-8f67-cfac35cfbcdd-kube-api-access-9cbgx" pod "b22587ca-cec2-41f1-8f67-cfac35cfbcdd" (UID: "b22587ca-cec2-41f1-8f67-cfac35cfbcdd"). InnerVolumeSpecName "kube-api-access-9cbgx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:28:02.162896 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:02.162866 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b22587ca-cec2-41f1-8f67-cfac35cfbcdd-maas-api-xks-ca" pod "b22587ca-cec2-41f1-8f67-cfac35cfbcdd" (UID: "b22587ca-cec2-41f1-8f67-cfac35cfbcdd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:28:02.262675 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:02.262600 4927 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b22587ca-cec2-41f1-8f67-cfac35cfbcdd-maas-api-xks-ca\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:02.262675 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:02.262629 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9cbgx\" (UniqueName: \"kubernetes.io/projected/b22587ca-cec2-41f1-8f67-cfac35cfbcdd-kube-api-access-9cbgx\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:02.262675 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:02.262640 4927 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b22587ca-cec2-41f1-8f67-cfac35cfbcdd-maas-api-openshift-ca\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:02.376419 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:02.376372 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:28:02.856735 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:02.856694 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:28:02.967564 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:02.967532 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-76b073-7zs2f" Oct 31 00:28:02.967741 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:02.967558 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-76b073-7zs2f" event={"ID":"b22587ca-cec2-41f1-8f67-cfac35cfbcdd","Type":"ContainerDied","Data":"30abe322d6ca41c2286c080e64bb47a980b9ee364117ddd0f5b5066d8b668711"} Oct 31 00:28:02.967741 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:02.967631 4927 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="30abe322d6ca41c2286c080e64bb47a980b9ee364117ddd0f5b5066d8b668711" Oct 31 00:28:03.020903 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:03.020867 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:28:03.029005 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:28:03.028970 4927 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb22587ca_cec2_41f1_8f67_cfac35cfbcdd.slice\": RecentStats: unable to find data in memory cache]" Oct 31 00:28:04.102404 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:04.102368 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:28:08.470184 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:08.470148 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-986d37-695476d5cd-99x6j"] Oct 31 00:28:08.581440 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:08.581402 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-986d37-5d7d64658c-h2tgp"] Oct 31 00:28:08.685162 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:08.685118 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:28:08.839543 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:08.839447 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:28:09.023576 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:09.023531 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:28:10.791485 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:10.791452 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-76b073-78787bc78-d6l68"] Oct 31 00:28:10.905118 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:10.905082 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-76b073-68467cb4b7-pf7qq"] Oct 31 00:28:11.147503 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:11.147408 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:28:11.300260 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:11.300218 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:28:12.241290 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:12.241249 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-986d37/premium-w3-986d37-kserve-5c7c7757fb-r99j7" podUID="f4e38df7-355f-4b47-a0b5-7e0b907f3326" containerName="main" containerID="cri-o://46b55086ef6e0ea80c41e66e0fe1951219feee8d5b2011565d514504d8de75e2" gracePeriod=45 Oct 31 00:28:12.575820 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:12.575790 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-986d37/premium-w3-986d37-kserve-5c7c7757fb-r99j7" Oct 31 00:28:12.592002 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:12.591973 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4e38df7-355f-4b47-a0b5-7e0b907f3326-kube-api-access-nfpgq\" (UniqueName: \"kubernetes.io/projected/f4e38df7-355f-4b47-a0b5-7e0b907f3326-kube-api-access-nfpgq\") pod \"f4e38df7-355f-4b47-a0b5-7e0b907f3326\" (UID: \"f4e38df7-355f-4b47-a0b5-7e0b907f3326\") " Oct 31 00:28:12.592133 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:12.592037 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f4e38df7-355f-4b47-a0b5-7e0b907f3326-tls-certs\" (UniqueName: \"kubernetes.io/secret/f4e38df7-355f-4b47-a0b5-7e0b907f3326-tls-certs\") pod \"f4e38df7-355f-4b47-a0b5-7e0b907f3326\" (UID: \"f4e38df7-355f-4b47-a0b5-7e0b907f3326\") " Oct 31 00:28:12.592177 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:12.592138 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4e38df7-355f-4b47-a0b5-7e0b907f3326-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f4e38df7-355f-4b47-a0b5-7e0b907f3326-dshm\") pod \"f4e38df7-355f-4b47-a0b5-7e0b907f3326\" (UID: \"f4e38df7-355f-4b47-a0b5-7e0b907f3326\") " Oct 31 00:28:12.592223 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:12.592185 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4e38df7-355f-4b47-a0b5-7e0b907f3326-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f4e38df7-355f-4b47-a0b5-7e0b907f3326-model-cache\") pod \"f4e38df7-355f-4b47-a0b5-7e0b907f3326\" (UID: \"f4e38df7-355f-4b47-a0b5-7e0b907f3326\") " Oct 31 00:28:12.592264 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:12.592253 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4e38df7-355f-4b47-a0b5-7e0b907f3326-home\" (UniqueName: \"kubernetes.io/empty-dir/f4e38df7-355f-4b47-a0b5-7e0b907f3326-home\") pod \"f4e38df7-355f-4b47-a0b5-7e0b907f3326\" (UID: \"f4e38df7-355f-4b47-a0b5-7e0b907f3326\") " Oct 31 00:28:12.592321 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:12.592292 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4e38df7-355f-4b47-a0b5-7e0b907f3326-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4e38df7-355f-4b47-a0b5-7e0b907f3326-tmp-dir\") pod \"f4e38df7-355f-4b47-a0b5-7e0b907f3326\" (UID: \"f4e38df7-355f-4b47-a0b5-7e0b907f3326\") " Oct 31 00:28:12.592549 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:12.592522 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4e38df7-355f-4b47-a0b5-7e0b907f3326-model-cache" pod "f4e38df7-355f-4b47-a0b5-7e0b907f3326" (UID: "f4e38df7-355f-4b47-a0b5-7e0b907f3326"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:28:12.592643 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:12.592607 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4e38df7-355f-4b47-a0b5-7e0b907f3326-home" pod "f4e38df7-355f-4b47-a0b5-7e0b907f3326" (UID: "f4e38df7-355f-4b47-a0b5-7e0b907f3326"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:28:12.592719 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:12.592637 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4e38df7-355f-4b47-a0b5-7e0b907f3326-tmp-dir" pod "f4e38df7-355f-4b47-a0b5-7e0b907f3326" (UID: "f4e38df7-355f-4b47-a0b5-7e0b907f3326"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:28:12.592903 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:12.592889 4927 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f4e38df7-355f-4b47-a0b5-7e0b907f3326-model-cache\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:12.592946 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:12.592906 4927 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f4e38df7-355f-4b47-a0b5-7e0b907f3326-home\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:12.592946 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:12.592915 4927 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4e38df7-355f-4b47-a0b5-7e0b907f3326-tmp-dir\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:12.594404 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:12.594383 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4e38df7-355f-4b47-a0b5-7e0b907f3326-dshm" pod "f4e38df7-355f-4b47-a0b5-7e0b907f3326" (UID: "f4e38df7-355f-4b47-a0b5-7e0b907f3326"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:28:12.594701 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:12.594681 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f4e38df7-355f-4b47-a0b5-7e0b907f3326-tls-certs" pod "f4e38df7-355f-4b47-a0b5-7e0b907f3326" (UID: "f4e38df7-355f-4b47-a0b5-7e0b907f3326"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:28:12.594770 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:12.594707 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4e38df7-355f-4b47-a0b5-7e0b907f3326-kube-api-access-nfpgq" pod "f4e38df7-355f-4b47-a0b5-7e0b907f3326" (UID: "f4e38df7-355f-4b47-a0b5-7e0b907f3326"). InnerVolumeSpecName "kube-api-access-nfpgq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:28:12.694053 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:12.694017 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nfpgq\" (UniqueName: \"kubernetes.io/projected/f4e38df7-355f-4b47-a0b5-7e0b907f3326-kube-api-access-nfpgq\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:12.694053 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:12.694047 4927 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f4e38df7-355f-4b47-a0b5-7e0b907f3326-tls-certs\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:12.694053 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:12.694060 4927 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f4e38df7-355f-4b47-a0b5-7e0b907f3326-dshm\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:13.069319 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:13.069285 4927 generic.go:356] "Generic (PLEG): container finished" podID="f4e38df7-355f-4b47-a0b5-7e0b907f3326" containerID="46b55086ef6e0ea80c41e66e0fe1951219feee8d5b2011565d514504d8de75e2" exitCode=0 Oct 31 00:28:13.069319 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:13.069345 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-986d37/premium-w3-986d37-kserve-5c7c7757fb-r99j7" Oct 31 00:28:13.069567 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:13.069378 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-986d37/premium-w3-986d37-kserve-5c7c7757fb-r99j7" event={"ID":"f4e38df7-355f-4b47-a0b5-7e0b907f3326","Type":"ContainerDied","Data":"46b55086ef6e0ea80c41e66e0fe1951219feee8d5b2011565d514504d8de75e2"} Oct 31 00:28:13.069567 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:13.069422 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-986d37/premium-w3-986d37-kserve-5c7c7757fb-r99j7" event={"ID":"f4e38df7-355f-4b47-a0b5-7e0b907f3326","Type":"ContainerDied","Data":"9d3db928da248abff45ca14834a2c8a6ac095af7911b08dc38e46163fac18215"} Oct 31 00:28:13.069567 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:13.069445 4927 scope.go:122] "RemoveContainer" containerID="46b55086ef6e0ea80c41e66e0fe1951219feee8d5b2011565d514504d8de75e2" Oct 31 00:28:13.086446 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:13.086425 4927 scope.go:122] "RemoveContainer" containerID="46b55086ef6e0ea80c41e66e0fe1951219feee8d5b2011565d514504d8de75e2" Oct 31 00:28:13.086955 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:28:13.086933 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"46b55086ef6e0ea80c41e66e0fe1951219feee8d5b2011565d514504d8de75e2\": container with ID starting with 46b55086ef6e0ea80c41e66e0fe1951219feee8d5b2011565d514504d8de75e2 not found: ID does not exist" containerID="46b55086ef6e0ea80c41e66e0fe1951219feee8d5b2011565d514504d8de75e2" Oct 31 00:28:13.087041 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:13.086962 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46b55086ef6e0ea80c41e66e0fe1951219feee8d5b2011565d514504d8de75e2"} err="failed to get container status \"46b55086ef6e0ea80c41e66e0fe1951219feee8d5b2011565d514504d8de75e2\": rpc error: code = NotFound desc = could not find container \"46b55086ef6e0ea80c41e66e0fe1951219feee8d5b2011565d514504d8de75e2\": container with ID starting with 46b55086ef6e0ea80c41e66e0fe1951219feee8d5b2011565d514504d8de75e2 not found: ID does not exist" Oct 31 00:28:13.240426 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:13.240394 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-986d37/premium-w3-986d37-kserve-5c7c7757fb-r99j7"] Oct 31 00:28:13.252474 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:13.245690 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-986d37/premium-w3-986d37-kserve-5c7c7757fb-r99j7"] Oct 31 00:28:13.376845 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:13.376753 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:28:13.470504 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:13.470451 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w3-986d37-695476d5cd-99x6j" podUID="915e2f48-c25a-4bcf-a1d1-edcc5245dfb8" containerName="payload-processing" containerID="cri-o://24f3926da24a3a209c72cb2885d1b74325abd3425173fa87e16cb25ee7a39d26" gracePeriod=25 Oct 31 00:28:13.582422 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:13.582369 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-986d37-5d7d64658c-h2tgp" podUID="fc5ca633-21a6-4853-88d1-cccb1e17ad4e" containerName="payload-pre-processing" containerID="cri-o://d17f0fe66e125dca99815f10b9d62d3654f065875af1bb42f2d57598170926cd" gracePeriod=25 Oct 31 00:28:13.760254 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:13.760223 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-986d37-695476d5cd-99x6j" Oct 31 00:28:13.808264 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:13.808223 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/915e2f48-c25a-4bcf-a1d1-edcc5245dfb8-kube-api-access-wwn86\" (UniqueName: \"kubernetes.io/projected/915e2f48-c25a-4bcf-a1d1-edcc5245dfb8-kube-api-access-wwn86\") pod \"915e2f48-c25a-4bcf-a1d1-edcc5245dfb8\" (UID: \"915e2f48-c25a-4bcf-a1d1-edcc5245dfb8\") " Oct 31 00:28:13.808457 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:13.808315 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/915e2f48-c25a-4bcf-a1d1-edcc5245dfb8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/915e2f48-c25a-4bcf-a1d1-edcc5245dfb8-plugins-config-volume\") pod \"915e2f48-c25a-4bcf-a1d1-edcc5245dfb8\" (UID: \"915e2f48-c25a-4bcf-a1d1-edcc5245dfb8\") " Oct 31 00:28:13.809635 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:13.809599 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/915e2f48-c25a-4bcf-a1d1-edcc5245dfb8-plugins-config-volume" pod "915e2f48-c25a-4bcf-a1d1-edcc5245dfb8" (UID: "915e2f48-c25a-4bcf-a1d1-edcc5245dfb8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:28:13.812286 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:13.812258 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/915e2f48-c25a-4bcf-a1d1-edcc5245dfb8-kube-api-access-wwn86" pod "915e2f48-c25a-4bcf-a1d1-edcc5245dfb8" (UID: "915e2f48-c25a-4bcf-a1d1-edcc5245dfb8"). InnerVolumeSpecName "kube-api-access-wwn86". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:28:13.868600 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:13.868574 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-986d37-5d7d64658c-h2tgp" Oct 31 00:28:13.909712 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:13.909686 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc5ca633-21a6-4853-88d1-cccb1e17ad4e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc5ca633-21a6-4853-88d1-cccb1e17ad4e-plugins-config-volume\") pod \"fc5ca633-21a6-4853-88d1-cccb1e17ad4e\" (UID: \"fc5ca633-21a6-4853-88d1-cccb1e17ad4e\") " Oct 31 00:28:13.909828 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:13.909806 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc5ca633-21a6-4853-88d1-cccb1e17ad4e-kube-api-access-25hhv\" (UniqueName: \"kubernetes.io/projected/fc5ca633-21a6-4853-88d1-cccb1e17ad4e-kube-api-access-25hhv\") pod \"fc5ca633-21a6-4853-88d1-cccb1e17ad4e\" (UID: \"fc5ca633-21a6-4853-88d1-cccb1e17ad4e\") " Oct 31 00:28:13.910044 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:13.910022 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc5ca633-21a6-4853-88d1-cccb1e17ad4e-plugins-config-volume" pod "fc5ca633-21a6-4853-88d1-cccb1e17ad4e" (UID: "fc5ca633-21a6-4853-88d1-cccb1e17ad4e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:28:13.910240 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:13.910223 4927 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc5ca633-21a6-4853-88d1-cccb1e17ad4e-plugins-config-volume\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:13.910279 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:13.910243 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wwn86\" (UniqueName: \"kubernetes.io/projected/915e2f48-c25a-4bcf-a1d1-edcc5245dfb8-kube-api-access-wwn86\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:13.910279 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:13.910253 4927 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/915e2f48-c25a-4bcf-a1d1-edcc5245dfb8-plugins-config-volume\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:13.911884 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:13.911863 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc5ca633-21a6-4853-88d1-cccb1e17ad4e-kube-api-access-25hhv" pod "fc5ca633-21a6-4853-88d1-cccb1e17ad4e" (UID: "fc5ca633-21a6-4853-88d1-cccb1e17ad4e"). InnerVolumeSpecName "kube-api-access-25hhv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:28:14.011739 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:14.011706 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-25hhv\" (UniqueName: \"kubernetes.io/projected/fc5ca633-21a6-4853-88d1-cccb1e17ad4e-kube-api-access-25hhv\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:14.083573 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:14.083533 4927 generic.go:356] "Generic (PLEG): container finished" podID="915e2f48-c25a-4bcf-a1d1-edcc5245dfb8" containerID="24f3926da24a3a209c72cb2885d1b74325abd3425173fa87e16cb25ee7a39d26" exitCode=0 Oct 31 00:28:14.083788 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:14.083590 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-986d37-695476d5cd-99x6j" Oct 31 00:28:14.083788 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:14.083630 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-986d37-695476d5cd-99x6j" event={"ID":"915e2f48-c25a-4bcf-a1d1-edcc5245dfb8","Type":"ContainerDied","Data":"24f3926da24a3a209c72cb2885d1b74325abd3425173fa87e16cb25ee7a39d26"} Oct 31 00:28:14.083788 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:14.083695 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-986d37-695476d5cd-99x6j" event={"ID":"915e2f48-c25a-4bcf-a1d1-edcc5245dfb8","Type":"ContainerDied","Data":"a7b8e2039573501c1b63f2e913796ce22bf4f71c26569b64363ad3735577fb3d"} Oct 31 00:28:14.083788 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:14.083726 4927 scope.go:122] "RemoveContainer" containerID="24f3926da24a3a209c72cb2885d1b74325abd3425173fa87e16cb25ee7a39d26" Oct 31 00:28:14.085883 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:14.085844 4927 generic.go:356] "Generic (PLEG): container finished" podID="fc5ca633-21a6-4853-88d1-cccb1e17ad4e" containerID="d17f0fe66e125dca99815f10b9d62d3654f065875af1bb42f2d57598170926cd" exitCode=0 Oct 31 00:28:14.085984 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:14.085958 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-986d37-5d7d64658c-h2tgp" event={"ID":"fc5ca633-21a6-4853-88d1-cccb1e17ad4e","Type":"ContainerDied","Data":"d17f0fe66e125dca99815f10b9d62d3654f065875af1bb42f2d57598170926cd"} Oct 31 00:28:14.086055 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:14.086001 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-986d37-5d7d64658c-h2tgp" event={"ID":"fc5ca633-21a6-4853-88d1-cccb1e17ad4e","Type":"ContainerDied","Data":"7f5c2ed32a1b765c09ac6a4025bc7e11dd96567981981f062e77e08ea9abc50f"} Oct 31 00:28:14.086122 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:14.086066 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-986d37-5d7d64658c-h2tgp" Oct 31 00:28:14.099019 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:14.099000 4927 scope.go:122] "RemoveContainer" containerID="24f3926da24a3a209c72cb2885d1b74325abd3425173fa87e16cb25ee7a39d26" Oct 31 00:28:14.099301 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:28:14.099278 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"24f3926da24a3a209c72cb2885d1b74325abd3425173fa87e16cb25ee7a39d26\": container with ID starting with 24f3926da24a3a209c72cb2885d1b74325abd3425173fa87e16cb25ee7a39d26 not found: ID does not exist" containerID="24f3926da24a3a209c72cb2885d1b74325abd3425173fa87e16cb25ee7a39d26" Oct 31 00:28:14.099464 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:14.099305 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24f3926da24a3a209c72cb2885d1b74325abd3425173fa87e16cb25ee7a39d26"} err="failed to get container status \"24f3926da24a3a209c72cb2885d1b74325abd3425173fa87e16cb25ee7a39d26\": rpc error: code = NotFound desc = could not find container \"24f3926da24a3a209c72cb2885d1b74325abd3425173fa87e16cb25ee7a39d26\": container with ID starting with 24f3926da24a3a209c72cb2885d1b74325abd3425173fa87e16cb25ee7a39d26 not found: ID does not exist" Oct 31 00:28:14.099464 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:14.099324 4927 scope.go:122] "RemoveContainer" containerID="d17f0fe66e125dca99815f10b9d62d3654f065875af1bb42f2d57598170926cd" Oct 31 00:28:14.110799 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:14.110777 4927 scope.go:122] "RemoveContainer" containerID="d17f0fe66e125dca99815f10b9d62d3654f065875af1bb42f2d57598170926cd" Oct 31 00:28:14.111102 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:28:14.111073 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d17f0fe66e125dca99815f10b9d62d3654f065875af1bb42f2d57598170926cd\": container with ID starting with d17f0fe66e125dca99815f10b9d62d3654f065875af1bb42f2d57598170926cd not found: ID does not exist" containerID="d17f0fe66e125dca99815f10b9d62d3654f065875af1bb42f2d57598170926cd" Oct 31 00:28:14.111231 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:14.111110 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d17f0fe66e125dca99815f10b9d62d3654f065875af1bb42f2d57598170926cd"} err="failed to get container status \"d17f0fe66e125dca99815f10b9d62d3654f065875af1bb42f2d57598170926cd\": rpc error: code = NotFound desc = could not find container \"d17f0fe66e125dca99815f10b9d62d3654f065875af1bb42f2d57598170926cd\": container with ID starting with d17f0fe66e125dca99815f10b9d62d3654f065875af1bb42f2d57598170926cd not found: ID does not exist" Oct 31 00:28:14.254700 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:14.254603 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-986d37-5d7d64658c-h2tgp"] Oct 31 00:28:14.260052 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:14.260023 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-986d37-5d7d64658c-h2tgp"] Oct 31 00:28:14.370730 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:14.370702 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-986d37-695476d5cd-99x6j"] Oct 31 00:28:14.379448 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:14.379419 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-986d37-695476d5cd-99x6j"] Oct 31 00:28:15.087141 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:15.087102 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="915e2f48-c25a-4bcf-a1d1-edcc5245dfb8" path="/var/lib/kubelet/pods/915e2f48-c25a-4bcf-a1d1-edcc5245dfb8/volumes" Oct 31 00:28:15.087650 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:15.087628 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f4e38df7-355f-4b47-a0b5-7e0b907f3326" path="/var/lib/kubelet/pods/f4e38df7-355f-4b47-a0b5-7e0b907f3326/volumes" Oct 31 00:28:15.088092 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:15.088072 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc5ca633-21a6-4853-88d1-cccb1e17ad4e" path="/var/lib/kubelet/pods/fc5ca633-21a6-4853-88d1-cccb1e17ad4e/volumes" Oct 31 00:28:15.792785 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:15.792732 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-76b073-78787bc78-d6l68" podUID="0f69c734-4a5f-427d-a682-c5bc7ebaa0b1" containerName="payload-processing" containerID="cri-o://25a5633a1341a3d4a982ecdd62e9411f191df7f94a0641dd2fcdd865c0a3a885" gracePeriod=25 Oct 31 00:28:15.905470 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:15.905398 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-76b073-68467cb4b7-pf7qq" podUID="e63da6f1-8b8f-451d-84c1-9b39411a2a59" containerName="payload-pre-processing" containerID="cri-o://929c28a1f7a209a0d96d173f92e660232d2acf9517c8bf272072c0ff64635e2c" gracePeriod=25 Oct 31 00:28:16.078552 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:16.078526 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-76b073-78787bc78-d6l68" Oct 31 00:28:16.101946 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:16.101913 4927 generic.go:356] "Generic (PLEG): container finished" podID="0f69c734-4a5f-427d-a682-c5bc7ebaa0b1" containerID="25a5633a1341a3d4a982ecdd62e9411f191df7f94a0641dd2fcdd865c0a3a885" exitCode=0 Oct 31 00:28:16.102087 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:16.101949 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-76b073-78787bc78-d6l68" Oct 31 00:28:16.102087 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:16.101987 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-76b073-78787bc78-d6l68" event={"ID":"0f69c734-4a5f-427d-a682-c5bc7ebaa0b1","Type":"ContainerDied","Data":"25a5633a1341a3d4a982ecdd62e9411f191df7f94a0641dd2fcdd865c0a3a885"} Oct 31 00:28:16.102087 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:16.102033 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-76b073-78787bc78-d6l68" event={"ID":"0f69c734-4a5f-427d-a682-c5bc7ebaa0b1","Type":"ContainerDied","Data":"b22e4e3b9a40a5c38a9d4d453b099218e67adb1307b1c52d0d5c3f04d8686448"} Oct 31 00:28:16.102087 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:16.102059 4927 scope.go:122] "RemoveContainer" containerID="25a5633a1341a3d4a982ecdd62e9411f191df7f94a0641dd2fcdd865c0a3a885" Oct 31 00:28:16.113723 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:16.113702 4927 scope.go:122] "RemoveContainer" containerID="25a5633a1341a3d4a982ecdd62e9411f191df7f94a0641dd2fcdd865c0a3a885" Oct 31 00:28:16.114030 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:28:16.114005 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"25a5633a1341a3d4a982ecdd62e9411f191df7f94a0641dd2fcdd865c0a3a885\": container with ID starting with 25a5633a1341a3d4a982ecdd62e9411f191df7f94a0641dd2fcdd865c0a3a885 not found: ID does not exist" containerID="25a5633a1341a3d4a982ecdd62e9411f191df7f94a0641dd2fcdd865c0a3a885" Oct 31 00:28:16.114104 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:16.114041 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25a5633a1341a3d4a982ecdd62e9411f191df7f94a0641dd2fcdd865c0a3a885"} err="failed to get container status \"25a5633a1341a3d4a982ecdd62e9411f191df7f94a0641dd2fcdd865c0a3a885\": rpc error: code = NotFound desc = could not find container \"25a5633a1341a3d4a982ecdd62e9411f191df7f94a0641dd2fcdd865c0a3a885\": container with ID starting with 25a5633a1341a3d4a982ecdd62e9411f191df7f94a0641dd2fcdd865c0a3a885 not found: ID does not exist" Oct 31 00:28:16.136021 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:16.135992 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f69c734-4a5f-427d-a682-c5bc7ebaa0b1-kube-api-access-fzrsk\" (UniqueName: \"kubernetes.io/projected/0f69c734-4a5f-427d-a682-c5bc7ebaa0b1-kube-api-access-fzrsk\") pod \"0f69c734-4a5f-427d-a682-c5bc7ebaa0b1\" (UID: \"0f69c734-4a5f-427d-a682-c5bc7ebaa0b1\") " Oct 31 00:28:16.136137 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:16.136050 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f69c734-4a5f-427d-a682-c5bc7ebaa0b1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f69c734-4a5f-427d-a682-c5bc7ebaa0b1-plugins-config-volume\") pod \"0f69c734-4a5f-427d-a682-c5bc7ebaa0b1\" (UID: \"0f69c734-4a5f-427d-a682-c5bc7ebaa0b1\") " Oct 31 00:28:16.136462 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:16.136435 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f69c734-4a5f-427d-a682-c5bc7ebaa0b1-plugins-config-volume" pod "0f69c734-4a5f-427d-a682-c5bc7ebaa0b1" (UID: "0f69c734-4a5f-427d-a682-c5bc7ebaa0b1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:28:16.136771 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:16.136746 4927 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f69c734-4a5f-427d-a682-c5bc7ebaa0b1-plugins-config-volume\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:16.138312 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:16.138283 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f69c734-4a5f-427d-a682-c5bc7ebaa0b1-kube-api-access-fzrsk" pod "0f69c734-4a5f-427d-a682-c5bc7ebaa0b1" (UID: "0f69c734-4a5f-427d-a682-c5bc7ebaa0b1"). InnerVolumeSpecName "kube-api-access-fzrsk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:28:16.237880 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:16.237842 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fzrsk\" (UniqueName: \"kubernetes.io/projected/0f69c734-4a5f-427d-a682-c5bc7ebaa0b1-kube-api-access-fzrsk\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:16.528486 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:16.528448 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-76b073-78787bc78-d6l68"] Oct 31 00:28:16.532449 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:16.532425 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-76b073-78787bc78-d6l68"] Oct 31 00:28:17.087716 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:17.087680 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f69c734-4a5f-427d-a682-c5bc7ebaa0b1" path="/var/lib/kubelet/pods/0f69c734-4a5f-427d-a682-c5bc7ebaa0b1/volumes" Oct 31 00:28:17.434023 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:17.433983 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-17cd123d-jj5sg"] Oct 31 00:28:17.434569 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:17.434553 4927 cpu_manager.go:414] "containerMap: removing container" podUID="fc5ca633-21a6-4853-88d1-cccb1e17ad4e" containerName="payload-pre-processing" Oct 31 00:28:17.434639 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:17.434570 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc5ca633-21a6-4853-88d1-cccb1e17ad4e" containerName="payload-pre-processing" Oct 31 00:28:17.434639 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:17.434593 4927 cpu_manager.go:414] "containerMap: removing container" podUID="0f69c734-4a5f-427d-a682-c5bc7ebaa0b1" containerName="payload-processing" Oct 31 00:28:17.434639 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:17.434600 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f69c734-4a5f-427d-a682-c5bc7ebaa0b1" containerName="payload-processing" Oct 31 00:28:17.434639 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:17.434609 4927 cpu_manager.go:414] "containerMap: removing container" podUID="915e2f48-c25a-4bcf-a1d1-edcc5245dfb8" containerName="payload-processing" Oct 31 00:28:17.434639 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:17.434614 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="915e2f48-c25a-4bcf-a1d1-edcc5245dfb8" containerName="payload-processing" Oct 31 00:28:17.434639 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:17.434621 4927 cpu_manager.go:414] "containerMap: removing container" podUID="b22587ca-cec2-41f1-8f67-cfac35cfbcdd" containerName="revoke-keys" Oct 31 00:28:17.434639 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:17.434626 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b22587ca-cec2-41f1-8f67-cfac35cfbcdd" containerName="revoke-keys" Oct 31 00:28:17.434858 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:17.434652 4927 cpu_manager.go:414] "containerMap: removing container" podUID="f4e38df7-355f-4b47-a0b5-7e0b907f3326" containerName="main" Oct 31 00:28:17.434858 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:17.434657 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4e38df7-355f-4b47-a0b5-7e0b907f3326" containerName="main" Oct 31 00:28:17.434858 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:17.434747 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4e38df7-355f-4b47-a0b5-7e0b907f3326" containerName="main" Oct 31 00:28:17.434858 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:17.434758 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc5ca633-21a6-4853-88d1-cccb1e17ad4e" containerName="payload-pre-processing" Oct 31 00:28:17.434858 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:17.434766 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f69c734-4a5f-427d-a682-c5bc7ebaa0b1" containerName="payload-processing" Oct 31 00:28:17.434858 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:17.434773 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="915e2f48-c25a-4bcf-a1d1-edcc5245dfb8" containerName="payload-processing" Oct 31 00:28:17.434858 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:17.434779 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="b22587ca-cec2-41f1-8f67-cfac35cfbcdd" containerName="revoke-keys" Oct 31 00:28:17.439467 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:17.439444 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17cd123d-jj5sg" Oct 31 00:28:17.497139 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:17.497098 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-17cd123d-jj5sg"] Oct 31 00:28:17.543641 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:17.543588 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:28:17.543903 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:17.543648 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4fdzs\"" Oct 31 00:28:17.557632 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:17.557597 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd537237-85e9-4a54-8ace-8ca915d2057d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-17cd123d-jj5sg\" (UID: \"dd537237-85e9-4a54-8ace-8ca915d2057d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17cd123d-jj5sg" Oct 31 00:28:17.557817 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:17.557674 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8fbt7\" (UniqueName: \"kubernetes.io/projected/dd537237-85e9-4a54-8ace-8ca915d2057d-kube-api-access-8fbt7\") pod \"maas-api-revoke-sub-17cd123d-jj5sg\" (UID: \"dd537237-85e9-4a54-8ace-8ca915d2057d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17cd123d-jj5sg" Oct 31 00:28:17.557817 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:17.557759 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd537237-85e9-4a54-8ace-8ca915d2057d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-17cd123d-jj5sg\" (UID: \"dd537237-85e9-4a54-8ace-8ca915d2057d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17cd123d-jj5sg" Oct 31 00:28:17.562540 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:17.560939 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-76b073-6d97458769-pplqh"] Oct 31 00:28:17.562540 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:17.561254 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-76b073-6d97458769-pplqh" podUID="7d4c3ca5-2a40-4959-b4df-8a24cf25b043" containerName="maas-api" containerID="cri-o://e9771c90fb3593a57dcc470c6f32dd42cdc8b1515a3292854cb03fb9ba13a3b7" gracePeriod=30 Oct 31 00:28:17.642610 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:17.642574 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:28:17.658696 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:17.658667 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd537237-85e9-4a54-8ace-8ca915d2057d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-17cd123d-jj5sg\" (UID: \"dd537237-85e9-4a54-8ace-8ca915d2057d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17cd123d-jj5sg" Oct 31 00:28:17.658858 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:17.658774 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd537237-85e9-4a54-8ace-8ca915d2057d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-17cd123d-jj5sg\" (UID: \"dd537237-85e9-4a54-8ace-8ca915d2057d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17cd123d-jj5sg" Oct 31 00:28:17.658858 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:17.658813 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8fbt7\" (UniqueName: \"kubernetes.io/projected/dd537237-85e9-4a54-8ace-8ca915d2057d-kube-api-access-8fbt7\") pod \"maas-api-revoke-sub-17cd123d-jj5sg\" (UID: \"dd537237-85e9-4a54-8ace-8ca915d2057d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17cd123d-jj5sg" Oct 31 00:28:17.659472 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:17.659446 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd537237-85e9-4a54-8ace-8ca915d2057d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-17cd123d-jj5sg\" (UID: \"dd537237-85e9-4a54-8ace-8ca915d2057d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17cd123d-jj5sg" Oct 31 00:28:17.660960 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:17.660937 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd537237-85e9-4a54-8ace-8ca915d2057d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-17cd123d-jj5sg\" (UID: \"dd537237-85e9-4a54-8ace-8ca915d2057d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17cd123d-jj5sg" Oct 31 00:28:17.724216 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:17.724142 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8fbt7\" (UniqueName: \"kubernetes.io/projected/dd537237-85e9-4a54-8ace-8ca915d2057d-kube-api-access-8fbt7\") pod \"maas-api-revoke-sub-17cd123d-jj5sg\" (UID: \"dd537237-85e9-4a54-8ace-8ca915d2057d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17cd123d-jj5sg" Oct 31 00:28:17.754173 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:17.753855 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17cd123d-jj5sg" Oct 31 00:28:17.844003 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:17.843976 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-76b073-6d97458769-pplqh" Oct 31 00:28:17.974452 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:17.973423 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7d4c3ca5-2a40-4959-b4df-8a24cf25b043-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d4c3ca5-2a40-4959-b4df-8a24cf25b043-metrics-certs\") pod \"7d4c3ca5-2a40-4959-b4df-8a24cf25b043\" (UID: \"7d4c3ca5-2a40-4959-b4df-8a24cf25b043\") " Oct 31 00:28:17.974452 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:17.973513 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d4c3ca5-2a40-4959-b4df-8a24cf25b043-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d4c3ca5-2a40-4959-b4df-8a24cf25b043-openshift-service-ca\") pod \"7d4c3ca5-2a40-4959-b4df-8a24cf25b043\" (UID: \"7d4c3ca5-2a40-4959-b4df-8a24cf25b043\") " Oct 31 00:28:17.974452 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:17.973645 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d4c3ca5-2a40-4959-b4df-8a24cf25b043-kube-api-access-cpqqp\" (UniqueName: \"kubernetes.io/projected/7d4c3ca5-2a40-4959-b4df-8a24cf25b043-kube-api-access-cpqqp\") pod \"7d4c3ca5-2a40-4959-b4df-8a24cf25b043\" (UID: \"7d4c3ca5-2a40-4959-b4df-8a24cf25b043\") " Oct 31 00:28:17.974452 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:17.973694 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7d4c3ca5-2a40-4959-b4df-8a24cf25b043-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7d4c3ca5-2a40-4959-b4df-8a24cf25b043-maas-api-tls\") pod \"7d4c3ca5-2a40-4959-b4df-8a24cf25b043\" (UID: \"7d4c3ca5-2a40-4959-b4df-8a24cf25b043\") " Oct 31 00:28:17.974818 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:17.974510 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d4c3ca5-2a40-4959-b4df-8a24cf25b043-openshift-service-ca" pod "7d4c3ca5-2a40-4959-b4df-8a24cf25b043" (UID: "7d4c3ca5-2a40-4959-b4df-8a24cf25b043"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:28:17.976809 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:17.976779 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d4c3ca5-2a40-4959-b4df-8a24cf25b043-metrics-certs" pod "7d4c3ca5-2a40-4959-b4df-8a24cf25b043" (UID: "7d4c3ca5-2a40-4959-b4df-8a24cf25b043"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:28:17.976989 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:17.976963 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d4c3ca5-2a40-4959-b4df-8a24cf25b043-maas-api-tls" pod "7d4c3ca5-2a40-4959-b4df-8a24cf25b043" (UID: "7d4c3ca5-2a40-4959-b4df-8a24cf25b043"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:28:17.978382 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:17.977779 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d4c3ca5-2a40-4959-b4df-8a24cf25b043-kube-api-access-cpqqp" pod "7d4c3ca5-2a40-4959-b4df-8a24cf25b043" (UID: "7d4c3ca5-2a40-4959-b4df-8a24cf25b043"). InnerVolumeSpecName "kube-api-access-cpqqp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:28:18.081650 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:18.080684 4927 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d4c3ca5-2a40-4959-b4df-8a24cf25b043-openshift-service-ca\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:18.081650 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:18.080722 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cpqqp\" (UniqueName: \"kubernetes.io/projected/7d4c3ca5-2a40-4959-b4df-8a24cf25b043-kube-api-access-cpqqp\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:18.081650 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:18.080737 4927 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7d4c3ca5-2a40-4959-b4df-8a24cf25b043-maas-api-tls\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:18.081650 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:18.080751 4927 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d4c3ca5-2a40-4959-b4df-8a24cf25b043-metrics-certs\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:18.127381 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:18.126343 4927 generic.go:356] "Generic (PLEG): container finished" podID="7d4c3ca5-2a40-4959-b4df-8a24cf25b043" containerID="e9771c90fb3593a57dcc470c6f32dd42cdc8b1515a3292854cb03fb9ba13a3b7" exitCode=0 Oct 31 00:28:18.127381 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:18.126468 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-76b073-6d97458769-pplqh" event={"ID":"7d4c3ca5-2a40-4959-b4df-8a24cf25b043","Type":"ContainerDied","Data":"e9771c90fb3593a57dcc470c6f32dd42cdc8b1515a3292854cb03fb9ba13a3b7"} Oct 31 00:28:18.127381 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:18.126520 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-76b073-6d97458769-pplqh" event={"ID":"7d4c3ca5-2a40-4959-b4df-8a24cf25b043","Type":"ContainerDied","Data":"768d2be4d7dea41fdade7a3b3bc7cc50d6cba35ca3c7fb3944b80e3fc62ee5ef"} Oct 31 00:28:18.127381 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:18.126551 4927 scope.go:122] "RemoveContainer" containerID="e9771c90fb3593a57dcc470c6f32dd42cdc8b1515a3292854cb03fb9ba13a3b7" Oct 31 00:28:18.127381 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:18.126693 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-76b073-6d97458769-pplqh" Oct 31 00:28:18.138861 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:18.138842 4927 scope.go:122] "RemoveContainer" containerID="e9771c90fb3593a57dcc470c6f32dd42cdc8b1515a3292854cb03fb9ba13a3b7" Oct 31 00:28:18.139119 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:28:18.139098 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9771c90fb3593a57dcc470c6f32dd42cdc8b1515a3292854cb03fb9ba13a3b7\": container with ID starting with e9771c90fb3593a57dcc470c6f32dd42cdc8b1515a3292854cb03fb9ba13a3b7 not found: ID does not exist" containerID="e9771c90fb3593a57dcc470c6f32dd42cdc8b1515a3292854cb03fb9ba13a3b7" Oct 31 00:28:18.139195 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:18.139131 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9771c90fb3593a57dcc470c6f32dd42cdc8b1515a3292854cb03fb9ba13a3b7"} err="failed to get container status \"e9771c90fb3593a57dcc470c6f32dd42cdc8b1515a3292854cb03fb9ba13a3b7\": rpc error: code = NotFound desc = could not find container \"e9771c90fb3593a57dcc470c6f32dd42cdc8b1515a3292854cb03fb9ba13a3b7\": container with ID starting with e9771c90fb3593a57dcc470c6f32dd42cdc8b1515a3292854cb03fb9ba13a3b7 not found: ID does not exist" Oct 31 00:28:18.191235 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:18.191205 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-17cd123d-jj5sg"] Oct 31 00:28:18.192081 ip-10-0-141-180 kubenswrapper[4927]: W1031 00:28:18.192055 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd537237_85e9_4a54_8ace_8ca915d2057d.slice/crio-255b4b50f3c69106013fe8df6e0effeefdad25599e74c17d3d87e64404959ada WatchSource:0}: Error finding container 255b4b50f3c69106013fe8df6e0effeefdad25599e74c17d3d87e64404959ada: Status 404 returned error can't find the container with id 255b4b50f3c69106013fe8df6e0effeefdad25599e74c17d3d87e64404959ada Oct 31 00:28:18.301066 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:18.300990 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-76b073-6d97458769-pplqh"] Oct 31 00:28:18.304002 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:18.303982 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-76b073-6d97458769-pplqh"] Oct 31 00:28:19.060694 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:19.060655 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:28:19.086695 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:19.086663 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d4c3ca5-2a40-4959-b4df-8a24cf25b043" path="/var/lib/kubelet/pods/7d4c3ca5-2a40-4959-b4df-8a24cf25b043/volumes" Oct 31 00:28:19.135371 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:19.135314 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17cd123d-jj5sg" event={"ID":"dd537237-85e9-4a54-8ace-8ca915d2057d","Type":"ContainerStarted","Data":"3cba08619886d1ebe196206e8895cbc422324fc8e3d2626f13c9a70ae3827283"} Oct 31 00:28:19.135850 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:19.135392 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17cd123d-jj5sg" event={"ID":"dd537237-85e9-4a54-8ace-8ca915d2057d","Type":"ContainerStarted","Data":"255b4b50f3c69106013fe8df6e0effeefdad25599e74c17d3d87e64404959ada"} Oct 31 00:28:19.256362 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:19.256286 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17cd123d-jj5sg" podStartSLOduration=2.256272001 podStartE2EDuration="2.256272001s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:28:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:28:19.25350173 +0000 UTC m=+2812.289832557" watchObservedRunningTime="2026-10-31 00:28:19.256272001 +0000 UTC m=+2812.292602824" Oct 31 00:28:19.262541 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:19.262507 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:28:19.449179 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:19.449141 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:28:19.560194 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:19.560159 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-76b073-7zs2f"] Oct 31 00:28:19.563470 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:19.563443 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-76b073-7zs2f"] Oct 31 00:28:19.678813 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:28:19.678778 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sqv69" containerName="extractor" Oct 31 00:28:20.017984 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:20.017940 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:28:20.146211 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:20.146179 4927 generic.go:356] "Generic (PLEG): container finished" podID="dd537237-85e9-4a54-8ace-8ca915d2057d" containerID="3cba08619886d1ebe196206e8895cbc422324fc8e3d2626f13c9a70ae3827283" exitCode=0 Oct 31 00:28:20.146650 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:20.146271 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17cd123d-jj5sg" event={"ID":"dd537237-85e9-4a54-8ace-8ca915d2057d","Type":"ContainerDied","Data":"3cba08619886d1ebe196206e8895cbc422324fc8e3d2626f13c9a70ae3827283"} Oct 31 00:28:21.085916 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:21.085881 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b22587ca-cec2-41f1-8f67-cfac35cfbcdd" path="/var/lib/kubelet/pods/b22587ca-cec2-41f1-8f67-cfac35cfbcdd/volumes" Oct 31 00:28:21.306970 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:21.306948 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17cd123d-jj5sg" Oct 31 00:28:21.415886 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:21.415857 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dd537237-85e9-4a54-8ace-8ca915d2057d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd537237-85e9-4a54-8ace-8ca915d2057d-maas-api-xks-ca\") pod \"dd537237-85e9-4a54-8ace-8ca915d2057d\" (UID: \"dd537237-85e9-4a54-8ace-8ca915d2057d\") " Oct 31 00:28:21.416030 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:21.415907 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd537237-85e9-4a54-8ace-8ca915d2057d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd537237-85e9-4a54-8ace-8ca915d2057d-maas-api-openshift-ca\") pod \"dd537237-85e9-4a54-8ace-8ca915d2057d\" (UID: \"dd537237-85e9-4a54-8ace-8ca915d2057d\") " Oct 31 00:28:21.416030 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:21.415992 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd537237-85e9-4a54-8ace-8ca915d2057d-kube-api-access-8fbt7\" (UniqueName: \"kubernetes.io/projected/dd537237-85e9-4a54-8ace-8ca915d2057d-kube-api-access-8fbt7\") pod \"dd537237-85e9-4a54-8ace-8ca915d2057d\" (UID: \"dd537237-85e9-4a54-8ace-8ca915d2057d\") " Oct 31 00:28:21.416290 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:21.416266 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd537237-85e9-4a54-8ace-8ca915d2057d-maas-api-openshift-ca" pod "dd537237-85e9-4a54-8ace-8ca915d2057d" (UID: "dd537237-85e9-4a54-8ace-8ca915d2057d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:28:21.418012 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:21.417990 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd537237-85e9-4a54-8ace-8ca915d2057d-kube-api-access-8fbt7" pod "dd537237-85e9-4a54-8ace-8ca915d2057d" (UID: "dd537237-85e9-4a54-8ace-8ca915d2057d"). InnerVolumeSpecName "kube-api-access-8fbt7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:28:21.418083 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:21.418045 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dd537237-85e9-4a54-8ace-8ca915d2057d-maas-api-xks-ca" pod "dd537237-85e9-4a54-8ace-8ca915d2057d" (UID: "dd537237-85e9-4a54-8ace-8ca915d2057d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:28:21.517685 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:21.517651 4927 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd537237-85e9-4a54-8ace-8ca915d2057d-maas-api-xks-ca\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:21.517685 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:21.517680 4927 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd537237-85e9-4a54-8ace-8ca915d2057d-maas-api-openshift-ca\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:21.517685 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:21.517692 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8fbt7\" (UniqueName: \"kubernetes.io/projected/dd537237-85e9-4a54-8ace-8ca915d2057d-kube-api-access-8fbt7\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:22.163132 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:22.163091 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17cd123d-jj5sg" Oct 31 00:28:22.163378 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:22.163128 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17cd123d-jj5sg" event={"ID":"dd537237-85e9-4a54-8ace-8ca915d2057d","Type":"ContainerDied","Data":"255b4b50f3c69106013fe8df6e0effeefdad25599e74c17d3d87e64404959ada"} Oct 31 00:28:22.163378 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:22.163187 4927 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="255b4b50f3c69106013fe8df6e0effeefdad25599e74c17d3d87e64404959ada" Oct 31 00:28:24.352043 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:24.352008 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:28:25.163037 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:25.162991 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-76b073-68467cb4b7-pf7qq" podUID="e63da6f1-8b8f-451d-84c1-9b39411a2a59" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 00:28:26.183405 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:26.183374 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:28:26.407755 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:26.407714 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:28:26.554433 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:26.554354 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-76b073-openshift-default-697d759dd5-ql7sd"] Oct 31 00:28:26.554628 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:26.554601 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-76b073-openshift-default-697d759dd5-ql7sd" podUID="78b89047-2bad-472e-a80d-0ab8f1755877" containerName="istio-proxy" containerID="cri-o://00f406a06fb7820ebcbd0fc66650dde9c2b11ce43d216b26ac4b9206655d1205" gracePeriod=30 Oct 31 00:28:26.565802 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:26.565766 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:28:27.413814 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:27.413783 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-49837c-d4w6f"] Oct 31 00:28:27.414347 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:27.414319 4927 cpu_manager.go:414] "containerMap: removing container" podUID="dd537237-85e9-4a54-8ace-8ca915d2057d" containerName="revoke-keys" Oct 31 00:28:27.414347 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:27.414345 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd537237-85e9-4a54-8ace-8ca915d2057d" containerName="revoke-keys" Oct 31 00:28:27.414441 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:27.414366 4927 cpu_manager.go:414] "containerMap: removing container" podUID="7d4c3ca5-2a40-4959-b4df-8a24cf25b043" containerName="maas-api" Oct 31 00:28:27.414441 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:27.414371 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d4c3ca5-2a40-4959-b4df-8a24cf25b043" containerName="maas-api" Oct 31 00:28:27.414504 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:27.414458 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd537237-85e9-4a54-8ace-8ca915d2057d" containerName="revoke-keys" Oct 31 00:28:27.414504 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:27.414471 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d4c3ca5-2a40-4959-b4df-8a24cf25b043" containerName="maas-api" Oct 31 00:28:27.417987 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:27.417972 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-49837c-d4w6f" Oct 31 00:28:27.473358 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:27.473255 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:28:27.473358 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:27.473307 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4fdzs\"" Oct 31 00:28:27.485948 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:27.485910 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e4c7d3b-23b7-4729-b15e-851878093e45-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-49837c-d4w6f\" (UID: \"4e4c7d3b-23b7-4729-b15e-851878093e45\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-49837c-d4w6f" Oct 31 00:28:27.486132 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:27.486095 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qlp4l\" (UniqueName: \"kubernetes.io/projected/4e4c7d3b-23b7-4729-b15e-851878093e45-kube-api-access-qlp4l\") pod \"maas-api-revoke-keys-e2e-inf-b-49837c-d4w6f\" (UID: \"4e4c7d3b-23b7-4729-b15e-851878093e45\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-49837c-d4w6f" Oct 31 00:28:27.486132 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:27.486128 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e4c7d3b-23b7-4729-b15e-851878093e45-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-49837c-d4w6f\" (UID: \"4e4c7d3b-23b7-4729-b15e-851878093e45\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-49837c-d4w6f" Oct 31 00:28:27.527861 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:27.527829 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-49837c-d4w6f"] Oct 31 00:28:27.587128 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:27.587094 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlp4l\" (UniqueName: \"kubernetes.io/projected/4e4c7d3b-23b7-4729-b15e-851878093e45-kube-api-access-qlp4l\") pod \"maas-api-revoke-keys-e2e-inf-b-49837c-d4w6f\" (UID: \"4e4c7d3b-23b7-4729-b15e-851878093e45\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-49837c-d4w6f" Oct 31 00:28:27.587128 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:27.587133 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e4c7d3b-23b7-4729-b15e-851878093e45-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-49837c-d4w6f\" (UID: \"4e4c7d3b-23b7-4729-b15e-851878093e45\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-49837c-d4w6f" Oct 31 00:28:27.587446 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:27.587193 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e4c7d3b-23b7-4729-b15e-851878093e45-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-49837c-d4w6f\" (UID: \"4e4c7d3b-23b7-4729-b15e-851878093e45\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-49837c-d4w6f" Oct 31 00:28:27.587937 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:27.587916 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e4c7d3b-23b7-4729-b15e-851878093e45-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-49837c-d4w6f\" (UID: \"4e4c7d3b-23b7-4729-b15e-851878093e45\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-49837c-d4w6f" Oct 31 00:28:27.589547 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:27.589524 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e4c7d3b-23b7-4729-b15e-851878093e45-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-49837c-d4w6f\" (UID: \"4e4c7d3b-23b7-4729-b15e-851878093e45\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-49837c-d4w6f" Oct 31 00:28:27.649289 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:27.649256 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qlp4l\" (UniqueName: \"kubernetes.io/projected/4e4c7d3b-23b7-4729-b15e-851878093e45-kube-api-access-qlp4l\") pod \"maas-api-revoke-keys-e2e-inf-b-49837c-d4w6f\" (UID: \"4e4c7d3b-23b7-4729-b15e-851878093e45\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-49837c-d4w6f" Oct 31 00:28:27.731380 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:27.731275 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-49837c-d4w6f" Oct 31 00:28:28.121014 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:28.120983 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-49837c-d4w6f"] Oct 31 00:28:28.122574 ip-10-0-141-180 kubenswrapper[4927]: W1031 00:28:28.122547 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e4c7d3b_23b7_4729_b15e_851878093e45.slice/crio-c46cb860cd30c43356a15d9143122be4b58aea875343b780ce4bb1cd3103acc1 WatchSource:0}: Error finding container c46cb860cd30c43356a15d9143122be4b58aea875343b780ce4bb1cd3103acc1: Status 404 returned error can't find the container with id c46cb860cd30c43356a15d9143122be4b58aea875343b780ce4bb1cd3103acc1 Oct 31 00:28:28.206887 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:28.206857 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-49837c-d4w6f" event={"ID":"4e4c7d3b-23b7-4729-b15e-851878093e45","Type":"ContainerStarted","Data":"c46cb860cd30c43356a15d9143122be4b58aea875343b780ce4bb1cd3103acc1"} Oct 31 00:28:29.126585 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:29.126551 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:28:29.215555 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:29.215468 4927 generic.go:356] "Generic (PLEG): container finished" podID="4e4c7d3b-23b7-4729-b15e-851878093e45" containerID="002ae68aa4fde212d44dc94e9ec1a9a24e875b14a3be99932a6f7f21f795a97d" exitCode=0 Oct 31 00:28:29.215555 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:29.215533 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-49837c-d4w6f" event={"ID":"4e4c7d3b-23b7-4729-b15e-851878093e45","Type":"ContainerDied","Data":"002ae68aa4fde212d44dc94e9ec1a9a24e875b14a3be99932a6f7f21f795a97d"} Oct 31 00:28:30.387828 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:30.387797 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-49837c-d4w6f" Oct 31 00:28:30.522638 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:30.521680 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4e4c7d3b-23b7-4729-b15e-851878093e45-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e4c7d3b-23b7-4729-b15e-851878093e45-maas-api-xks-ca\") pod \"4e4c7d3b-23b7-4729-b15e-851878093e45\" (UID: \"4e4c7d3b-23b7-4729-b15e-851878093e45\") " Oct 31 00:28:30.522638 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:30.521745 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e4c7d3b-23b7-4729-b15e-851878093e45-kube-api-access-qlp4l\" (UniqueName: \"kubernetes.io/projected/4e4c7d3b-23b7-4729-b15e-851878093e45-kube-api-access-qlp4l\") pod \"4e4c7d3b-23b7-4729-b15e-851878093e45\" (UID: \"4e4c7d3b-23b7-4729-b15e-851878093e45\") " Oct 31 00:28:30.522638 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:30.521790 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e4c7d3b-23b7-4729-b15e-851878093e45-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e4c7d3b-23b7-4729-b15e-851878093e45-maas-api-openshift-ca\") pod \"4e4c7d3b-23b7-4729-b15e-851878093e45\" (UID: \"4e4c7d3b-23b7-4729-b15e-851878093e45\") " Oct 31 00:28:30.522638 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:30.522535 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e4c7d3b-23b7-4729-b15e-851878093e45-maas-api-openshift-ca" pod "4e4c7d3b-23b7-4729-b15e-851878093e45" (UID: "4e4c7d3b-23b7-4729-b15e-851878093e45"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:28:30.524230 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:30.524193 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e4c7d3b-23b7-4729-b15e-851878093e45-maas-api-xks-ca" pod "4e4c7d3b-23b7-4729-b15e-851878093e45" (UID: "4e4c7d3b-23b7-4729-b15e-851878093e45"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:28:30.524566 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:30.524540 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e4c7d3b-23b7-4729-b15e-851878093e45-kube-api-access-qlp4l" pod "4e4c7d3b-23b7-4729-b15e-851878093e45" (UID: "4e4c7d3b-23b7-4729-b15e-851878093e45"). InnerVolumeSpecName "kube-api-access-qlp4l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:28:30.623374 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:30.623320 4927 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e4c7d3b-23b7-4729-b15e-851878093e45-maas-api-xks-ca\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:30.623374 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:30.623366 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qlp4l\" (UniqueName: \"kubernetes.io/projected/4e4c7d3b-23b7-4729-b15e-851878093e45-kube-api-access-qlp4l\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:30.623374 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:30.623378 4927 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e4c7d3b-23b7-4729-b15e-851878093e45-maas-api-openshift-ca\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:31.231565 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:31.231530 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-49837c-d4w6f" event={"ID":"4e4c7d3b-23b7-4729-b15e-851878093e45","Type":"ContainerDied","Data":"c46cb860cd30c43356a15d9143122be4b58aea875343b780ce4bb1cd3103acc1"} Oct 31 00:28:31.231565 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:31.231557 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-49837c-d4w6f" Oct 31 00:28:31.231781 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:31.231583 4927 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c46cb860cd30c43356a15d9143122be4b58aea875343b780ce4bb1cd3103acc1" Oct 31 00:28:31.818484 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:31.818430 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-76b073-openshift-default-697d759dd5-ql7sd" Oct 31 00:28:31.936772 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:31.936740 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78b89047-2bad-472e-a80d-0ab8f1755877-kube-api-access-cdqhf\" (UniqueName: \"kubernetes.io/projected/78b89047-2bad-472e-a80d-0ab8f1755877-kube-api-access-cdqhf\") pod \"78b89047-2bad-472e-a80d-0ab8f1755877\" (UID: \"78b89047-2bad-472e-a80d-0ab8f1755877\") " Oct 31 00:28:31.936934 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:31.936782 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78b89047-2bad-472e-a80d-0ab8f1755877-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78b89047-2bad-472e-a80d-0ab8f1755877-workload-certs\") pod \"78b89047-2bad-472e-a80d-0ab8f1755877\" (UID: \"78b89047-2bad-472e-a80d-0ab8f1755877\") " Oct 31 00:28:31.936934 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:31.936816 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78b89047-2bad-472e-a80d-0ab8f1755877-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78b89047-2bad-472e-a80d-0ab8f1755877-istio-data\") pod \"78b89047-2bad-472e-a80d-0ab8f1755877\" (UID: \"78b89047-2bad-472e-a80d-0ab8f1755877\") " Oct 31 00:28:31.936934 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:31.936857 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78b89047-2bad-472e-a80d-0ab8f1755877-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78b89047-2bad-472e-a80d-0ab8f1755877-workload-socket\") pod \"78b89047-2bad-472e-a80d-0ab8f1755877\" (UID: \"78b89047-2bad-472e-a80d-0ab8f1755877\") " Oct 31 00:28:31.936934 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:31.936913 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78b89047-2bad-472e-a80d-0ab8f1755877-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78b89047-2bad-472e-a80d-0ab8f1755877-istio-envoy\") pod \"78b89047-2bad-472e-a80d-0ab8f1755877\" (UID: \"78b89047-2bad-472e-a80d-0ab8f1755877\") " Oct 31 00:28:31.937189 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:31.936936 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78b89047-2bad-472e-a80d-0ab8f1755877-istio-token\" (UniqueName: \"kubernetes.io/projected/78b89047-2bad-472e-a80d-0ab8f1755877-istio-token\") pod \"78b89047-2bad-472e-a80d-0ab8f1755877\" (UID: \"78b89047-2bad-472e-a80d-0ab8f1755877\") " Oct 31 00:28:31.937189 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:31.937002 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/78b89047-2bad-472e-a80d-0ab8f1755877-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78b89047-2bad-472e-a80d-0ab8f1755877-istio-podinfo\") pod \"78b89047-2bad-472e-a80d-0ab8f1755877\" (UID: \"78b89047-2bad-472e-a80d-0ab8f1755877\") " Oct 31 00:28:31.937189 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:31.937046 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/78b89047-2bad-472e-a80d-0ab8f1755877-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78b89047-2bad-472e-a80d-0ab8f1755877-istiod-ca-cert\") pod \"78b89047-2bad-472e-a80d-0ab8f1755877\" (UID: \"78b89047-2bad-472e-a80d-0ab8f1755877\") " Oct 31 00:28:31.937189 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:31.937112 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78b89047-2bad-472e-a80d-0ab8f1755877-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78b89047-2bad-472e-a80d-0ab8f1755877-credential-socket\") pod \"78b89047-2bad-472e-a80d-0ab8f1755877\" (UID: \"78b89047-2bad-472e-a80d-0ab8f1755877\") " Oct 31 00:28:31.937189 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:31.937143 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78b89047-2bad-472e-a80d-0ab8f1755877-istio-data" pod "78b89047-2bad-472e-a80d-0ab8f1755877" (UID: "78b89047-2bad-472e-a80d-0ab8f1755877"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:28:31.937517 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:31.937255 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78b89047-2bad-472e-a80d-0ab8f1755877-workload-socket" pod "78b89047-2bad-472e-a80d-0ab8f1755877" (UID: "78b89047-2bad-472e-a80d-0ab8f1755877"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:28:31.937585 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:31.937516 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78b89047-2bad-472e-a80d-0ab8f1755877-credential-socket" pod "78b89047-2bad-472e-a80d-0ab8f1755877" (UID: "78b89047-2bad-472e-a80d-0ab8f1755877"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:28:31.937585 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:31.937533 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78b89047-2bad-472e-a80d-0ab8f1755877-istiod-ca-cert" pod "78b89047-2bad-472e-a80d-0ab8f1755877" (UID: "78b89047-2bad-472e-a80d-0ab8f1755877"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:28:31.937691 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:31.937536 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78b89047-2bad-472e-a80d-0ab8f1755877-workload-certs" pod "78b89047-2bad-472e-a80d-0ab8f1755877" (UID: "78b89047-2bad-472e-a80d-0ab8f1755877"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:28:31.937691 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:31.937680 4927 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78b89047-2bad-472e-a80d-0ab8f1755877-workload-socket\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:31.937802 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:31.937700 4927 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78b89047-2bad-472e-a80d-0ab8f1755877-istiod-ca-cert\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:31.937802 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:31.937714 4927 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78b89047-2bad-472e-a80d-0ab8f1755877-credential-socket\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:31.937802 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:31.937728 4927 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78b89047-2bad-472e-a80d-0ab8f1755877-istio-data\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:31.939252 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:31.939225 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78b89047-2bad-472e-a80d-0ab8f1755877-istio-envoy" pod "78b89047-2bad-472e-a80d-0ab8f1755877" (UID: "78b89047-2bad-472e-a80d-0ab8f1755877"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:28:31.939252 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:31.939245 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78b89047-2bad-472e-a80d-0ab8f1755877-kube-api-access-cdqhf" pod "78b89047-2bad-472e-a80d-0ab8f1755877" (UID: "78b89047-2bad-472e-a80d-0ab8f1755877"). InnerVolumeSpecName "kube-api-access-cdqhf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:28:31.939252 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:31.939228 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78b89047-2bad-472e-a80d-0ab8f1755877-istio-token" pod "78b89047-2bad-472e-a80d-0ab8f1755877" (UID: "78b89047-2bad-472e-a80d-0ab8f1755877"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:28:31.939458 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:31.939273 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/78b89047-2bad-472e-a80d-0ab8f1755877-istio-podinfo" pod "78b89047-2bad-472e-a80d-0ab8f1755877" (UID: "78b89047-2bad-472e-a80d-0ab8f1755877"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:28:32.039146 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:32.039107 4927 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78b89047-2bad-472e-a80d-0ab8f1755877-istio-podinfo\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:32.039146 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:32.039145 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cdqhf\" (UniqueName: \"kubernetes.io/projected/78b89047-2bad-472e-a80d-0ab8f1755877-kube-api-access-cdqhf\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:32.039326 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:32.039161 4927 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78b89047-2bad-472e-a80d-0ab8f1755877-workload-certs\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:32.039326 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:32.039175 4927 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78b89047-2bad-472e-a80d-0ab8f1755877-istio-envoy\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:32.039326 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:32.039188 4927 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/78b89047-2bad-472e-a80d-0ab8f1755877-istio-token\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:32.240788 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:32.240749 4927 generic.go:356] "Generic (PLEG): container finished" podID="78b89047-2bad-472e-a80d-0ab8f1755877" containerID="00f406a06fb7820ebcbd0fc66650dde9c2b11ce43d216b26ac4b9206655d1205" exitCode=0 Oct 31 00:28:32.240951 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:32.240808 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-76b073-openshift-default-697d759dd5-ql7sd" Oct 31 00:28:32.240951 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:32.240837 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-76b073-openshift-default-697d759dd5-ql7sd" event={"ID":"78b89047-2bad-472e-a80d-0ab8f1755877","Type":"ContainerDied","Data":"00f406a06fb7820ebcbd0fc66650dde9c2b11ce43d216b26ac4b9206655d1205"} Oct 31 00:28:32.240951 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:32.240894 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-76b073-openshift-default-697d759dd5-ql7sd" event={"ID":"78b89047-2bad-472e-a80d-0ab8f1755877","Type":"ContainerDied","Data":"c4906276f547c7ece3d9b2f5e76254af194ba639ea5bedf8417cc3abc6cb6078"} Oct 31 00:28:32.240951 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:32.240917 4927 scope.go:122] "RemoveContainer" containerID="00f406a06fb7820ebcbd0fc66650dde9c2b11ce43d216b26ac4b9206655d1205" Oct 31 00:28:32.252192 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:32.252176 4927 scope.go:122] "RemoveContainer" containerID="cf6775d7b3f36612239f5a7641c6558beb5c7e657831a0ecd62d672653c6541e" Oct 31 00:28:32.264513 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:32.264492 4927 scope.go:122] "RemoveContainer" containerID="00f406a06fb7820ebcbd0fc66650dde9c2b11ce43d216b26ac4b9206655d1205" Oct 31 00:28:32.264781 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:28:32.264761 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00f406a06fb7820ebcbd0fc66650dde9c2b11ce43d216b26ac4b9206655d1205\": container with ID starting with 00f406a06fb7820ebcbd0fc66650dde9c2b11ce43d216b26ac4b9206655d1205 not found: ID does not exist" containerID="00f406a06fb7820ebcbd0fc66650dde9c2b11ce43d216b26ac4b9206655d1205" Oct 31 00:28:32.264843 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:32.264790 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00f406a06fb7820ebcbd0fc66650dde9c2b11ce43d216b26ac4b9206655d1205"} err="failed to get container status \"00f406a06fb7820ebcbd0fc66650dde9c2b11ce43d216b26ac4b9206655d1205\": rpc error: code = NotFound desc = could not find container \"00f406a06fb7820ebcbd0fc66650dde9c2b11ce43d216b26ac4b9206655d1205\": container with ID starting with 00f406a06fb7820ebcbd0fc66650dde9c2b11ce43d216b26ac4b9206655d1205 not found: ID does not exist" Oct 31 00:28:32.264843 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:32.264808 4927 scope.go:122] "RemoveContainer" containerID="cf6775d7b3f36612239f5a7641c6558beb5c7e657831a0ecd62d672653c6541e" Oct 31 00:28:32.265050 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:28:32.265019 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf6775d7b3f36612239f5a7641c6558beb5c7e657831a0ecd62d672653c6541e\": container with ID starting with cf6775d7b3f36612239f5a7641c6558beb5c7e657831a0ecd62d672653c6541e not found: ID does not exist" containerID="cf6775d7b3f36612239f5a7641c6558beb5c7e657831a0ecd62d672653c6541e" Oct 31 00:28:32.265168 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:32.265050 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf6775d7b3f36612239f5a7641c6558beb5c7e657831a0ecd62d672653c6541e"} err="failed to get container status \"cf6775d7b3f36612239f5a7641c6558beb5c7e657831a0ecd62d672653c6541e\": rpc error: code = NotFound desc = could not find container \"cf6775d7b3f36612239f5a7641c6558beb5c7e657831a0ecd62d672653c6541e\": container with ID starting with cf6775d7b3f36612239f5a7641c6558beb5c7e657831a0ecd62d672653c6541e not found: ID does not exist" Oct 31 00:28:32.412629 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:32.412598 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-76b073-openshift-default-697d759dd5-ql7sd"] Oct 31 00:28:32.463827 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:32.463791 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-76b073-openshift-default-697d759dd5-ql7sd"] Oct 31 00:28:33.086302 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:33.086269 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="78b89047-2bad-472e-a80d-0ab8f1755877" path="/var/lib/kubelet/pods/78b89047-2bad-472e-a80d-0ab8f1755877/volumes" Oct 31 00:28:33.411910 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:33.411879 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:28:34.159153 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:34.159121 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:28:34.342482 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:34.342446 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:28:34.876701 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:34.876665 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:28:35.034324 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:35.034286 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:28:35.162695 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:35.162635 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-76b073-68467cb4b7-pf7qq" podUID="e63da6f1-8b8f-451d-84c1-9b39411a2a59" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 00:28:35.187579 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:35.187536 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:28:35.339973 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:35.339935 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:28:35.432935 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:35.432842 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-34f1ce-tlltn"] Oct 31 00:28:35.483263 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:35.483231 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-34f1ce-tlltn"] Oct 31 00:28:35.878942 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:35.878856 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:28:36.214088 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:36.214060 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-76b073-68467cb4b7-pf7qq" Oct 31 00:28:36.274103 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:36.274065 4927 generic.go:356] "Generic (PLEG): container finished" podID="e63da6f1-8b8f-451d-84c1-9b39411a2a59" containerID="929c28a1f7a209a0d96d173f92e660232d2acf9517c8bf272072c0ff64635e2c" exitCode=0 Oct 31 00:28:36.274103 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:36.274093 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-76b073-68467cb4b7-pf7qq" event={"ID":"e63da6f1-8b8f-451d-84c1-9b39411a2a59","Type":"ContainerDied","Data":"929c28a1f7a209a0d96d173f92e660232d2acf9517c8bf272072c0ff64635e2c"} Oct 31 00:28:36.274103 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:36.274106 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-76b073-68467cb4b7-pf7qq" Oct 31 00:28:36.274413 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:36.274141 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-76b073-68467cb4b7-pf7qq" event={"ID":"e63da6f1-8b8f-451d-84c1-9b39411a2a59","Type":"ContainerDied","Data":"26b10c16016249d7950f706bd67acb85253a2c34936f3b572f15f4c78721956f"} Oct 31 00:28:36.274413 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:36.274164 4927 scope.go:122] "RemoveContainer" containerID="929c28a1f7a209a0d96d173f92e660232d2acf9517c8bf272072c0ff64635e2c" Oct 31 00:28:36.283839 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:36.283820 4927 scope.go:122] "RemoveContainer" containerID="929c28a1f7a209a0d96d173f92e660232d2acf9517c8bf272072c0ff64635e2c" Oct 31 00:28:36.284130 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:28:36.284111 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"929c28a1f7a209a0d96d173f92e660232d2acf9517c8bf272072c0ff64635e2c\": container with ID starting with 929c28a1f7a209a0d96d173f92e660232d2acf9517c8bf272072c0ff64635e2c not found: ID does not exist" containerID="929c28a1f7a209a0d96d173f92e660232d2acf9517c8bf272072c0ff64635e2c" Oct 31 00:28:36.284215 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:36.284144 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"929c28a1f7a209a0d96d173f92e660232d2acf9517c8bf272072c0ff64635e2c"} err="failed to get container status \"929c28a1f7a209a0d96d173f92e660232d2acf9517c8bf272072c0ff64635e2c\": rpc error: code = NotFound desc = could not find container \"929c28a1f7a209a0d96d173f92e660232d2acf9517c8bf272072c0ff64635e2c\": container with ID starting with 929c28a1f7a209a0d96d173f92e660232d2acf9517c8bf272072c0ff64635e2c not found: ID does not exist" Oct 31 00:28:36.291477 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:36.291458 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e63da6f1-8b8f-451d-84c1-9b39411a2a59-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e63da6f1-8b8f-451d-84c1-9b39411a2a59-plugins-config-volume\") pod \"e63da6f1-8b8f-451d-84c1-9b39411a2a59\" (UID: \"e63da6f1-8b8f-451d-84c1-9b39411a2a59\") " Oct 31 00:28:36.291580 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:36.291515 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e63da6f1-8b8f-451d-84c1-9b39411a2a59-kube-api-access-245lg\" (UniqueName: \"kubernetes.io/projected/e63da6f1-8b8f-451d-84c1-9b39411a2a59-kube-api-access-245lg\") pod \"e63da6f1-8b8f-451d-84c1-9b39411a2a59\" (UID: \"e63da6f1-8b8f-451d-84c1-9b39411a2a59\") " Oct 31 00:28:36.291824 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:36.291801 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e63da6f1-8b8f-451d-84c1-9b39411a2a59-plugins-config-volume" pod "e63da6f1-8b8f-451d-84c1-9b39411a2a59" (UID: "e63da6f1-8b8f-451d-84c1-9b39411a2a59"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:28:36.293532 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:36.293511 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e63da6f1-8b8f-451d-84c1-9b39411a2a59-kube-api-access-245lg" pod "e63da6f1-8b8f-451d-84c1-9b39411a2a59" (UID: "e63da6f1-8b8f-451d-84c1-9b39411a2a59"). InnerVolumeSpecName "kube-api-access-245lg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:28:36.393113 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:36.393075 4927 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e63da6f1-8b8f-451d-84c1-9b39411a2a59-plugins-config-volume\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:36.393113 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:36.393106 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-245lg\" (UniqueName: \"kubernetes.io/projected/e63da6f1-8b8f-451d-84c1-9b39411a2a59-kube-api-access-245lg\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:36.585527 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:36.585485 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:28:36.700482 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:36.700400 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-76b073-68467cb4b7-pf7qq"] Oct 31 00:28:36.703976 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:36.703948 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-76b073-68467cb4b7-pf7qq"] Oct 31 00:28:36.740604 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:36.740567 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:28:37.089222 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:37.089180 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="505b964b-07ad-4e68-bbf9-4ce388e91aa0" path="/var/lib/kubelet/pods/505b964b-07ad-4e68-bbf9-4ce388e91aa0/volumes" Oct 31 00:28:37.089745 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:37.089722 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e63da6f1-8b8f-451d-84c1-9b39411a2a59" path="/var/lib/kubelet/pods/e63da6f1-8b8f-451d-84c1-9b39411a2a59/volumes" Oct 31 00:28:39.324997 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:39.324964 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:28:39.523803 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:39.523762 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:28:40.258283 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:40.258248 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:28:42.275358 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:42.275289 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:28:42.483928 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:42.483890 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:28:42.646973 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:42.646885 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:28:42.856691 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:42.856657 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:28:44.995350 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:44.995302 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:28:49.677564 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:28:49.677523 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sqv69" containerName="extractor" Oct 31 00:28:49.678166 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:49.678130 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:28:54.300202 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:54.300155 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:28:56.802200 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:56.802159 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-sz5n2"] Oct 31 00:28:56.803067 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:56.803038 4927 cpu_manager.go:414] "containerMap: removing container" podUID="e63da6f1-8b8f-451d-84c1-9b39411a2a59" containerName="payload-pre-processing" Oct 31 00:28:56.803067 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:56.803065 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e63da6f1-8b8f-451d-84c1-9b39411a2a59" containerName="payload-pre-processing" Oct 31 00:28:56.803266 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:56.803102 4927 cpu_manager.go:414] "containerMap: removing container" podUID="78b89047-2bad-472e-a80d-0ab8f1755877" containerName="istio-proxy" Oct 31 00:28:56.803266 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:56.803110 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78b89047-2bad-472e-a80d-0ab8f1755877" containerName="istio-proxy" Oct 31 00:28:56.803266 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:56.803145 4927 cpu_manager.go:414] "containerMap: removing container" podUID="4e4c7d3b-23b7-4729-b15e-851878093e45" containerName="revoke-keys" Oct 31 00:28:56.803266 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:56.803153 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e4c7d3b-23b7-4729-b15e-851878093e45" containerName="revoke-keys" Oct 31 00:28:56.803442 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:56.803272 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e4c7d3b-23b7-4729-b15e-851878093e45" containerName="revoke-keys" Oct 31 00:28:56.803442 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:56.803292 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="e63da6f1-8b8f-451d-84c1-9b39411a2a59" containerName="payload-pre-processing" Oct 31 00:28:56.803442 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:56.803305 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="78b89047-2bad-472e-a80d-0ab8f1755877" containerName="istio-proxy" Oct 31 00:28:56.803442 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:56.803316 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="78b89047-2bad-472e-a80d-0ab8f1755877" containerName="istio-proxy" Oct 31 00:28:56.807252 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:56.807232 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-sz5n2" Oct 31 00:28:56.859680 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:56.859644 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/720ee1c8-fc8a-432f-9257-0f365c666232-istiod-ca-cert\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-sz5n2\" (UID: \"720ee1c8-fc8a-432f-9257-0f365c666232\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-sz5n2" Oct 31 00:28:56.859890 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:56.859711 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d58lp\" (UniqueName: \"kubernetes.io/projected/720ee1c8-fc8a-432f-9257-0f365c666232-kube-api-access-d58lp\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-sz5n2\" (UID: \"720ee1c8-fc8a-432f-9257-0f365c666232\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-sz5n2" Oct 31 00:28:56.859890 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:56.859802 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/720ee1c8-fc8a-432f-9257-0f365c666232-credential-socket\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-sz5n2\" (UID: \"720ee1c8-fc8a-432f-9257-0f365c666232\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-sz5n2" Oct 31 00:28:56.859890 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:56.859842 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/720ee1c8-fc8a-432f-9257-0f365c666232-istio-envoy\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-sz5n2\" (UID: \"720ee1c8-fc8a-432f-9257-0f365c666232\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-sz5n2" Oct 31 00:28:56.859890 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:56.859876 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/720ee1c8-fc8a-432f-9257-0f365c666232-workload-socket\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-sz5n2\" (UID: \"720ee1c8-fc8a-432f-9257-0f365c666232\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-sz5n2" Oct 31 00:28:56.860084 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:56.859930 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/720ee1c8-fc8a-432f-9257-0f365c666232-istio-data\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-sz5n2\" (UID: \"720ee1c8-fc8a-432f-9257-0f365c666232\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-sz5n2" Oct 31 00:28:56.860084 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:56.859999 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/720ee1c8-fc8a-432f-9257-0f365c666232-istio-podinfo\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-sz5n2\" (UID: \"720ee1c8-fc8a-432f-9257-0f365c666232\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-sz5n2" Oct 31 00:28:56.860179 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:56.860153 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/720ee1c8-fc8a-432f-9257-0f365c666232-istio-token\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-sz5n2\" (UID: \"720ee1c8-fc8a-432f-9257-0f365c666232\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-sz5n2" Oct 31 00:28:56.860179 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:56.860172 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/720ee1c8-fc8a-432f-9257-0f365c666232-workload-certs\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-sz5n2\" (UID: \"720ee1c8-fc8a-432f-9257-0f365c666232\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-sz5n2" Oct 31 00:28:56.861069 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:56.861050 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-986d37-openshift-default-dockercfg-rm9vh\"" Oct 31 00:28:56.915885 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:56.915852 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-sz5n2"] Oct 31 00:28:56.960942 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:56.960910 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/720ee1c8-fc8a-432f-9257-0f365c666232-istio-token\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-sz5n2\" (UID: \"720ee1c8-fc8a-432f-9257-0f365c666232\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-sz5n2" Oct 31 00:28:56.961115 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:56.960946 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/720ee1c8-fc8a-432f-9257-0f365c666232-workload-certs\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-sz5n2\" (UID: \"720ee1c8-fc8a-432f-9257-0f365c666232\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-sz5n2" Oct 31 00:28:56.961115 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:56.960989 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/720ee1c8-fc8a-432f-9257-0f365c666232-istiod-ca-cert\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-sz5n2\" (UID: \"720ee1c8-fc8a-432f-9257-0f365c666232\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-sz5n2" Oct 31 00:28:56.961115 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:56.961019 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d58lp\" (UniqueName: \"kubernetes.io/projected/720ee1c8-fc8a-432f-9257-0f365c666232-kube-api-access-d58lp\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-sz5n2\" (UID: \"720ee1c8-fc8a-432f-9257-0f365c666232\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-sz5n2" Oct 31 00:28:56.961115 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:56.961040 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/720ee1c8-fc8a-432f-9257-0f365c666232-credential-socket\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-sz5n2\" (UID: \"720ee1c8-fc8a-432f-9257-0f365c666232\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-sz5n2" Oct 31 00:28:56.961115 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:56.961065 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/720ee1c8-fc8a-432f-9257-0f365c666232-istio-envoy\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-sz5n2\" (UID: \"720ee1c8-fc8a-432f-9257-0f365c666232\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-sz5n2" Oct 31 00:28:56.961115 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:56.961094 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/720ee1c8-fc8a-432f-9257-0f365c666232-workload-socket\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-sz5n2\" (UID: \"720ee1c8-fc8a-432f-9257-0f365c666232\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-sz5n2" Oct 31 00:28:56.961531 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:56.961313 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/720ee1c8-fc8a-432f-9257-0f365c666232-istio-data\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-sz5n2\" (UID: \"720ee1c8-fc8a-432f-9257-0f365c666232\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-sz5n2" Oct 31 00:28:56.961531 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:56.961364 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/720ee1c8-fc8a-432f-9257-0f365c666232-istio-podinfo\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-sz5n2\" (UID: \"720ee1c8-fc8a-432f-9257-0f365c666232\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-sz5n2" Oct 31 00:28:56.961650 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:56.961568 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/720ee1c8-fc8a-432f-9257-0f365c666232-credential-socket\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-sz5n2\" (UID: \"720ee1c8-fc8a-432f-9257-0f365c666232\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-sz5n2" Oct 31 00:28:56.961650 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:56.961610 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/720ee1c8-fc8a-432f-9257-0f365c666232-workload-socket\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-sz5n2\" (UID: \"720ee1c8-fc8a-432f-9257-0f365c666232\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-sz5n2" Oct 31 00:28:56.961879 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:56.961854 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/720ee1c8-fc8a-432f-9257-0f365c666232-istio-data\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-sz5n2\" (UID: \"720ee1c8-fc8a-432f-9257-0f365c666232\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-sz5n2" Oct 31 00:28:56.962008 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:56.961918 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/720ee1c8-fc8a-432f-9257-0f365c666232-workload-certs\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-sz5n2\" (UID: \"720ee1c8-fc8a-432f-9257-0f365c666232\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-sz5n2" Oct 31 00:28:56.962008 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:56.961937 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/720ee1c8-fc8a-432f-9257-0f365c666232-istiod-ca-cert\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-sz5n2\" (UID: \"720ee1c8-fc8a-432f-9257-0f365c666232\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-sz5n2" Oct 31 00:28:56.963546 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:56.963530 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/720ee1c8-fc8a-432f-9257-0f365c666232-istio-envoy\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-sz5n2\" (UID: \"720ee1c8-fc8a-432f-9257-0f365c666232\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-sz5n2" Oct 31 00:28:56.963970 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:56.963953 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/720ee1c8-fc8a-432f-9257-0f365c666232-istio-podinfo\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-sz5n2\" (UID: \"720ee1c8-fc8a-432f-9257-0f365c666232\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-sz5n2" Oct 31 00:28:57.023039 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:57.023004 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/720ee1c8-fc8a-432f-9257-0f365c666232-istio-token\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-sz5n2\" (UID: \"720ee1c8-fc8a-432f-9257-0f365c666232\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-sz5n2" Oct 31 00:28:57.023174 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:57.023007 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d58lp\" (UniqueName: \"kubernetes.io/projected/720ee1c8-fc8a-432f-9257-0f365c666232-kube-api-access-d58lp\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-sz5n2\" (UID: \"720ee1c8-fc8a-432f-9257-0f365c666232\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-sz5n2" Oct 31 00:28:57.121503 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:57.121430 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-sz5n2" Oct 31 00:28:57.197917 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:57.196853 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:28:57.390050 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:57.389968 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:28:57.563471 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:57.563445 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-sz5n2"] Oct 31 00:28:57.564857 ip-10-0-141-180 kubenswrapper[4927]: W1031 00:28:57.564830 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod720ee1c8_fc8a_432f_9257_0f365c666232.slice/crio-1d879394660ed06e5cf36030e17367b64a2c3d6ce120ffe3c62151ec9459905c WatchSource:0}: Error finding container 1d879394660ed06e5cf36030e17367b64a2c3d6ce120ffe3c62151ec9459905c: Status 404 returned error can't find the container with id 1d879394660ed06e5cf36030e17367b64a2c3d6ce120ffe3c62151ec9459905c Oct 31 00:28:57.567143 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:57.567110 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:28:57.567221 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:57.567175 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:28:57.567221 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:57.567212 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:28:58.336357 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:58.329614 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:28:58.460356 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:58.460101 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-sz5n2" event={"ID":"720ee1c8-fc8a-432f-9257-0f365c666232","Type":"ContainerStarted","Data":"d59119ca246ae3c4a44c7e7634c79643a3386c3e68212a87acb78dfefe3b72e7"} Oct 31 00:28:58.460356 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:58.460171 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-sz5n2" event={"ID":"720ee1c8-fc8a-432f-9257-0f365c666232","Type":"ContainerStarted","Data":"1d879394660ed06e5cf36030e17367b64a2c3d6ce120ffe3c62151ec9459905c"} Oct 31 00:28:59.128354 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:59.128064 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-sz5n2" Oct 31 00:28:59.128354 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:59.128240 4927 patch_prober.go:29] interesting pod/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-sz5n2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.2:15021/healthz/ready\": dial tcp 10.132.1.2:15021: connect: connection refused" start-of-body= Oct 31 00:28:59.128354 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:59.128286 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-sz5n2" podUID="720ee1c8-fc8a-432f-9257-0f365c666232" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.2:15021/healthz/ready\": dial tcp 10.132.1.2:15021: connect: connection refused" Oct 31 00:28:59.770580 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:59.770540 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:28:59.993762 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:28:59.993727 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:29:00.128409 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:00.127736 4927 patch_prober.go:29] interesting pod/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-sz5n2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.2:15021/healthz/ready\": dial tcp 10.132.1.2:15021: connect: connection refused" start-of-body= Oct 31 00:29:00.128409 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:00.127817 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-sz5n2" podUID="720ee1c8-fc8a-432f-9257-0f365c666232" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.2:15021/healthz/ready\": dial tcp 10.132.1.2:15021: connect: connection refused" Oct 31 00:29:00.148249 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:00.148206 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:29:00.242357 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:00.241593 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-sz5n2" podStartSLOduration=4.241573087 podStartE2EDuration="4.241573087s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:28:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:28:58.694237523 +0000 UTC m=+2851.730568346" watchObservedRunningTime="2026-10-31 00:29:00.241573087 +0000 UTC m=+2853.277903910" Oct 31 00:29:00.244434 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:00.244399 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-986d37-dhjvh"] Oct 31 00:29:00.249398 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:00.249372 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-986d37-dhjvh"] Oct 31 00:29:00.683764 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:00.683727 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:29:01.087202 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:01.087125 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6dc275c7-21fb-4b5c-ba2c-21b88ca78efc" path="/var/lib/kubelet/pods/6dc275c7-21fb-4b5c-ba2c-21b88ca78efc/volumes" Oct 31 00:29:01.129789 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:01.129751 4927 patch_prober.go:29] interesting pod/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-sz5n2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.2:15021/healthz/ready\": dial tcp 10.132.1.2:15021: connect: connection refused" start-of-body= Oct 31 00:29:01.130093 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:01.130061 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-sz5n2" podUID="720ee1c8-fc8a-432f-9257-0f365c666232" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.2:15021/healthz/ready\": dial tcp 10.132.1.2:15021: connect: connection refused" Oct 31 00:29:02.130825 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:02.130775 4927 patch_prober.go:29] interesting pod/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-sz5n2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.2:15021/healthz/ready\": dial tcp 10.132.1.2:15021: connect: connection refused" start-of-body= Oct 31 00:29:02.131416 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:02.130847 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-sz5n2" podUID="720ee1c8-fc8a-432f-9257-0f365c666232" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.2:15021/healthz/ready\": dial tcp 10.132.1.2:15021: connect: connection refused" Oct 31 00:29:03.129357 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:03.122757 4927 patch_prober.go:29] interesting pod/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-sz5n2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.2:15021/healthz/ready\": dial tcp 10.132.1.2:15021: connect: connection refused" start-of-body= Oct 31 00:29:03.129357 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:03.122835 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-sz5n2" podUID="720ee1c8-fc8a-432f-9257-0f365c666232" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.2:15021/healthz/ready\": dial tcp 10.132.1.2:15021: connect: connection refused" Oct 31 00:29:04.133199 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:04.124056 4927 patch_prober.go:29] interesting pod/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-sz5n2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.2:15021/healthz/ready\": dial tcp 10.132.1.2:15021: connect: connection refused" start-of-body= Oct 31 00:29:04.133199 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:04.124146 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-sz5n2" podUID="720ee1c8-fc8a-432f-9257-0f365c666232" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.2:15021/healthz/ready\": dial tcp 10.132.1.2:15021: connect: connection refused" Oct 31 00:29:04.206481 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:04.206446 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:29:05.722052 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:05.722014 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:29:05.926542 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:05.926500 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:29:06.112052 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:06.111955 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:29:06.125085 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:06.125052 4927 patch_prober.go:29] interesting pod/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-sz5n2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.2:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:29:06.125247 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:06.125118 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-sz5n2" podUID="720ee1c8-fc8a-432f-9257-0f365c666232" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.2:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:29:06.199070 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:06.199037 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-49837c-d4w6f"] Oct 31 00:29:06.244165 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:06.244123 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-49837c-d4w6f"] Oct 31 00:29:06.292838 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:06.292791 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-sz5n2" Oct 31 00:29:06.293739 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:06.293712 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-sz5n2" Oct 31 00:29:06.294453 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:06.294429 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-sz5n2" Oct 31 00:29:06.895921 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:06.895874 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:29:07.087133 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:07.087098 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e4c7d3b-23b7-4729-b15e-851878093e45" path="/var/lib/kubelet/pods/4e4c7d3b-23b7-4729-b15e-851878093e45/volumes" Oct 31 00:29:07.292614 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:07.292577 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:29:07.490760 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:07.490718 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:29:07.648173 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:07.648096 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-sz5n2"] Oct 31 00:29:07.651416 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:07.651384 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:29:08.547371 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:08.547285 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-sz5n2" podUID="720ee1c8-fc8a-432f-9257-0f365c666232" containerName="istio-proxy" containerID="cri-o://d59119ca246ae3c4a44c7e7634c79643a3386c3e68212a87acb78dfefe3b72e7" gracePeriod=30 Oct 31 00:29:10.893864 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:10.893819 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:29:12.955691 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:12.955652 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:29:13.205936 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:13.205851 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:29:13.388484 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:13.388441 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:29:13.545503 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:13.545418 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:29:13.815783 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:13.815725 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-sz5n2" Oct 31 00:29:13.886225 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:13.886197 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/720ee1c8-fc8a-432f-9257-0f365c666232-istio-token\" (UniqueName: \"kubernetes.io/projected/720ee1c8-fc8a-432f-9257-0f365c666232-istio-token\") pod \"720ee1c8-fc8a-432f-9257-0f365c666232\" (UID: \"720ee1c8-fc8a-432f-9257-0f365c666232\") " Oct 31 00:29:13.886225 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:13.886230 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/720ee1c8-fc8a-432f-9257-0f365c666232-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/720ee1c8-fc8a-432f-9257-0f365c666232-istiod-ca-cert\") pod \"720ee1c8-fc8a-432f-9257-0f365c666232\" (UID: \"720ee1c8-fc8a-432f-9257-0f365c666232\") " Oct 31 00:29:13.886521 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:13.886256 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/720ee1c8-fc8a-432f-9257-0f365c666232-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/720ee1c8-fc8a-432f-9257-0f365c666232-istio-envoy\") pod \"720ee1c8-fc8a-432f-9257-0f365c666232\" (UID: \"720ee1c8-fc8a-432f-9257-0f365c666232\") " Oct 31 00:29:13.886521 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:13.886354 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/720ee1c8-fc8a-432f-9257-0f365c666232-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/720ee1c8-fc8a-432f-9257-0f365c666232-workload-certs\") pod \"720ee1c8-fc8a-432f-9257-0f365c666232\" (UID: \"720ee1c8-fc8a-432f-9257-0f365c666232\") " Oct 31 00:29:13.886521 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:13.886451 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/720ee1c8-fc8a-432f-9257-0f365c666232-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/720ee1c8-fc8a-432f-9257-0f365c666232-istio-podinfo\") pod \"720ee1c8-fc8a-432f-9257-0f365c666232\" (UID: \"720ee1c8-fc8a-432f-9257-0f365c666232\") " Oct 31 00:29:13.886521 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:13.886514 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/720ee1c8-fc8a-432f-9257-0f365c666232-kube-api-access-d58lp\" (UniqueName: \"kubernetes.io/projected/720ee1c8-fc8a-432f-9257-0f365c666232-kube-api-access-d58lp\") pod \"720ee1c8-fc8a-432f-9257-0f365c666232\" (UID: \"720ee1c8-fc8a-432f-9257-0f365c666232\") " Oct 31 00:29:13.886757 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:13.886568 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/720ee1c8-fc8a-432f-9257-0f365c666232-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/720ee1c8-fc8a-432f-9257-0f365c666232-istio-data\") pod \"720ee1c8-fc8a-432f-9257-0f365c666232\" (UID: \"720ee1c8-fc8a-432f-9257-0f365c666232\") " Oct 31 00:29:13.886757 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:13.886616 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/720ee1c8-fc8a-432f-9257-0f365c666232-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/720ee1c8-fc8a-432f-9257-0f365c666232-credential-socket\") pod \"720ee1c8-fc8a-432f-9257-0f365c666232\" (UID: \"720ee1c8-fc8a-432f-9257-0f365c666232\") " Oct 31 00:29:13.886757 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:13.886672 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/720ee1c8-fc8a-432f-9257-0f365c666232-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/720ee1c8-fc8a-432f-9257-0f365c666232-workload-socket\") pod \"720ee1c8-fc8a-432f-9257-0f365c666232\" (UID: \"720ee1c8-fc8a-432f-9257-0f365c666232\") " Oct 31 00:29:13.886757 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:13.886672 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/720ee1c8-fc8a-432f-9257-0f365c666232-workload-certs" pod "720ee1c8-fc8a-432f-9257-0f365c666232" (UID: "720ee1c8-fc8a-432f-9257-0f365c666232"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:29:13.886985 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:13.886778 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/720ee1c8-fc8a-432f-9257-0f365c666232-istiod-ca-cert" pod "720ee1c8-fc8a-432f-9257-0f365c666232" (UID: "720ee1c8-fc8a-432f-9257-0f365c666232"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:29:13.886985 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:13.886905 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/720ee1c8-fc8a-432f-9257-0f365c666232-istio-data" pod "720ee1c8-fc8a-432f-9257-0f365c666232" (UID: "720ee1c8-fc8a-432f-9257-0f365c666232"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:29:13.887102 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:13.887045 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/720ee1c8-fc8a-432f-9257-0f365c666232-credential-socket" pod "720ee1c8-fc8a-432f-9257-0f365c666232" (UID: "720ee1c8-fc8a-432f-9257-0f365c666232"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:29:13.887102 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:13.887059 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/720ee1c8-fc8a-432f-9257-0f365c666232-workload-socket" pod "720ee1c8-fc8a-432f-9257-0f365c666232" (UID: "720ee1c8-fc8a-432f-9257-0f365c666232"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:29:13.887232 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:13.887215 4927 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/720ee1c8-fc8a-432f-9257-0f365c666232-credential-socket\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:13.887302 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:13.887240 4927 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/720ee1c8-fc8a-432f-9257-0f365c666232-workload-socket\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:13.887302 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:13.887256 4927 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/720ee1c8-fc8a-432f-9257-0f365c666232-istiod-ca-cert\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:13.887302 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:13.887270 4927 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/720ee1c8-fc8a-432f-9257-0f365c666232-workload-certs\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:13.887302 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:13.887284 4927 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/720ee1c8-fc8a-432f-9257-0f365c666232-istio-data\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:13.888599 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:13.888574 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/720ee1c8-fc8a-432f-9257-0f365c666232-istio-envoy" pod "720ee1c8-fc8a-432f-9257-0f365c666232" (UID: "720ee1c8-fc8a-432f-9257-0f365c666232"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:29:13.888784 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:13.888763 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/720ee1c8-fc8a-432f-9257-0f365c666232-istio-podinfo" pod "720ee1c8-fc8a-432f-9257-0f365c666232" (UID: "720ee1c8-fc8a-432f-9257-0f365c666232"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:29:13.889041 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:13.889016 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/720ee1c8-fc8a-432f-9257-0f365c666232-istio-token" pod "720ee1c8-fc8a-432f-9257-0f365c666232" (UID: "720ee1c8-fc8a-432f-9257-0f365c666232"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:29:13.889153 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:13.889044 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/720ee1c8-fc8a-432f-9257-0f365c666232-kube-api-access-d58lp" pod "720ee1c8-fc8a-432f-9257-0f365c666232" (UID: "720ee1c8-fc8a-432f-9257-0f365c666232"). InnerVolumeSpecName "kube-api-access-d58lp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:29:13.988671 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:13.988634 4927 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/720ee1c8-fc8a-432f-9257-0f365c666232-istio-token\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:13.988671 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:13.988667 4927 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/720ee1c8-fc8a-432f-9257-0f365c666232-istio-envoy\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:13.988671 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:13.988677 4927 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/720ee1c8-fc8a-432f-9257-0f365c666232-istio-podinfo\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:13.989110 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:13.988687 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d58lp\" (UniqueName: \"kubernetes.io/projected/720ee1c8-fc8a-432f-9257-0f365c666232-kube-api-access-d58lp\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:14.159026 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:14.158988 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-d5360e-vcgvf"] Oct 31 00:29:14.159537 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:14.159520 4927 cpu_manager.go:414] "containerMap: removing container" podUID="78b89047-2bad-472e-a80d-0ab8f1755877" containerName="istio-proxy" Oct 31 00:29:14.159537 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:14.159536 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78b89047-2bad-472e-a80d-0ab8f1755877" containerName="istio-proxy" Oct 31 00:29:14.159726 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:14.159568 4927 cpu_manager.go:414] "containerMap: removing container" podUID="720ee1c8-fc8a-432f-9257-0f365c666232" containerName="istio-proxy" Oct 31 00:29:14.159726 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:14.159573 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="720ee1c8-fc8a-432f-9257-0f365c666232" containerName="istio-proxy" Oct 31 00:29:14.159726 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:14.159659 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="720ee1c8-fc8a-432f-9257-0f365c666232" containerName="istio-proxy" Oct 31 00:29:14.163284 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:14.163264 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-d5360e-vcgvf" Oct 31 00:29:14.217621 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:14.217586 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:29:14.217801 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:14.217663 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4fdzs\"" Oct 31 00:29:14.222621 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:14.221667 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-d5360e-vcgvf"] Oct 31 00:29:14.292443 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:14.292401 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/567477d1-1811-4e35-9f57-9f8eea08988a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-d5360e-vcgvf\" (UID: \"567477d1-1811-4e35-9f57-9f8eea08988a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-d5360e-vcgvf" Oct 31 00:29:14.292609 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:14.292450 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/567477d1-1811-4e35-9f57-9f8eea08988a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-d5360e-vcgvf\" (UID: \"567477d1-1811-4e35-9f57-9f8eea08988a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-d5360e-vcgvf" Oct 31 00:29:14.292609 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:14.292479 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6nb9\" (UniqueName: \"kubernetes.io/projected/567477d1-1811-4e35-9f57-9f8eea08988a-kube-api-access-h6nb9\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-d5360e-vcgvf\" (UID: \"567477d1-1811-4e35-9f57-9f8eea08988a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-d5360e-vcgvf" Oct 31 00:29:14.393692 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:14.393660 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/567477d1-1811-4e35-9f57-9f8eea08988a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-d5360e-vcgvf\" (UID: \"567477d1-1811-4e35-9f57-9f8eea08988a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-d5360e-vcgvf" Oct 31 00:29:14.393692 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:14.393696 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/567477d1-1811-4e35-9f57-9f8eea08988a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-d5360e-vcgvf\" (UID: \"567477d1-1811-4e35-9f57-9f8eea08988a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-d5360e-vcgvf" Oct 31 00:29:14.393949 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:14.393716 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6nb9\" (UniqueName: \"kubernetes.io/projected/567477d1-1811-4e35-9f57-9f8eea08988a-kube-api-access-h6nb9\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-d5360e-vcgvf\" (UID: \"567477d1-1811-4e35-9f57-9f8eea08988a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-d5360e-vcgvf" Oct 31 00:29:14.394277 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:14.394256 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/567477d1-1811-4e35-9f57-9f8eea08988a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-d5360e-vcgvf\" (UID: \"567477d1-1811-4e35-9f57-9f8eea08988a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-d5360e-vcgvf" Oct 31 00:29:14.396055 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:14.396030 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/567477d1-1811-4e35-9f57-9f8eea08988a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-d5360e-vcgvf\" (UID: \"567477d1-1811-4e35-9f57-9f8eea08988a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-d5360e-vcgvf" Oct 31 00:29:14.467597 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:14.467525 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6nb9\" (UniqueName: \"kubernetes.io/projected/567477d1-1811-4e35-9f57-9f8eea08988a-kube-api-access-h6nb9\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-d5360e-vcgvf\" (UID: \"567477d1-1811-4e35-9f57-9f8eea08988a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-d5360e-vcgvf" Oct 31 00:29:14.477351 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:14.477316 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-d5360e-vcgvf" Oct 31 00:29:14.595422 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:14.595389 4927 generic.go:356] "Generic (PLEG): container finished" podID="720ee1c8-fc8a-432f-9257-0f365c666232" containerID="d59119ca246ae3c4a44c7e7634c79643a3386c3e68212a87acb78dfefe3b72e7" exitCode=0 Oct 31 00:29:14.595598 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:14.595512 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-sz5n2" Oct 31 00:29:14.595598 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:14.595535 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-sz5n2" event={"ID":"720ee1c8-fc8a-432f-9257-0f365c666232","Type":"ContainerDied","Data":"d59119ca246ae3c4a44c7e7634c79643a3386c3e68212a87acb78dfefe3b72e7"} Oct 31 00:29:14.595722 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:14.595606 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-sz5n2" event={"ID":"720ee1c8-fc8a-432f-9257-0f365c666232","Type":"ContainerDied","Data":"1d879394660ed06e5cf36030e17367b64a2c3d6ce120ffe3c62151ec9459905c"} Oct 31 00:29:14.595722 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:14.595641 4927 scope.go:122] "RemoveContainer" containerID="d59119ca246ae3c4a44c7e7634c79643a3386c3e68212a87acb78dfefe3b72e7" Oct 31 00:29:14.612669 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:14.612642 4927 scope.go:122] "RemoveContainer" containerID="d59119ca246ae3c4a44c7e7634c79643a3386c3e68212a87acb78dfefe3b72e7" Oct 31 00:29:14.613033 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:29:14.613009 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d59119ca246ae3c4a44c7e7634c79643a3386c3e68212a87acb78dfefe3b72e7\": container with ID starting with d59119ca246ae3c4a44c7e7634c79643a3386c3e68212a87acb78dfefe3b72e7 not found: ID does not exist" containerID="d59119ca246ae3c4a44c7e7634c79643a3386c3e68212a87acb78dfefe3b72e7" Oct 31 00:29:14.613130 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:14.613043 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d59119ca246ae3c4a44c7e7634c79643a3386c3e68212a87acb78dfefe3b72e7"} err="failed to get container status \"d59119ca246ae3c4a44c7e7634c79643a3386c3e68212a87acb78dfefe3b72e7\": rpc error: code = NotFound desc = could not find container \"d59119ca246ae3c4a44c7e7634c79643a3386c3e68212a87acb78dfefe3b72e7\": container with ID starting with d59119ca246ae3c4a44c7e7634c79643a3386c3e68212a87acb78dfefe3b72e7 not found: ID does not exist" Oct 31 00:29:14.765744 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:14.765666 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-sz5n2"] Oct 31 00:29:14.775041 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:14.775004 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-sz5n2"] Oct 31 00:29:14.879938 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:14.879904 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-d5360e-vcgvf"] Oct 31 00:29:14.881188 ip-10-0-141-180 kubenswrapper[4927]: W1031 00:29:14.881159 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod567477d1_1811_4e35_9f57_9f8eea08988a.slice/crio-dcc16f14800e942c00b2104ff387c10d26fd3f86fbc4c653ae2018a693c691e3 WatchSource:0}: Error finding container dcc16f14800e942c00b2104ff387c10d26fd3f86fbc4c653ae2018a693c691e3: Status 404 returned error can't find the container with id dcc16f14800e942c00b2104ff387c10d26fd3f86fbc4c653ae2018a693c691e3 Oct 31 00:29:15.086691 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:15.086620 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="720ee1c8-fc8a-432f-9257-0f365c666232" path="/var/lib/kubelet/pods/720ee1c8-fc8a-432f-9257-0f365c666232/volumes" Oct 31 00:29:15.604453 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:15.604410 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-d5360e-vcgvf" event={"ID":"567477d1-1811-4e35-9f57-9f8eea08988a","Type":"ContainerStarted","Data":"5a5db4e56ed1644f5ed54fb05f56577e974d70330bf9988c5c7d0cc94b2a8f4b"} Oct 31 00:29:15.604682 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:15.604477 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-d5360e-vcgvf" event={"ID":"567477d1-1811-4e35-9f57-9f8eea08988a","Type":"ContainerStarted","Data":"dcc16f14800e942c00b2104ff387c10d26fd3f86fbc4c653ae2018a693c691e3"} Oct 31 00:29:15.630456 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:15.630420 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:29:15.727090 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:15.727026 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-d5360e-vcgvf" podStartSLOduration=1.7270057319999998 podStartE2EDuration="1.727005732s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:29:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:29:15.722414271 +0000 UTC m=+2868.758745111" watchObservedRunningTime="2026-10-31 00:29:15.727005732 +0000 UTC m=+2868.763336556" Oct 31 00:29:16.617156 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:16.617112 4927 generic.go:356] "Generic (PLEG): container finished" podID="567477d1-1811-4e35-9f57-9f8eea08988a" containerID="5a5db4e56ed1644f5ed54fb05f56577e974d70330bf9988c5c7d0cc94b2a8f4b" exitCode=0 Oct 31 00:29:16.617156 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:16.617157 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-d5360e-vcgvf" event={"ID":"567477d1-1811-4e35-9f57-9f8eea08988a","Type":"ContainerDied","Data":"5a5db4e56ed1644f5ed54fb05f56577e974d70330bf9988c5c7d0cc94b2a8f4b"} Oct 31 00:29:17.774975 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:17.774944 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-d5360e-vcgvf" Oct 31 00:29:17.832912 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:17.832880 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/567477d1-1811-4e35-9f57-9f8eea08988a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/567477d1-1811-4e35-9f57-9f8eea08988a-maas-api-xks-ca\") pod \"567477d1-1811-4e35-9f57-9f8eea08988a\" (UID: \"567477d1-1811-4e35-9f57-9f8eea08988a\") " Oct 31 00:29:17.833073 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:17.833000 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/567477d1-1811-4e35-9f57-9f8eea08988a-kube-api-access-h6nb9\" (UniqueName: \"kubernetes.io/projected/567477d1-1811-4e35-9f57-9f8eea08988a-kube-api-access-h6nb9\") pod \"567477d1-1811-4e35-9f57-9f8eea08988a\" (UID: \"567477d1-1811-4e35-9f57-9f8eea08988a\") " Oct 31 00:29:17.833121 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:17.833098 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/567477d1-1811-4e35-9f57-9f8eea08988a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/567477d1-1811-4e35-9f57-9f8eea08988a-maas-api-openshift-ca\") pod \"567477d1-1811-4e35-9f57-9f8eea08988a\" (UID: \"567477d1-1811-4e35-9f57-9f8eea08988a\") " Oct 31 00:29:17.833510 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:17.833481 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/567477d1-1811-4e35-9f57-9f8eea08988a-maas-api-openshift-ca" pod "567477d1-1811-4e35-9f57-9f8eea08988a" (UID: "567477d1-1811-4e35-9f57-9f8eea08988a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:29:17.835019 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:17.834990 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/567477d1-1811-4e35-9f57-9f8eea08988a-maas-api-xks-ca" pod "567477d1-1811-4e35-9f57-9f8eea08988a" (UID: "567477d1-1811-4e35-9f57-9f8eea08988a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:29:17.835140 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:17.835047 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/567477d1-1811-4e35-9f57-9f8eea08988a-kube-api-access-h6nb9" pod "567477d1-1811-4e35-9f57-9f8eea08988a" (UID: "567477d1-1811-4e35-9f57-9f8eea08988a"). InnerVolumeSpecName "kube-api-access-h6nb9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:29:17.935057 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:17.935028 4927 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/567477d1-1811-4e35-9f57-9f8eea08988a-maas-api-xks-ca\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:17.935057 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:17.935057 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h6nb9\" (UniqueName: \"kubernetes.io/projected/567477d1-1811-4e35-9f57-9f8eea08988a-kube-api-access-h6nb9\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:17.935248 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:17.935068 4927 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/567477d1-1811-4e35-9f57-9f8eea08988a-maas-api-openshift-ca\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:18.634498 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:18.634466 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-d5360e-vcgvf" Oct 31 00:29:18.634682 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:18.634498 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-d5360e-vcgvf" event={"ID":"567477d1-1811-4e35-9f57-9f8eea08988a","Type":"ContainerDied","Data":"dcc16f14800e942c00b2104ff387c10d26fd3f86fbc4c653ae2018a693c691e3"} Oct 31 00:29:18.634682 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:18.634548 4927 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dcc16f14800e942c00b2104ff387c10d26fd3f86fbc4c653ae2018a693c691e3" Oct 31 00:29:19.678560 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:29:19.678527 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sqv69" containerName="extractor" Oct 31 00:29:20.589024 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:20.588982 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:29:25.188399 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:25.188360 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:29:25.562735 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:25.562635 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-d5360e-854fc9bc5d-mc76v"] Oct 31 00:29:25.671188 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:25.671152 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-d5360e-b6fb568dc-htsps"] Oct 31 00:29:25.924956 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:25.924916 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:29:26.102566 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:26.102533 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:29:29.877111 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:29.877072 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:29:30.563445 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:30.563402 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-d5360e-854fc9bc5d-mc76v" podUID="e27553d8-0637-46a1-bd84-186e69ba3f15" containerName="payload-processing" containerID="cri-o://91f4ab37f571dda42eb43047b701d99c5460a0a6931d51411f117c8fd6314e5b" gracePeriod=25 Oct 31 00:29:30.671524 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:30.671448 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-d5360e-b6fb568dc-htsps" podUID="64551325-0270-49ec-bbc7-748cd5c3611e" containerName="payload-pre-processing" containerID="cri-o://ba6a08f7b995d7314ecd91ce1bd1015e3420e269e81e81380e4573b45f77b353" gracePeriod=25 Oct 31 00:29:30.721925 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:30.721892 4927 generic.go:356] "Generic (PLEG): container finished" podID="e27553d8-0637-46a1-bd84-186e69ba3f15" containerID="91f4ab37f571dda42eb43047b701d99c5460a0a6931d51411f117c8fd6314e5b" exitCode=0 Oct 31 00:29:30.722079 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:30.721967 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-d5360e-854fc9bc5d-mc76v" event={"ID":"e27553d8-0637-46a1-bd84-186e69ba3f15","Type":"ContainerDied","Data":"91f4ab37f571dda42eb43047b701d99c5460a0a6931d51411f117c8fd6314e5b"} Oct 31 00:29:30.857613 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:30.855938 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-d5360e-854fc9bc5d-mc76v" Oct 31 00:29:30.958651 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:30.958615 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-d5360e-b6fb568dc-htsps" Oct 31 00:29:30.980391 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:30.980364 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e27553d8-0637-46a1-bd84-186e69ba3f15-kube-api-access-lhrht\" (UniqueName: \"kubernetes.io/projected/e27553d8-0637-46a1-bd84-186e69ba3f15-kube-api-access-lhrht\") pod \"e27553d8-0637-46a1-bd84-186e69ba3f15\" (UID: \"e27553d8-0637-46a1-bd84-186e69ba3f15\") " Oct 31 00:29:30.980530 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:30.980511 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e27553d8-0637-46a1-bd84-186e69ba3f15-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e27553d8-0637-46a1-bd84-186e69ba3f15-plugins-config-volume\") pod \"e27553d8-0637-46a1-bd84-186e69ba3f15\" (UID: \"e27553d8-0637-46a1-bd84-186e69ba3f15\") " Oct 31 00:29:30.980867 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:30.980845 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e27553d8-0637-46a1-bd84-186e69ba3f15-plugins-config-volume" pod "e27553d8-0637-46a1-bd84-186e69ba3f15" (UID: "e27553d8-0637-46a1-bd84-186e69ba3f15"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:29:30.980940 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:30.980928 4927 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e27553d8-0637-46a1-bd84-186e69ba3f15-plugins-config-volume\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:30.982501 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:30.982478 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e27553d8-0637-46a1-bd84-186e69ba3f15-kube-api-access-lhrht" pod "e27553d8-0637-46a1-bd84-186e69ba3f15" (UID: "e27553d8-0637-46a1-bd84-186e69ba3f15"). InnerVolumeSpecName "kube-api-access-lhrht". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:29:31.081452 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:31.081355 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64551325-0270-49ec-bbc7-748cd5c3611e-kube-api-access-dhvfx\" (UniqueName: \"kubernetes.io/projected/64551325-0270-49ec-bbc7-748cd5c3611e-kube-api-access-dhvfx\") pod \"64551325-0270-49ec-bbc7-748cd5c3611e\" (UID: \"64551325-0270-49ec-bbc7-748cd5c3611e\") " Oct 31 00:29:31.081452 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:31.081427 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/64551325-0270-49ec-bbc7-748cd5c3611e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64551325-0270-49ec-bbc7-748cd5c3611e-plugins-config-volume\") pod \"64551325-0270-49ec-bbc7-748cd5c3611e\" (UID: \"64551325-0270-49ec-bbc7-748cd5c3611e\") " Oct 31 00:29:31.081816 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:31.081794 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64551325-0270-49ec-bbc7-748cd5c3611e-plugins-config-volume" pod "64551325-0270-49ec-bbc7-748cd5c3611e" (UID: "64551325-0270-49ec-bbc7-748cd5c3611e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:29:31.081898 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:31.081881 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lhrht\" (UniqueName: \"kubernetes.io/projected/e27553d8-0637-46a1-bd84-186e69ba3f15-kube-api-access-lhrht\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:31.083731 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:31.083708 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64551325-0270-49ec-bbc7-748cd5c3611e-kube-api-access-dhvfx" pod "64551325-0270-49ec-bbc7-748cd5c3611e" (UID: "64551325-0270-49ec-bbc7-748cd5c3611e"). InnerVolumeSpecName "kube-api-access-dhvfx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:29:31.182661 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:31.182629 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dhvfx\" (UniqueName: \"kubernetes.io/projected/64551325-0270-49ec-bbc7-748cd5c3611e-kube-api-access-dhvfx\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:31.182661 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:31.182657 4927 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64551325-0270-49ec-bbc7-748cd5c3611e-plugins-config-volume\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:31.731178 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:31.731139 4927 generic.go:356] "Generic (PLEG): container finished" podID="64551325-0270-49ec-bbc7-748cd5c3611e" containerID="ba6a08f7b995d7314ecd91ce1bd1015e3420e269e81e81380e4573b45f77b353" exitCode=0 Oct 31 00:29:31.731409 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:31.731204 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-d5360e-b6fb568dc-htsps" Oct 31 00:29:31.731409 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:31.731228 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-d5360e-b6fb568dc-htsps" event={"ID":"64551325-0270-49ec-bbc7-748cd5c3611e","Type":"ContainerDied","Data":"ba6a08f7b995d7314ecd91ce1bd1015e3420e269e81e81380e4573b45f77b353"} Oct 31 00:29:31.731409 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:31.731325 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-d5360e-b6fb568dc-htsps" event={"ID":"64551325-0270-49ec-bbc7-748cd5c3611e","Type":"ContainerDied","Data":"1afd116633824e8e387efbb61d4466aff7d4aa353069c8c3f36e62e4df3e7775"} Oct 31 00:29:31.731409 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:31.731374 4927 scope.go:122] "RemoveContainer" containerID="ba6a08f7b995d7314ecd91ce1bd1015e3420e269e81e81380e4573b45f77b353" Oct 31 00:29:31.732973 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:31.732953 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-d5360e-854fc9bc5d-mc76v" event={"ID":"e27553d8-0637-46a1-bd84-186e69ba3f15","Type":"ContainerDied","Data":"ab8aeb82b05e46a9685b9c9b057826962a1e280dee9e6334bf4c97fe3e13fd9a"} Oct 31 00:29:31.733093 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:31.732994 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-d5360e-854fc9bc5d-mc76v" Oct 31 00:29:31.743786 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:31.743769 4927 scope.go:122] "RemoveContainer" containerID="ba6a08f7b995d7314ecd91ce1bd1015e3420e269e81e81380e4573b45f77b353" Oct 31 00:29:31.744108 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:29:31.744077 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba6a08f7b995d7314ecd91ce1bd1015e3420e269e81e81380e4573b45f77b353\": container with ID starting with ba6a08f7b995d7314ecd91ce1bd1015e3420e269e81e81380e4573b45f77b353 not found: ID does not exist" containerID="ba6a08f7b995d7314ecd91ce1bd1015e3420e269e81e81380e4573b45f77b353" Oct 31 00:29:31.744178 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:31.744113 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba6a08f7b995d7314ecd91ce1bd1015e3420e269e81e81380e4573b45f77b353"} err="failed to get container status \"ba6a08f7b995d7314ecd91ce1bd1015e3420e269e81e81380e4573b45f77b353\": rpc error: code = NotFound desc = could not find container \"ba6a08f7b995d7314ecd91ce1bd1015e3420e269e81e81380e4573b45f77b353\": container with ID starting with ba6a08f7b995d7314ecd91ce1bd1015e3420e269e81e81380e4573b45f77b353 not found: ID does not exist" Oct 31 00:29:31.744178 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:31.744139 4927 scope.go:122] "RemoveContainer" containerID="91f4ab37f571dda42eb43047b701d99c5460a0a6931d51411f117c8fd6314e5b" Oct 31 00:29:31.907191 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:31.907119 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-d5360e-854fc9bc5d-mc76v"] Oct 31 00:29:31.914785 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:31.914743 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-d5360e-854fc9bc5d-mc76v"] Oct 31 00:29:32.026851 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:32.026813 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-d5360e-b6fb568dc-htsps"] Oct 31 00:29:32.031091 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:32.031063 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-d5360e-b6fb568dc-htsps"] Oct 31 00:29:33.086266 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:33.086226 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="64551325-0270-49ec-bbc7-748cd5c3611e" path="/var/lib/kubelet/pods/64551325-0270-49ec-bbc7-748cd5c3611e/volumes" Oct 31 00:29:33.086830 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:33.086809 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e27553d8-0637-46a1-bd84-186e69ba3f15" path="/var/lib/kubelet/pods/e27553d8-0637-46a1-bd84-186e69ba3f15/volumes" Oct 31 00:29:34.497847 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:34.497809 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:29:34.655029 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:34.654989 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:29:38.393743 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:38.393703 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-d5360e-788c77fc85-gkswq"] Oct 31 00:29:38.394184 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:38.393913 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-d5360e-788c77fc85-gkswq" podUID="e3fe218a-9831-44c3-a5eb-09e14994459f" containerName="maas-api" containerID="cri-o://fdc6c293ed30d34db466ce67c69be2fb33cd7c9ad334aac595af65735a2fc755" gracePeriod=30 Oct 31 00:29:38.500589 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:38.500552 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:29:38.673931 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:38.673905 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-d5360e-788c77fc85-gkswq" Oct 31 00:29:38.785032 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:38.784995 4927 generic.go:356] "Generic (PLEG): container finished" podID="e3fe218a-9831-44c3-a5eb-09e14994459f" containerID="fdc6c293ed30d34db466ce67c69be2fb33cd7c9ad334aac595af65735a2fc755" exitCode=0 Oct 31 00:29:38.785230 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:38.785043 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-d5360e-788c77fc85-gkswq" Oct 31 00:29:38.785230 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:38.785064 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-d5360e-788c77fc85-gkswq" event={"ID":"e3fe218a-9831-44c3-a5eb-09e14994459f","Type":"ContainerDied","Data":"fdc6c293ed30d34db466ce67c69be2fb33cd7c9ad334aac595af65735a2fc755"} Oct 31 00:29:38.785230 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:38.785111 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-d5360e-788c77fc85-gkswq" event={"ID":"e3fe218a-9831-44c3-a5eb-09e14994459f","Type":"ContainerDied","Data":"502882133dbbfd58969cd003340eaf3262df532050fb38ef651e3629d5336a8a"} Oct 31 00:29:38.785230 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:38.785133 4927 scope.go:122] "RemoveContainer" containerID="fdc6c293ed30d34db466ce67c69be2fb33cd7c9ad334aac595af65735a2fc755" Oct 31 00:29:38.795563 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:38.795539 4927 scope.go:122] "RemoveContainer" containerID="fdc6c293ed30d34db466ce67c69be2fb33cd7c9ad334aac595af65735a2fc755" Oct 31 00:29:38.795939 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:29:38.795914 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fdc6c293ed30d34db466ce67c69be2fb33cd7c9ad334aac595af65735a2fc755\": container with ID starting with fdc6c293ed30d34db466ce67c69be2fb33cd7c9ad334aac595af65735a2fc755 not found: ID does not exist" containerID="fdc6c293ed30d34db466ce67c69be2fb33cd7c9ad334aac595af65735a2fc755" Oct 31 00:29:38.796122 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:38.796086 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fdc6c293ed30d34db466ce67c69be2fb33cd7c9ad334aac595af65735a2fc755"} err="failed to get container status \"fdc6c293ed30d34db466ce67c69be2fb33cd7c9ad334aac595af65735a2fc755\": rpc error: code = NotFound desc = could not find container \"fdc6c293ed30d34db466ce67c69be2fb33cd7c9ad334aac595af65735a2fc755\": container with ID starting with fdc6c293ed30d34db466ce67c69be2fb33cd7c9ad334aac595af65735a2fc755 not found: ID does not exist" Oct 31 00:29:38.871304 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:38.871267 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e3fe218a-9831-44c3-a5eb-09e14994459f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e3fe218a-9831-44c3-a5eb-09e14994459f-metrics-certs\") pod \"e3fe218a-9831-44c3-a5eb-09e14994459f\" (UID: \"e3fe218a-9831-44c3-a5eb-09e14994459f\") " Oct 31 00:29:38.871537 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:38.871385 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e3fe218a-9831-44c3-a5eb-09e14994459f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e3fe218a-9831-44c3-a5eb-09e14994459f-openshift-service-ca\") pod \"e3fe218a-9831-44c3-a5eb-09e14994459f\" (UID: \"e3fe218a-9831-44c3-a5eb-09e14994459f\") " Oct 31 00:29:38.871537 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:38.871423 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3fe218a-9831-44c3-a5eb-09e14994459f-kube-api-access-p5cps\" (UniqueName: \"kubernetes.io/projected/e3fe218a-9831-44c3-a5eb-09e14994459f-kube-api-access-p5cps\") pod \"e3fe218a-9831-44c3-a5eb-09e14994459f\" (UID: \"e3fe218a-9831-44c3-a5eb-09e14994459f\") " Oct 31 00:29:38.871537 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:38.871512 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e3fe218a-9831-44c3-a5eb-09e14994459f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e3fe218a-9831-44c3-a5eb-09e14994459f-maas-api-tls\") pod \"e3fe218a-9831-44c3-a5eb-09e14994459f\" (UID: \"e3fe218a-9831-44c3-a5eb-09e14994459f\") " Oct 31 00:29:38.871790 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:38.871761 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3fe218a-9831-44c3-a5eb-09e14994459f-openshift-service-ca" pod "e3fe218a-9831-44c3-a5eb-09e14994459f" (UID: "e3fe218a-9831-44c3-a5eb-09e14994459f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:29:38.871932 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:38.871913 4927 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e3fe218a-9831-44c3-a5eb-09e14994459f-openshift-service-ca\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:38.873555 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:38.873533 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e3fe218a-9831-44c3-a5eb-09e14994459f-maas-api-tls" pod "e3fe218a-9831-44c3-a5eb-09e14994459f" (UID: "e3fe218a-9831-44c3-a5eb-09e14994459f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:29:38.873659 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:38.873565 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e3fe218a-9831-44c3-a5eb-09e14994459f-metrics-certs" pod "e3fe218a-9831-44c3-a5eb-09e14994459f" (UID: "e3fe218a-9831-44c3-a5eb-09e14994459f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:29:38.873659 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:38.873648 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3fe218a-9831-44c3-a5eb-09e14994459f-kube-api-access-p5cps" pod "e3fe218a-9831-44c3-a5eb-09e14994459f" (UID: "e3fe218a-9831-44c3-a5eb-09e14994459f"). InnerVolumeSpecName "kube-api-access-p5cps". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:29:38.972787 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:38.972753 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p5cps\" (UniqueName: \"kubernetes.io/projected/e3fe218a-9831-44c3-a5eb-09e14994459f-kube-api-access-p5cps\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:38.972787 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:38.972784 4927 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e3fe218a-9831-44c3-a5eb-09e14994459f-maas-api-tls\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:38.972787 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:38.972793 4927 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e3fe218a-9831-44c3-a5eb-09e14994459f-metrics-certs\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:39.212173 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:39.212138 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-d5360e-788c77fc85-gkswq"] Oct 31 00:29:39.289421 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:39.289303 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-d5360e-788c77fc85-gkswq"] Oct 31 00:29:39.982172 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:39.982131 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:29:40.214993 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:40.214957 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:29:40.368548 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:40.368466 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:29:40.461530 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:40.461493 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-d5360e-vcgvf"] Oct 31 00:29:40.512683 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:40.512648 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-d5360e-vcgvf"] Oct 31 00:29:40.989833 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:40.989802 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:29:41.086194 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:41.086157 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="567477d1-1811-4e35-9f57-9f8eea08988a" path="/var/lib/kubelet/pods/567477d1-1811-4e35-9f57-9f8eea08988a/volumes" Oct 31 00:29:41.086762 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:41.086740 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e3fe218a-9831-44c3-a5eb-09e14994459f" path="/var/lib/kubelet/pods/e3fe218a-9831-44c3-a5eb-09e14994459f/volumes" Oct 31 00:29:45.081628 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:45.081587 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:29:47.247323 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:47.247288 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:29:47.429641 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:47.429607 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:29:47.611907 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:47.611822 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-d5360e-openshift-default-fbccfc549-d828w"] Oct 31 00:29:47.612109 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:47.612084 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-d5360e-openshift-default-fbccfc549-d828w" podUID="37108b54-01b5-4e2c-add9-4e3aa1f3cd94" containerName="istio-proxy" containerID="cri-o://8b5f60f2b79a29f7928e516a727b54d3bd66b732c36c54850aaacf43c6888d51" gracePeriod=30 Oct 31 00:29:47.622584 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:47.622556 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:29:48.417599 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:48.417564 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-957019-tcgnw"] Oct 31 00:29:48.418085 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:48.418071 4927 cpu_manager.go:414] "containerMap: removing container" podUID="567477d1-1811-4e35-9f57-9f8eea08988a" containerName="revoke-keys" Oct 31 00:29:48.418131 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:48.418086 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="567477d1-1811-4e35-9f57-9f8eea08988a" containerName="revoke-keys" Oct 31 00:29:48.418131 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:48.418119 4927 cpu_manager.go:414] "containerMap: removing container" podUID="64551325-0270-49ec-bbc7-748cd5c3611e" containerName="payload-pre-processing" Oct 31 00:29:48.418131 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:48.418124 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64551325-0270-49ec-bbc7-748cd5c3611e" containerName="payload-pre-processing" Oct 31 00:29:48.418231 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:48.418134 4927 cpu_manager.go:414] "containerMap: removing container" podUID="e3fe218a-9831-44c3-a5eb-09e14994459f" containerName="maas-api" Oct 31 00:29:48.418231 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:48.418140 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3fe218a-9831-44c3-a5eb-09e14994459f" containerName="maas-api" Oct 31 00:29:48.418231 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:48.418149 4927 cpu_manager.go:414] "containerMap: removing container" podUID="e27553d8-0637-46a1-bd84-186e69ba3f15" containerName="payload-processing" Oct 31 00:29:48.418231 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:48.418154 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e27553d8-0637-46a1-bd84-186e69ba3f15" containerName="payload-processing" Oct 31 00:29:48.418231 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:48.418227 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="64551325-0270-49ec-bbc7-748cd5c3611e" containerName="payload-pre-processing" Oct 31 00:29:48.418444 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:48.418237 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3fe218a-9831-44c3-a5eb-09e14994459f" containerName="maas-api" Oct 31 00:29:48.418444 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:48.418243 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="e27553d8-0637-46a1-bd84-186e69ba3f15" containerName="payload-processing" Oct 31 00:29:48.418444 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:48.418252 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="567477d1-1811-4e35-9f57-9f8eea08988a" containerName="revoke-keys" Oct 31 00:29:48.423150 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:48.423133 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-957019-tcgnw" Oct 31 00:29:48.481007 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:48.480974 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-957019-tcgnw"] Oct 31 00:29:48.511670 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:48.511640 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmx5l\" (UniqueName: \"kubernetes.io/projected/68687d7b-81bd-4129-af1a-be643b52aa58-kube-api-access-bmx5l\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-957019-tcgnw\" (UID: \"68687d7b-81bd-4129-af1a-be643b52aa58\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-957019-tcgnw" Oct 31 00:29:48.511845 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:48.511791 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68687d7b-81bd-4129-af1a-be643b52aa58-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-957019-tcgnw\" (UID: \"68687d7b-81bd-4129-af1a-be643b52aa58\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-957019-tcgnw" Oct 31 00:29:48.511912 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:48.511871 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68687d7b-81bd-4129-af1a-be643b52aa58-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-957019-tcgnw\" (UID: \"68687d7b-81bd-4129-af1a-be643b52aa58\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-957019-tcgnw" Oct 31 00:29:48.527178 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:48.527152 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:29:48.527294 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:48.527185 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4fdzs\"" Oct 31 00:29:48.612538 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:48.612503 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmx5l\" (UniqueName: \"kubernetes.io/projected/68687d7b-81bd-4129-af1a-be643b52aa58-kube-api-access-bmx5l\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-957019-tcgnw\" (UID: \"68687d7b-81bd-4129-af1a-be643b52aa58\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-957019-tcgnw" Oct 31 00:29:48.612708 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:48.612588 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68687d7b-81bd-4129-af1a-be643b52aa58-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-957019-tcgnw\" (UID: \"68687d7b-81bd-4129-af1a-be643b52aa58\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-957019-tcgnw" Oct 31 00:29:48.612708 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:48.612637 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68687d7b-81bd-4129-af1a-be643b52aa58-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-957019-tcgnw\" (UID: \"68687d7b-81bd-4129-af1a-be643b52aa58\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-957019-tcgnw" Oct 31 00:29:48.613309 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:48.613281 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68687d7b-81bd-4129-af1a-be643b52aa58-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-957019-tcgnw\" (UID: \"68687d7b-81bd-4129-af1a-be643b52aa58\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-957019-tcgnw" Oct 31 00:29:48.614905 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:48.614888 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68687d7b-81bd-4129-af1a-be643b52aa58-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-957019-tcgnw\" (UID: \"68687d7b-81bd-4129-af1a-be643b52aa58\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-957019-tcgnw" Oct 31 00:29:48.679569 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:48.679504 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmx5l\" (UniqueName: \"kubernetes.io/projected/68687d7b-81bd-4129-af1a-be643b52aa58-kube-api-access-bmx5l\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-957019-tcgnw\" (UID: \"68687d7b-81bd-4129-af1a-be643b52aa58\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-957019-tcgnw" Oct 31 00:29:48.737496 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:48.737460 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-957019-tcgnw" Oct 31 00:29:49.068684 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:49.068602 4927 patch_prober.go:29] interesting pod/e2e-shared-a-w0-d5360e-openshift-default-fbccfc549-d828w container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 00:29:49.068684 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:49.068662 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w0-d5360e-openshift-default-fbccfc549-d828w" podUID="37108b54-01b5-4e2c-add9-4e3aa1f3cd94" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:29:49.120659 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:49.120631 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-957019-tcgnw"] Oct 31 00:29:49.122858 ip-10-0-141-180 kubenswrapper[4927]: W1031 00:29:49.122831 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68687d7b_81bd_4129_af1a_be643b52aa58.slice/crio-60e057b9f34753a57a0328ff2d65d1eb3b62f8fd20de9698b8c805dd6b01e0f2 WatchSource:0}: Error finding container 60e057b9f34753a57a0328ff2d65d1eb3b62f8fd20de9698b8c805dd6b01e0f2: Status 404 returned error can't find the container with id 60e057b9f34753a57a0328ff2d65d1eb3b62f8fd20de9698b8c805dd6b01e0f2 Oct 31 00:29:49.678864 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:29:49.678823 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sqv69" containerName="extractor" Oct 31 00:29:49.871642 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:49.871597 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-957019-tcgnw" event={"ID":"68687d7b-81bd-4129-af1a-be643b52aa58","Type":"ContainerStarted","Data":"54629f8914550a4458602e3c6b5dca77b4c74e4289f7812b2b9ce7cd2e8ebec9"} Oct 31 00:29:49.871851 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:49.871665 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-957019-tcgnw" event={"ID":"68687d7b-81bd-4129-af1a-be643b52aa58","Type":"ContainerStarted","Data":"60e057b9f34753a57a0328ff2d65d1eb3b62f8fd20de9698b8c805dd6b01e0f2"} Oct 31 00:29:49.887228 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:49.887198 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:29:49.994733 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:49.994631 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-957019-tcgnw" podStartSLOduration=1.994614882 podStartE2EDuration="1.994614882s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:29:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:29:49.992026641 +0000 UTC m=+2903.028357485" watchObservedRunningTime="2026-10-31 00:29:49.994614882 +0000 UTC m=+2903.030945703" Oct 31 00:29:50.880059 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:50.880023 4927 generic.go:356] "Generic (PLEG): container finished" podID="68687d7b-81bd-4129-af1a-be643b52aa58" containerID="54629f8914550a4458602e3c6b5dca77b4c74e4289f7812b2b9ce7cd2e8ebec9" exitCode=0 Oct 31 00:29:50.880527 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:50.880107 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-957019-tcgnw" event={"ID":"68687d7b-81bd-4129-af1a-be643b52aa58","Type":"ContainerDied","Data":"54629f8914550a4458602e3c6b5dca77b4c74e4289f7812b2b9ce7cd2e8ebec9"} Oct 31 00:29:52.032179 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:52.032142 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-957019-tcgnw" Oct 31 00:29:52.153772 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:52.153676 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/68687d7b-81bd-4129-af1a-be643b52aa58-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68687d7b-81bd-4129-af1a-be643b52aa58-maas-api-xks-ca\") pod \"68687d7b-81bd-4129-af1a-be643b52aa58\" (UID: \"68687d7b-81bd-4129-af1a-be643b52aa58\") " Oct 31 00:29:52.153969 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:52.153869 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68687d7b-81bd-4129-af1a-be643b52aa58-kube-api-access-bmx5l\" (UniqueName: \"kubernetes.io/projected/68687d7b-81bd-4129-af1a-be643b52aa58-kube-api-access-bmx5l\") pod \"68687d7b-81bd-4129-af1a-be643b52aa58\" (UID: \"68687d7b-81bd-4129-af1a-be643b52aa58\") " Oct 31 00:29:52.153969 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:52.153948 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/68687d7b-81bd-4129-af1a-be643b52aa58-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68687d7b-81bd-4129-af1a-be643b52aa58-maas-api-openshift-ca\") pod \"68687d7b-81bd-4129-af1a-be643b52aa58\" (UID: \"68687d7b-81bd-4129-af1a-be643b52aa58\") " Oct 31 00:29:52.154734 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:52.154710 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68687d7b-81bd-4129-af1a-be643b52aa58-maas-api-openshift-ca" pod "68687d7b-81bd-4129-af1a-be643b52aa58" (UID: "68687d7b-81bd-4129-af1a-be643b52aa58"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:29:52.157202 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:52.157173 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/68687d7b-81bd-4129-af1a-be643b52aa58-maas-api-xks-ca" pod "68687d7b-81bd-4129-af1a-be643b52aa58" (UID: "68687d7b-81bd-4129-af1a-be643b52aa58"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:29:52.160574 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:52.160519 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68687d7b-81bd-4129-af1a-be643b52aa58-kube-api-access-bmx5l" pod "68687d7b-81bd-4129-af1a-be643b52aa58" (UID: "68687d7b-81bd-4129-af1a-be643b52aa58"). InnerVolumeSpecName "kube-api-access-bmx5l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:29:52.255561 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:52.255320 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bmx5l\" (UniqueName: \"kubernetes.io/projected/68687d7b-81bd-4129-af1a-be643b52aa58-kube-api-access-bmx5l\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:52.255561 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:52.255397 4927 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68687d7b-81bd-4129-af1a-be643b52aa58-maas-api-openshift-ca\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:52.255561 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:52.255414 4927 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68687d7b-81bd-4129-af1a-be643b52aa58-maas-api-xks-ca\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:52.863236 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:52.863207 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-d5360e-openshift-default-fbccfc549-d828w" Oct 31 00:29:52.895035 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:52.894958 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-957019-tcgnw" Oct 31 00:29:52.895035 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:52.894990 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-957019-tcgnw" event={"ID":"68687d7b-81bd-4129-af1a-be643b52aa58","Type":"ContainerDied","Data":"60e057b9f34753a57a0328ff2d65d1eb3b62f8fd20de9698b8c805dd6b01e0f2"} Oct 31 00:29:52.895234 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:52.895078 4927 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="60e057b9f34753a57a0328ff2d65d1eb3b62f8fd20de9698b8c805dd6b01e0f2" Oct 31 00:29:52.896784 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:52.896757 4927 generic.go:356] "Generic (PLEG): container finished" podID="37108b54-01b5-4e2c-add9-4e3aa1f3cd94" containerID="8b5f60f2b79a29f7928e516a727b54d3bd66b732c36c54850aaacf43c6888d51" exitCode=0 Oct 31 00:29:52.896916 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:52.896793 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-d5360e-openshift-default-fbccfc549-d828w" event={"ID":"37108b54-01b5-4e2c-add9-4e3aa1f3cd94","Type":"ContainerDied","Data":"8b5f60f2b79a29f7928e516a727b54d3bd66b732c36c54850aaacf43c6888d51"} Oct 31 00:29:52.896916 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:52.896802 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-d5360e-openshift-default-fbccfc549-d828w" Oct 31 00:29:52.896916 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:52.896836 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-d5360e-openshift-default-fbccfc549-d828w" event={"ID":"37108b54-01b5-4e2c-add9-4e3aa1f3cd94","Type":"ContainerDied","Data":"8dffbad23077f74dd5e2da8f2cab7363f373041631cc3b9a1dcb0919c4a07efc"} Oct 31 00:29:52.896916 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:52.896865 4927 scope.go:122] "RemoveContainer" containerID="8b5f60f2b79a29f7928e516a727b54d3bd66b732c36c54850aaacf43c6888d51" Oct 31 00:29:52.907923 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:52.907904 4927 scope.go:122] "RemoveContainer" containerID="1afa8b2c79b4cf4621ad2544e1b2e946d465ab8c3fd31dbf19c27ef7e50497fb" Oct 31 00:29:52.917486 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:52.917467 4927 scope.go:122] "RemoveContainer" containerID="8b5f60f2b79a29f7928e516a727b54d3bd66b732c36c54850aaacf43c6888d51" Oct 31 00:29:52.917777 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:29:52.917760 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b5f60f2b79a29f7928e516a727b54d3bd66b732c36c54850aaacf43c6888d51\": container with ID starting with 8b5f60f2b79a29f7928e516a727b54d3bd66b732c36c54850aaacf43c6888d51 not found: ID does not exist" containerID="8b5f60f2b79a29f7928e516a727b54d3bd66b732c36c54850aaacf43c6888d51" Oct 31 00:29:52.917849 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:52.917786 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b5f60f2b79a29f7928e516a727b54d3bd66b732c36c54850aaacf43c6888d51"} err="failed to get container status \"8b5f60f2b79a29f7928e516a727b54d3bd66b732c36c54850aaacf43c6888d51\": rpc error: code = NotFound desc = could not find container \"8b5f60f2b79a29f7928e516a727b54d3bd66b732c36c54850aaacf43c6888d51\": container with ID starting with 8b5f60f2b79a29f7928e516a727b54d3bd66b732c36c54850aaacf43c6888d51 not found: ID does not exist" Oct 31 00:29:52.917849 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:52.917804 4927 scope.go:122] "RemoveContainer" containerID="1afa8b2c79b4cf4621ad2544e1b2e946d465ab8c3fd31dbf19c27ef7e50497fb" Oct 31 00:29:52.918085 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:29:52.918066 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1afa8b2c79b4cf4621ad2544e1b2e946d465ab8c3fd31dbf19c27ef7e50497fb\": container with ID starting with 1afa8b2c79b4cf4621ad2544e1b2e946d465ab8c3fd31dbf19c27ef7e50497fb not found: ID does not exist" containerID="1afa8b2c79b4cf4621ad2544e1b2e946d465ab8c3fd31dbf19c27ef7e50497fb" Oct 31 00:29:52.918142 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:52.918091 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1afa8b2c79b4cf4621ad2544e1b2e946d465ab8c3fd31dbf19c27ef7e50497fb"} err="failed to get container status \"1afa8b2c79b4cf4621ad2544e1b2e946d465ab8c3fd31dbf19c27ef7e50497fb\": rpc error: code = NotFound desc = could not find container \"1afa8b2c79b4cf4621ad2544e1b2e946d465ab8c3fd31dbf19c27ef7e50497fb\": container with ID starting with 1afa8b2c79b4cf4621ad2544e1b2e946d465ab8c3fd31dbf19c27ef7e50497fb not found: ID does not exist" Oct 31 00:29:52.963520 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:52.963482 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37108b54-01b5-4e2c-add9-4e3aa1f3cd94-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37108b54-01b5-4e2c-add9-4e3aa1f3cd94-workload-socket\") pod \"37108b54-01b5-4e2c-add9-4e3aa1f3cd94\" (UID: \"37108b54-01b5-4e2c-add9-4e3aa1f3cd94\") " Oct 31 00:29:52.963692 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:52.963540 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37108b54-01b5-4e2c-add9-4e3aa1f3cd94-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37108b54-01b5-4e2c-add9-4e3aa1f3cd94-workload-certs\") pod \"37108b54-01b5-4e2c-add9-4e3aa1f3cd94\" (UID: \"37108b54-01b5-4e2c-add9-4e3aa1f3cd94\") " Oct 31 00:29:52.963692 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:52.963595 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/37108b54-01b5-4e2c-add9-4e3aa1f3cd94-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37108b54-01b5-4e2c-add9-4e3aa1f3cd94-istiod-ca-cert\") pod \"37108b54-01b5-4e2c-add9-4e3aa1f3cd94\" (UID: \"37108b54-01b5-4e2c-add9-4e3aa1f3cd94\") " Oct 31 00:29:52.963692 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:52.963651 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37108b54-01b5-4e2c-add9-4e3aa1f3cd94-kube-api-access-2qdvx\" (UniqueName: \"kubernetes.io/projected/37108b54-01b5-4e2c-add9-4e3aa1f3cd94-kube-api-access-2qdvx\") pod \"37108b54-01b5-4e2c-add9-4e3aa1f3cd94\" (UID: \"37108b54-01b5-4e2c-add9-4e3aa1f3cd94\") " Oct 31 00:29:52.963895 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:52.963700 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37108b54-01b5-4e2c-add9-4e3aa1f3cd94-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37108b54-01b5-4e2c-add9-4e3aa1f3cd94-istio-envoy\") pod \"37108b54-01b5-4e2c-add9-4e3aa1f3cd94\" (UID: \"37108b54-01b5-4e2c-add9-4e3aa1f3cd94\") " Oct 31 00:29:52.963895 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:52.963734 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/37108b54-01b5-4e2c-add9-4e3aa1f3cd94-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37108b54-01b5-4e2c-add9-4e3aa1f3cd94-istio-podinfo\") pod \"37108b54-01b5-4e2c-add9-4e3aa1f3cd94\" (UID: \"37108b54-01b5-4e2c-add9-4e3aa1f3cd94\") " Oct 31 00:29:52.963895 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:52.963771 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37108b54-01b5-4e2c-add9-4e3aa1f3cd94-istio-token\" (UniqueName: \"kubernetes.io/projected/37108b54-01b5-4e2c-add9-4e3aa1f3cd94-istio-token\") pod \"37108b54-01b5-4e2c-add9-4e3aa1f3cd94\" (UID: \"37108b54-01b5-4e2c-add9-4e3aa1f3cd94\") " Oct 31 00:29:52.963895 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:52.963799 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37108b54-01b5-4e2c-add9-4e3aa1f3cd94-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37108b54-01b5-4e2c-add9-4e3aa1f3cd94-istio-data\") pod \"37108b54-01b5-4e2c-add9-4e3aa1f3cd94\" (UID: \"37108b54-01b5-4e2c-add9-4e3aa1f3cd94\") " Oct 31 00:29:52.963895 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:52.963821 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37108b54-01b5-4e2c-add9-4e3aa1f3cd94-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37108b54-01b5-4e2c-add9-4e3aa1f3cd94-credential-socket\") pod \"37108b54-01b5-4e2c-add9-4e3aa1f3cd94\" (UID: \"37108b54-01b5-4e2c-add9-4e3aa1f3cd94\") " Oct 31 00:29:52.964174 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:52.963811 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37108b54-01b5-4e2c-add9-4e3aa1f3cd94-workload-socket" pod "37108b54-01b5-4e2c-add9-4e3aa1f3cd94" (UID: "37108b54-01b5-4e2c-add9-4e3aa1f3cd94"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:29:52.964264 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:52.963901 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37108b54-01b5-4e2c-add9-4e3aa1f3cd94-workload-certs" pod "37108b54-01b5-4e2c-add9-4e3aa1f3cd94" (UID: "37108b54-01b5-4e2c-add9-4e3aa1f3cd94"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:29:52.964323 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:52.964123 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37108b54-01b5-4e2c-add9-4e3aa1f3cd94-istio-data" pod "37108b54-01b5-4e2c-add9-4e3aa1f3cd94" (UID: "37108b54-01b5-4e2c-add9-4e3aa1f3cd94"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:29:52.964323 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:52.964137 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37108b54-01b5-4e2c-add9-4e3aa1f3cd94-credential-socket" pod "37108b54-01b5-4e2c-add9-4e3aa1f3cd94" (UID: "37108b54-01b5-4e2c-add9-4e3aa1f3cd94"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:29:52.964323 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:52.964274 4927 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37108b54-01b5-4e2c-add9-4e3aa1f3cd94-workload-socket\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:52.964531 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:52.964389 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37108b54-01b5-4e2c-add9-4e3aa1f3cd94-istiod-ca-cert" pod "37108b54-01b5-4e2c-add9-4e3aa1f3cd94" (UID: "37108b54-01b5-4e2c-add9-4e3aa1f3cd94"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:29:52.965977 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:52.965947 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37108b54-01b5-4e2c-add9-4e3aa1f3cd94-istio-envoy" pod "37108b54-01b5-4e2c-add9-4e3aa1f3cd94" (UID: "37108b54-01b5-4e2c-add9-4e3aa1f3cd94"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:29:52.965977 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:52.965951 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/37108b54-01b5-4e2c-add9-4e3aa1f3cd94-istio-podinfo" pod "37108b54-01b5-4e2c-add9-4e3aa1f3cd94" (UID: "37108b54-01b5-4e2c-add9-4e3aa1f3cd94"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:29:52.966120 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:52.966039 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37108b54-01b5-4e2c-add9-4e3aa1f3cd94-istio-token" pod "37108b54-01b5-4e2c-add9-4e3aa1f3cd94" (UID: "37108b54-01b5-4e2c-add9-4e3aa1f3cd94"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:29:52.966120 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:52.966066 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37108b54-01b5-4e2c-add9-4e3aa1f3cd94-kube-api-access-2qdvx" pod "37108b54-01b5-4e2c-add9-4e3aa1f3cd94" (UID: "37108b54-01b5-4e2c-add9-4e3aa1f3cd94"). InnerVolumeSpecName "kube-api-access-2qdvx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:29:53.065533 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:53.065499 4927 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37108b54-01b5-4e2c-add9-4e3aa1f3cd94-istio-podinfo\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:53.065533 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:53.065529 4927 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37108b54-01b5-4e2c-add9-4e3aa1f3cd94-istio-token\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:53.065533 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:53.065539 4927 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37108b54-01b5-4e2c-add9-4e3aa1f3cd94-istio-data\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:53.065982 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:53.065549 4927 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37108b54-01b5-4e2c-add9-4e3aa1f3cd94-credential-socket\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:53.065982 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:53.065557 4927 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37108b54-01b5-4e2c-add9-4e3aa1f3cd94-workload-certs\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:53.065982 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:53.065566 4927 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37108b54-01b5-4e2c-add9-4e3aa1f3cd94-istiod-ca-cert\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:53.065982 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:53.065574 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2qdvx\" (UniqueName: \"kubernetes.io/projected/37108b54-01b5-4e2c-add9-4e3aa1f3cd94-kube-api-access-2qdvx\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:53.065982 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:53.065584 4927 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37108b54-01b5-4e2c-add9-4e3aa1f3cd94-istio-envoy\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:53.318783 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:53.318746 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-d5360e-openshift-default-fbccfc549-d828w"] Oct 31 00:29:53.322203 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:53.322179 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-d5360e-openshift-default-fbccfc549-d828w"] Oct 31 00:29:54.692734 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:54.692696 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:29:55.087428 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:55.087353 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="37108b54-01b5-4e2c-add9-4e3aa1f3cd94" path="/var/lib/kubelet/pods/37108b54-01b5-4e2c-add9-4e3aa1f3cd94/volumes" Oct 31 00:29:59.338534 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:59.338484 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:29:59.820464 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:59.820428 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-957019-788c577f9b-2shs7"] Oct 31 00:29:59.921761 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:29:59.921722 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-957019-76d548dcf-bbqz5"] Oct 31 00:30:00.167861 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:00.167825 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ddf61a-29890110-rfdvn"] Oct 31 00:30:00.168646 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:00.168612 4927 cpu_manager.go:414] "containerMap: removing container" podUID="37108b54-01b5-4e2c-add9-4e3aa1f3cd94" containerName="istio-proxy" Oct 31 00:30:00.168646 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:00.168638 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37108b54-01b5-4e2c-add9-4e3aa1f3cd94" containerName="istio-proxy" Oct 31 00:30:00.168805 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:00.168675 4927 cpu_manager.go:414] "containerMap: removing container" podUID="37108b54-01b5-4e2c-add9-4e3aa1f3cd94" containerName="istio-proxy" Oct 31 00:30:00.168805 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:00.168685 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37108b54-01b5-4e2c-add9-4e3aa1f3cd94" containerName="istio-proxy" Oct 31 00:30:00.168805 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:00.168704 4927 cpu_manager.go:414] "containerMap: removing container" podUID="68687d7b-81bd-4129-af1a-be643b52aa58" containerName="revoke-keys" Oct 31 00:30:00.168805 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:00.168711 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68687d7b-81bd-4129-af1a-be643b52aa58" containerName="revoke-keys" Oct 31 00:30:00.168981 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:00.168843 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="37108b54-01b5-4e2c-add9-4e3aa1f3cd94" containerName="istio-proxy" Oct 31 00:30:00.168981 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:00.168860 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="68687d7b-81bd-4129-af1a-be643b52aa58" containerName="revoke-keys" Oct 31 00:30:00.172993 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:00.172972 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-957019-29890110-bpqlv"] Oct 31 00:30:00.173095 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:00.173083 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ddf61a-29890110-rfdvn" Oct 31 00:30:00.173683 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:00.173659 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="37108b54-01b5-4e2c-add9-4e3aa1f3cd94" containerName="istio-proxy" Oct 31 00:30:00.176843 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:00.176823 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-957019-29890110-bpqlv" Oct 31 00:30:00.194575 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:00.194549 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-46b586-29890110-mvw2k"] Oct 31 00:30:00.198188 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:00.198170 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-46b586-29890110-mvw2k" Oct 31 00:30:00.199122 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:00.199095 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:30:00.249850 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:00.249810 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0fda0711-f23d-409b-af43-23d6bbb8aac7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-957019-29890110-bpqlv\" (UID: \"0fda0711-f23d-409b-af43-23d6bbb8aac7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-957019-29890110-bpqlv" Oct 31 00:30:00.250047 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:00.249862 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nssm5\" (UniqueName: \"kubernetes.io/projected/8b01ad9a-b63a-40ea-9dfb-4ea15d74bc50-kube-api-access-nssm5\") pod \"maas-api-key-cleanup-e2e-worker-w0-46b586-29890110-mvw2k\" (UID: \"8b01ad9a-b63a-40ea-9dfb-4ea15d74bc50\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-46b586-29890110-mvw2k" Oct 31 00:30:00.250047 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:00.249905 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b75ee4b6-1c0b-4a64-a4a9-14da52d5f7db-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-ddf61a-29890110-rfdvn\" (UID: \"b75ee4b6-1c0b-4a64-a4a9-14da52d5f7db\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ddf61a-29890110-rfdvn" Oct 31 00:30:00.250047 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:00.249933 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5q77b\" (UniqueName: \"kubernetes.io/projected/0fda0711-f23d-409b-af43-23d6bbb8aac7-kube-api-access-5q77b\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-957019-29890110-bpqlv\" (UID: \"0fda0711-f23d-409b-af43-23d6bbb8aac7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-957019-29890110-bpqlv" Oct 31 00:30:00.250195 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:00.250045 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9h7nm\" (UniqueName: \"kubernetes.io/projected/b75ee4b6-1c0b-4a64-a4a9-14da52d5f7db-kube-api-access-9h7nm\") pod \"maas-api-key-cleanup-e2e-inf-a-ddf61a-29890110-rfdvn\" (UID: \"b75ee4b6-1c0b-4a64-a4a9-14da52d5f7db\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ddf61a-29890110-rfdvn" Oct 31 00:30:00.251809 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:00.251778 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b01ad9a-b63a-40ea-9dfb-4ea15d74bc50-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-46b586-29890110-mvw2k\" (UID: \"8b01ad9a-b63a-40ea-9dfb-4ea15d74bc50\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-46b586-29890110-mvw2k" Oct 31 00:30:00.256436 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:00.256409 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-957019-29890110-bpqlv"] Oct 31 00:30:00.260919 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:00.260895 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ddf61a-29890110-rfdvn"] Oct 31 00:30:00.283248 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:00.283219 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-46b586-29890110-mvw2k"] Oct 31 00:30:00.328610 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:00.328580 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4fdzs\"" Oct 31 00:30:00.366834 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:00.366798 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9h7nm\" (UniqueName: \"kubernetes.io/projected/b75ee4b6-1c0b-4a64-a4a9-14da52d5f7db-kube-api-access-9h7nm\") pod \"maas-api-key-cleanup-e2e-inf-a-ddf61a-29890110-rfdvn\" (UID: \"b75ee4b6-1c0b-4a64-a4a9-14da52d5f7db\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ddf61a-29890110-rfdvn" Oct 31 00:30:00.367469 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:00.367444 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b01ad9a-b63a-40ea-9dfb-4ea15d74bc50-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-46b586-29890110-mvw2k\" (UID: \"8b01ad9a-b63a-40ea-9dfb-4ea15d74bc50\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-46b586-29890110-mvw2k" Oct 31 00:30:00.371535 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:00.371208 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:30:00.371535 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:00.371260 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b01ad9a-b63a-40ea-9dfb-4ea15d74bc50-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-46b586-29890110-mvw2k\" (UID: \"8b01ad9a-b63a-40ea-9dfb-4ea15d74bc50\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-46b586-29890110-mvw2k" Oct 31 00:30:00.371880 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:00.371849 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0fda0711-f23d-409b-af43-23d6bbb8aac7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-957019-29890110-bpqlv\" (UID: \"0fda0711-f23d-409b-af43-23d6bbb8aac7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-957019-29890110-bpqlv" Oct 31 00:30:00.372010 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:00.371995 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nssm5\" (UniqueName: \"kubernetes.io/projected/8b01ad9a-b63a-40ea-9dfb-4ea15d74bc50-kube-api-access-nssm5\") pod \"maas-api-key-cleanup-e2e-worker-w0-46b586-29890110-mvw2k\" (UID: \"8b01ad9a-b63a-40ea-9dfb-4ea15d74bc50\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-46b586-29890110-mvw2k" Oct 31 00:30:00.372450 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:00.372430 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b75ee4b6-1c0b-4a64-a4a9-14da52d5f7db-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-ddf61a-29890110-rfdvn\" (UID: \"b75ee4b6-1c0b-4a64-a4a9-14da52d5f7db\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ddf61a-29890110-rfdvn" Oct 31 00:30:00.373217 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:00.372609 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0fda0711-f23d-409b-af43-23d6bbb8aac7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-957019-29890110-bpqlv\" (UID: \"0fda0711-f23d-409b-af43-23d6bbb8aac7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-957019-29890110-bpqlv" Oct 31 00:30:00.373217 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:00.373175 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b75ee4b6-1c0b-4a64-a4a9-14da52d5f7db-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-ddf61a-29890110-rfdvn\" (UID: \"b75ee4b6-1c0b-4a64-a4a9-14da52d5f7db\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ddf61a-29890110-rfdvn" Oct 31 00:30:00.373395 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:00.373121 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5q77b\" (UniqueName: \"kubernetes.io/projected/0fda0711-f23d-409b-af43-23d6bbb8aac7-kube-api-access-5q77b\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-957019-29890110-bpqlv\" (UID: \"0fda0711-f23d-409b-af43-23d6bbb8aac7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-957019-29890110-bpqlv" Oct 31 00:30:00.432432 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:00.432313 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9h7nm\" (UniqueName: \"kubernetes.io/projected/b75ee4b6-1c0b-4a64-a4a9-14da52d5f7db-kube-api-access-9h7nm\") pod \"maas-api-key-cleanup-e2e-inf-a-ddf61a-29890110-rfdvn\" (UID: \"b75ee4b6-1c0b-4a64-a4a9-14da52d5f7db\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ddf61a-29890110-rfdvn" Oct 31 00:30:00.434220 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:00.434190 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nssm5\" (UniqueName: \"kubernetes.io/projected/8b01ad9a-b63a-40ea-9dfb-4ea15d74bc50-kube-api-access-nssm5\") pod \"maas-api-key-cleanup-e2e-worker-w0-46b586-29890110-mvw2k\" (UID: \"8b01ad9a-b63a-40ea-9dfb-4ea15d74bc50\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-46b586-29890110-mvw2k" Oct 31 00:30:00.434848 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:00.434830 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5q77b\" (UniqueName: \"kubernetes.io/projected/0fda0711-f23d-409b-af43-23d6bbb8aac7-kube-api-access-5q77b\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-957019-29890110-bpqlv\" (UID: \"0fda0711-f23d-409b-af43-23d6bbb8aac7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-957019-29890110-bpqlv" Oct 31 00:30:00.487674 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:00.487643 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ddf61a-29890110-rfdvn" Oct 31 00:30:00.493773 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:00.493751 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-957019-29890110-bpqlv" Oct 31 00:30:00.510905 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:00.510874 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-46b586-29890110-mvw2k" Oct 31 00:30:00.880269 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:00.880237 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ddf61a-29890110-rfdvn"] Oct 31 00:30:00.880962 ip-10-0-141-180 kubenswrapper[4927]: W1031 00:30:00.880929 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb75ee4b6_1c0b_4a64_a4a9_14da52d5f7db.slice/crio-e7b00950900a25f049646a224df8f132a1405bc73dddee0fe4883c0bc772a701 WatchSource:0}: Error finding container e7b00950900a25f049646a224df8f132a1405bc73dddee0fe4883c0bc772a701: Status 404 returned error can't find the container with id e7b00950900a25f049646a224df8f132a1405bc73dddee0fe4883c0bc772a701 Oct 31 00:30:00.930990 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:00.930958 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-957019-29890110-bpqlv"] Oct 31 00:30:00.933471 ip-10-0-141-180 kubenswrapper[4927]: W1031 00:30:00.933446 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fda0711_f23d_409b_af43_23d6bbb8aac7.slice/crio-1e48883c0794ee022ddc92b10cad3541ff48467f9eb9dd9c3e974318628fc5e6 WatchSource:0}: Error finding container 1e48883c0794ee022ddc92b10cad3541ff48467f9eb9dd9c3e974318628fc5e6: Status 404 returned error can't find the container with id 1e48883c0794ee022ddc92b10cad3541ff48467f9eb9dd9c3e974318628fc5e6 Oct 31 00:30:00.934155 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:00.934135 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-46b586-29890110-mvw2k"] Oct 31 00:30:00.934490 ip-10-0-141-180 kubenswrapper[4927]: W1031 00:30:00.934469 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b01ad9a_b63a_40ea_9dfb_4ea15d74bc50.slice/crio-d344cf3643627c8116598e20a629b5ad4ef5193603b9a2455a40093ffead5a2b WatchSource:0}: Error finding container d344cf3643627c8116598e20a629b5ad4ef5193603b9a2455a40093ffead5a2b: Status 404 returned error can't find the container with id d344cf3643627c8116598e20a629b5ad4ef5193603b9a2455a40093ffead5a2b Oct 31 00:30:00.957477 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:00.957451 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ddf61a-29890110-rfdvn" event={"ID":"b75ee4b6-1c0b-4a64-a4a9-14da52d5f7db","Type":"ContainerStarted","Data":"e7b00950900a25f049646a224df8f132a1405bc73dddee0fe4883c0bc772a701"} Oct 31 00:30:00.958673 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:00.958652 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-957019-29890110-bpqlv" event={"ID":"0fda0711-f23d-409b-af43-23d6bbb8aac7","Type":"ContainerStarted","Data":"1e48883c0794ee022ddc92b10cad3541ff48467f9eb9dd9c3e974318628fc5e6"} Oct 31 00:30:00.959869 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:00.959848 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-46b586-29890110-mvw2k" event={"ID":"8b01ad9a-b63a-40ea-9dfb-4ea15d74bc50","Type":"ContainerStarted","Data":"d344cf3643627c8116598e20a629b5ad4ef5193603b9a2455a40093ffead5a2b"} Oct 31 00:30:01.968824 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:01.968783 4927 generic.go:356] "Generic (PLEG): container finished" podID="b75ee4b6-1c0b-4a64-a4a9-14da52d5f7db" containerID="1d36e053a6e3cf693c6a82bd4bc1f490e5908ba2722ec36d4ca3fe4ec5c61246" exitCode=60 Oct 31 00:30:01.969275 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:01.968867 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ddf61a-29890110-rfdvn" event={"ID":"b75ee4b6-1c0b-4a64-a4a9-14da52d5f7db","Type":"ContainerDied","Data":"1d36e053a6e3cf693c6a82bd4bc1f490e5908ba2722ec36d4ca3fe4ec5c61246"} Oct 31 00:30:01.969275 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:01.969118 4927 scope.go:122] "RemoveContainer" containerID="1d36e053a6e3cf693c6a82bd4bc1f490e5908ba2722ec36d4ca3fe4ec5c61246" Oct 31 00:30:01.970718 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:01.970700 4927 generic.go:356] "Generic (PLEG): container finished" podID="0fda0711-f23d-409b-af43-23d6bbb8aac7" containerID="c007c3861c6ef3af294bda30bdceb246bb4ebf9a8d7f008a0f8fa2453c058d6b" exitCode=60 Oct 31 00:30:01.970807 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:01.970787 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-957019-29890110-bpqlv" event={"ID":"0fda0711-f23d-409b-af43-23d6bbb8aac7","Type":"ContainerDied","Data":"c007c3861c6ef3af294bda30bdceb246bb4ebf9a8d7f008a0f8fa2453c058d6b"} Oct 31 00:30:01.970999 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:01.970978 4927 scope.go:122] "RemoveContainer" containerID="c007c3861c6ef3af294bda30bdceb246bb4ebf9a8d7f008a0f8fa2453c058d6b" Oct 31 00:30:01.972739 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:01.972718 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-46b586-29890110-mvw2k" event={"ID":"8b01ad9a-b63a-40ea-9dfb-4ea15d74bc50","Type":"ContainerStarted","Data":"7242c0e2e960d3ee51a859005f283483ab879acdd64479079284ebfbafd90b66"} Oct 31 00:30:02.325816 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:02.325765 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-46b586-29890110-mvw2k" podStartSLOduration=2.325751312 podStartE2EDuration="2.325751312s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:30:02.323395847 +0000 UTC m=+2915.359726671" watchObservedRunningTime="2026-10-31 00:30:02.325751312 +0000 UTC m=+2915.362082133" Oct 31 00:30:02.981530 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:02.981494 4927 generic.go:356] "Generic (PLEG): container finished" podID="0fda0711-f23d-409b-af43-23d6bbb8aac7" containerID="3b822201af9d57d9c63fc4e2805ef459c0732e7ae6de0ebd8906d4119fb8c07e" exitCode=60 Oct 31 00:30:02.981939 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:02.981659 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-957019-29890110-bpqlv" event={"ID":"0fda0711-f23d-409b-af43-23d6bbb8aac7","Type":"ContainerDied","Data":"3b822201af9d57d9c63fc4e2805ef459c0732e7ae6de0ebd8906d4119fb8c07e"} Oct 31 00:30:02.981939 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:02.981704 4927 scope.go:122] "RemoveContainer" containerID="c007c3861c6ef3af294bda30bdceb246bb4ebf9a8d7f008a0f8fa2453c058d6b" Oct 31 00:30:02.981939 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:02.981828 4927 scope.go:122] "RemoveContainer" containerID="3b822201af9d57d9c63fc4e2805ef459c0732e7ae6de0ebd8906d4119fb8c07e" Oct 31 00:30:02.982150 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:30:02.982128 4927 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-957019-29890110-bpqlv_odh-ai-gateway-infra(0fda0711-f23d-409b-af43-23d6bbb8aac7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-957019-29890110-bpqlv" podUID="0fda0711-f23d-409b-af43-23d6bbb8aac7" Oct 31 00:30:02.983698 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:02.983675 4927 generic.go:356] "Generic (PLEG): container finished" podID="8b01ad9a-b63a-40ea-9dfb-4ea15d74bc50" containerID="7242c0e2e960d3ee51a859005f283483ab879acdd64479079284ebfbafd90b66" exitCode=60 Oct 31 00:30:02.983821 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:02.983751 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-46b586-29890110-mvw2k" event={"ID":"8b01ad9a-b63a-40ea-9dfb-4ea15d74bc50","Type":"ContainerDied","Data":"7242c0e2e960d3ee51a859005f283483ab879acdd64479079284ebfbafd90b66"} Oct 31 00:30:02.983942 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:02.983922 4927 scope.go:122] "RemoveContainer" containerID="7242c0e2e960d3ee51a859005f283483ab879acdd64479079284ebfbafd90b66" Oct 31 00:30:02.986425 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:02.986403 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ddf61a-29890110-rfdvn" event={"ID":"b75ee4b6-1c0b-4a64-a4a9-14da52d5f7db","Type":"ContainerStarted","Data":"ec35e77001edfa2c1882e272a20beaa0c835c081a5343b86ddd2cc72658c3e04"} Oct 31 00:30:03.350414 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:03.350353 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ddf61a-29890110-rfdvn" podStartSLOduration=3.3503244150000002 podStartE2EDuration="3.350324415s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:30:03.348312052 +0000 UTC m=+2916.384642876" watchObservedRunningTime="2026-10-31 00:30:03.350324415 +0000 UTC m=+2916.386655236" Oct 31 00:30:03.995326 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:03.995218 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:30:03.996163 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:03.996132 4927 generic.go:356] "Generic (PLEG): container finished" podID="b75ee4b6-1c0b-4a64-a4a9-14da52d5f7db" containerID="ec35e77001edfa2c1882e272a20beaa0c835c081a5343b86ddd2cc72658c3e04" exitCode=60 Oct 31 00:30:03.996299 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:03.996212 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ddf61a-29890110-rfdvn" event={"ID":"b75ee4b6-1c0b-4a64-a4a9-14da52d5f7db","Type":"ContainerDied","Data":"ec35e77001edfa2c1882e272a20beaa0c835c081a5343b86ddd2cc72658c3e04"} Oct 31 00:30:03.996299 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:03.996276 4927 scope.go:122] "RemoveContainer" containerID="1d36e053a6e3cf693c6a82bd4bc1f490e5908ba2722ec36d4ca3fe4ec5c61246" Oct 31 00:30:03.996574 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:03.996559 4927 scope.go:122] "RemoveContainer" containerID="ec35e77001edfa2c1882e272a20beaa0c835c081a5343b86ddd2cc72658c3e04" Oct 31 00:30:03.996903 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:30:03.996881 4927 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-ddf61a-29890110-rfdvn_odh-ai-gateway-infra(b75ee4b6-1c0b-4a64-a4a9-14da52d5f7db)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ddf61a-29890110-rfdvn" podUID="b75ee4b6-1c0b-4a64-a4a9-14da52d5f7db" Oct 31 00:30:03.998449 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:03.998430 4927 scope.go:122] "RemoveContainer" containerID="3b822201af9d57d9c63fc4e2805ef459c0732e7ae6de0ebd8906d4119fb8c07e" Oct 31 00:30:03.998692 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:30:03.998668 4927 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-957019-29890110-bpqlv_odh-ai-gateway-infra(0fda0711-f23d-409b-af43-23d6bbb8aac7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-957019-29890110-bpqlv" podUID="0fda0711-f23d-409b-af43-23d6bbb8aac7" Oct 31 00:30:04.000093 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:04.000070 4927 generic.go:356] "Generic (PLEG): container finished" podID="8b01ad9a-b63a-40ea-9dfb-4ea15d74bc50" containerID="09ba4e35a398de95d41917b071e49917cb5c6c79a34817f133eb5ddf9c70ab35" exitCode=60 Oct 31 00:30:04.000186 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:04.000150 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-46b586-29890110-mvw2k" event={"ID":"8b01ad9a-b63a-40ea-9dfb-4ea15d74bc50","Type":"ContainerDied","Data":"09ba4e35a398de95d41917b071e49917cb5c6c79a34817f133eb5ddf9c70ab35"} Oct 31 00:30:04.000355 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:04.000324 4927 scope.go:122] "RemoveContainer" containerID="09ba4e35a398de95d41917b071e49917cb5c6c79a34817f133eb5ddf9c70ab35" Oct 31 00:30:04.000560 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:30:04.000542 4927 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-46b586-29890110-mvw2k_odh-ai-gateway-infra(8b01ad9a-b63a-40ea-9dfb-4ea15d74bc50)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-46b586-29890110-mvw2k" podUID="8b01ad9a-b63a-40ea-9dfb-4ea15d74bc50" Oct 31 00:30:04.009244 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:04.009221 4927 scope.go:122] "RemoveContainer" containerID="7242c0e2e960d3ee51a859005f283483ab879acdd64479079284ebfbafd90b66" Oct 31 00:30:04.821294 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:04.821239 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-957019-788c577f9b-2shs7" podUID="ed77f29b-ed8c-428a-a8f7-1c55892d64a8" containerName="payload-processing" containerID="cri-o://99501923286cd113e2da0d7532e8dde4c29e085e88863f546ad571f09c2db9fd" gracePeriod=25 Oct 31 00:30:04.922508 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:04.922443 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-957019-76d548dcf-bbqz5" podUID="358e61e1-8b72-4d73-b3ec-9016ebe766df" containerName="payload-pre-processing" containerID="cri-o://ac2ec1840a36d20ada7e088ed1f12b11a9b7fbdfd9a9ef229d46de9154756250" gracePeriod=25 Oct 31 00:30:05.014110 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:05.014070 4927 generic.go:356] "Generic (PLEG): container finished" podID="ed77f29b-ed8c-428a-a8f7-1c55892d64a8" containerID="99501923286cd113e2da0d7532e8dde4c29e085e88863f546ad571f09c2db9fd" exitCode=0 Oct 31 00:30:05.014639 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:05.014231 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-957019-788c577f9b-2shs7" event={"ID":"ed77f29b-ed8c-428a-a8f7-1c55892d64a8","Type":"ContainerDied","Data":"99501923286cd113e2da0d7532e8dde4c29e085e88863f546ad571f09c2db9fd"} Oct 31 00:30:05.016655 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:05.016628 4927 scope.go:122] "RemoveContainer" containerID="09ba4e35a398de95d41917b071e49917cb5c6c79a34817f133eb5ddf9c70ab35" Oct 31 00:30:05.017048 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:30:05.017012 4927 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-46b586-29890110-mvw2k_odh-ai-gateway-infra(8b01ad9a-b63a-40ea-9dfb-4ea15d74bc50)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-46b586-29890110-mvw2k" podUID="8b01ad9a-b63a-40ea-9dfb-4ea15d74bc50" Oct 31 00:30:05.018695 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:05.018673 4927 scope.go:122] "RemoveContainer" containerID="ec35e77001edfa2c1882e272a20beaa0c835c081a5343b86ddd2cc72658c3e04" Oct 31 00:30:05.018982 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:30:05.018959 4927 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-ddf61a-29890110-rfdvn_odh-ai-gateway-infra(b75ee4b6-1c0b-4a64-a4a9-14da52d5f7db)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ddf61a-29890110-rfdvn" podUID="b75ee4b6-1c0b-4a64-a4a9-14da52d5f7db" Oct 31 00:30:05.097254 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:05.097217 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-957019-788c577f9b-2shs7" Oct 31 00:30:05.198568 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:05.198539 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-957019-76d548dcf-bbqz5" Oct 31 00:30:05.229100 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:05.229062 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed77f29b-ed8c-428a-a8f7-1c55892d64a8-kube-api-access-fb8gp\" (UniqueName: \"kubernetes.io/projected/ed77f29b-ed8c-428a-a8f7-1c55892d64a8-kube-api-access-fb8gp\") pod \"ed77f29b-ed8c-428a-a8f7-1c55892d64a8\" (UID: \"ed77f29b-ed8c-428a-a8f7-1c55892d64a8\") " Oct 31 00:30:05.229310 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:05.229186 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed77f29b-ed8c-428a-a8f7-1c55892d64a8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed77f29b-ed8c-428a-a8f7-1c55892d64a8-plugins-config-volume\") pod \"ed77f29b-ed8c-428a-a8f7-1c55892d64a8\" (UID: \"ed77f29b-ed8c-428a-a8f7-1c55892d64a8\") " Oct 31 00:30:05.229600 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:05.229576 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed77f29b-ed8c-428a-a8f7-1c55892d64a8-plugins-config-volume" pod "ed77f29b-ed8c-428a-a8f7-1c55892d64a8" (UID: "ed77f29b-ed8c-428a-a8f7-1c55892d64a8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:30:05.231269 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:05.231240 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed77f29b-ed8c-428a-a8f7-1c55892d64a8-kube-api-access-fb8gp" pod "ed77f29b-ed8c-428a-a8f7-1c55892d64a8" (UID: "ed77f29b-ed8c-428a-a8f7-1c55892d64a8"). InnerVolumeSpecName "kube-api-access-fb8gp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:30:05.330296 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:05.330216 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/358e61e1-8b72-4d73-b3ec-9016ebe766df-kube-api-access-pndm9\" (UniqueName: \"kubernetes.io/projected/358e61e1-8b72-4d73-b3ec-9016ebe766df-kube-api-access-pndm9\") pod \"358e61e1-8b72-4d73-b3ec-9016ebe766df\" (UID: \"358e61e1-8b72-4d73-b3ec-9016ebe766df\") " Oct 31 00:30:05.330548 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:05.330378 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/358e61e1-8b72-4d73-b3ec-9016ebe766df-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/358e61e1-8b72-4d73-b3ec-9016ebe766df-plugins-config-volume\") pod \"358e61e1-8b72-4d73-b3ec-9016ebe766df\" (UID: \"358e61e1-8b72-4d73-b3ec-9016ebe766df\") " Oct 31 00:30:05.330720 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:05.330701 4927 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed77f29b-ed8c-428a-a8f7-1c55892d64a8-plugins-config-volume\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:05.330776 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:05.330728 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fb8gp\" (UniqueName: \"kubernetes.io/projected/ed77f29b-ed8c-428a-a8f7-1c55892d64a8-kube-api-access-fb8gp\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:05.330812 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:05.330784 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/358e61e1-8b72-4d73-b3ec-9016ebe766df-plugins-config-volume" pod "358e61e1-8b72-4d73-b3ec-9016ebe766df" (UID: "358e61e1-8b72-4d73-b3ec-9016ebe766df"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:30:05.332381 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:05.332358 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/358e61e1-8b72-4d73-b3ec-9016ebe766df-kube-api-access-pndm9" pod "358e61e1-8b72-4d73-b3ec-9016ebe766df" (UID: "358e61e1-8b72-4d73-b3ec-9016ebe766df"). InnerVolumeSpecName "kube-api-access-pndm9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:30:05.431784 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:05.431744 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pndm9\" (UniqueName: \"kubernetes.io/projected/358e61e1-8b72-4d73-b3ec-9016ebe766df-kube-api-access-pndm9\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:05.431784 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:05.431779 4927 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/358e61e1-8b72-4d73-b3ec-9016ebe766df-plugins-config-volume\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:06.026916 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:06.026879 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-957019-788c577f9b-2shs7" Oct 31 00:30:06.026916 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:06.026899 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-957019-788c577f9b-2shs7" event={"ID":"ed77f29b-ed8c-428a-a8f7-1c55892d64a8","Type":"ContainerDied","Data":"488c18389ffd9db13a0da965bee820f1572087eff0c93eabd0fd584d8290b9c0"} Oct 31 00:30:06.027479 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:06.026991 4927 scope.go:122] "RemoveContainer" containerID="99501923286cd113e2da0d7532e8dde4c29e085e88863f546ad571f09c2db9fd" Oct 31 00:30:06.028599 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:06.028564 4927 generic.go:356] "Generic (PLEG): container finished" podID="358e61e1-8b72-4d73-b3ec-9016ebe766df" containerID="ac2ec1840a36d20ada7e088ed1f12b11a9b7fbdfd9a9ef229d46de9154756250" exitCode=0 Oct 31 00:30:06.028725 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:06.028627 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-957019-76d548dcf-bbqz5" Oct 31 00:30:06.028725 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:06.028642 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-957019-76d548dcf-bbqz5" event={"ID":"358e61e1-8b72-4d73-b3ec-9016ebe766df","Type":"ContainerDied","Data":"ac2ec1840a36d20ada7e088ed1f12b11a9b7fbdfd9a9ef229d46de9154756250"} Oct 31 00:30:06.028725 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:06.028690 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-957019-76d548dcf-bbqz5" event={"ID":"358e61e1-8b72-4d73-b3ec-9016ebe766df","Type":"ContainerDied","Data":"5dd73fd4e823bab43538738cb06e02c89e1bac3dcaf347b874a023d039689873"} Oct 31 00:30:06.039131 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:06.039108 4927 scope.go:122] "RemoveContainer" containerID="ac2ec1840a36d20ada7e088ed1f12b11a9b7fbdfd9a9ef229d46de9154756250" Oct 31 00:30:06.049972 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:06.049819 4927 scope.go:122] "RemoveContainer" containerID="ac2ec1840a36d20ada7e088ed1f12b11a9b7fbdfd9a9ef229d46de9154756250" Oct 31 00:30:06.050167 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:30:06.050147 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac2ec1840a36d20ada7e088ed1f12b11a9b7fbdfd9a9ef229d46de9154756250\": container with ID starting with ac2ec1840a36d20ada7e088ed1f12b11a9b7fbdfd9a9ef229d46de9154756250 not found: ID does not exist" containerID="ac2ec1840a36d20ada7e088ed1f12b11a9b7fbdfd9a9ef229d46de9154756250" Oct 31 00:30:06.050221 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:06.050176 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac2ec1840a36d20ada7e088ed1f12b11a9b7fbdfd9a9ef229d46de9154756250"} err="failed to get container status \"ac2ec1840a36d20ada7e088ed1f12b11a9b7fbdfd9a9ef229d46de9154756250\": rpc error: code = NotFound desc = could not find container \"ac2ec1840a36d20ada7e088ed1f12b11a9b7fbdfd9a9ef229d46de9154756250\": container with ID starting with ac2ec1840a36d20ada7e088ed1f12b11a9b7fbdfd9a9ef229d46de9154756250 not found: ID does not exist" Oct 31 00:30:06.197642 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:06.197606 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-957019-76d548dcf-bbqz5"] Oct 31 00:30:06.248911 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:06.248880 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-957019-76d548dcf-bbqz5"] Oct 31 00:30:06.316587 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:06.316552 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-957019-788c577f9b-2shs7"] Oct 31 00:30:06.321138 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:06.321106 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-957019-788c577f9b-2shs7"] Oct 31 00:30:07.088042 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:07.087948 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="358e61e1-8b72-4d73-b3ec-9016ebe766df" path="/var/lib/kubelet/pods/358e61e1-8b72-4d73-b3ec-9016ebe766df/volumes" Oct 31 00:30:07.088586 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:07.088562 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed77f29b-ed8c-428a-a8f7-1c55892d64a8" path="/var/lib/kubelet/pods/ed77f29b-ed8c-428a-a8f7-1c55892d64a8/volumes" Oct 31 00:30:08.580808 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:08.580770 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:30:13.208603 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:13.208562 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:30:14.079447 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:14.079414 4927 scope.go:122] "RemoveContainer" containerID="ec35e77001edfa2c1882e272a20beaa0c835c081a5343b86ddd2cc72658c3e04" Oct 31 00:30:14.079629 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:14.079521 4927 scope.go:122] "RemoveContainer" containerID="3b822201af9d57d9c63fc4e2805ef459c0732e7ae6de0ebd8906d4119fb8c07e" Oct 31 00:30:14.615795 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:14.615701 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:30:15.104047 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:15.104015 4927 generic.go:356] "Generic (PLEG): container finished" podID="b75ee4b6-1c0b-4a64-a4a9-14da52d5f7db" containerID="7bc59ba55e2a87465225c41fdbafad571b9e88095081e5d72ccaf3a5f386cfa7" exitCode=60 Oct 31 00:30:15.104261 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:15.104087 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ddf61a-29890110-rfdvn" event={"ID":"b75ee4b6-1c0b-4a64-a4a9-14da52d5f7db","Type":"ContainerDied","Data":"7bc59ba55e2a87465225c41fdbafad571b9e88095081e5d72ccaf3a5f386cfa7"} Oct 31 00:30:15.104261 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:15.104142 4927 scope.go:122] "RemoveContainer" containerID="ec35e77001edfa2c1882e272a20beaa0c835c081a5343b86ddd2cc72658c3e04" Oct 31 00:30:15.104572 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:15.104552 4927 scope.go:122] "RemoveContainer" containerID="7bc59ba55e2a87465225c41fdbafad571b9e88095081e5d72ccaf3a5f386cfa7" Oct 31 00:30:15.104865 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:30:15.104836 4927 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-ddf61a-29890110-rfdvn_odh-ai-gateway-infra(b75ee4b6-1c0b-4a64-a4a9-14da52d5f7db)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ddf61a-29890110-rfdvn" podUID="b75ee4b6-1c0b-4a64-a4a9-14da52d5f7db" Oct 31 00:30:15.106108 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:15.106088 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-957019-29890110-bpqlv" event={"ID":"0fda0711-f23d-409b-af43-23d6bbb8aac7","Type":"ContainerStarted","Data":"1fa14de6f339ce02988aa68b23310e5bd94bb48ca94e9e25e9769d3ef9ebc2e2"} Oct 31 00:30:15.343797 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:15.343691 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-957019-29890110-bpqlv" podStartSLOduration=15.343674034 podStartE2EDuration="15.343674034s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:30:15.341358044 +0000 UTC m=+2928.377688867" watchObservedRunningTime="2026-10-31 00:30:15.343674034 +0000 UTC m=+2928.380004861" Oct 31 00:30:16.079260 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:16.079228 4927 scope.go:122] "RemoveContainer" containerID="09ba4e35a398de95d41917b071e49917cb5c6c79a34817f133eb5ddf9c70ab35" Oct 31 00:30:16.119229 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:16.119187 4927 generic.go:356] "Generic (PLEG): container finished" podID="0fda0711-f23d-409b-af43-23d6bbb8aac7" containerID="1fa14de6f339ce02988aa68b23310e5bd94bb48ca94e9e25e9769d3ef9ebc2e2" exitCode=60 Oct 31 00:30:16.119387 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:16.119251 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-957019-29890110-bpqlv" event={"ID":"0fda0711-f23d-409b-af43-23d6bbb8aac7","Type":"ContainerDied","Data":"1fa14de6f339ce02988aa68b23310e5bd94bb48ca94e9e25e9769d3ef9ebc2e2"} Oct 31 00:30:16.119387 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:16.119303 4927 scope.go:122] "RemoveContainer" containerID="3b822201af9d57d9c63fc4e2805ef459c0732e7ae6de0ebd8906d4119fb8c07e" Oct 31 00:30:16.119695 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:16.119677 4927 scope.go:122] "RemoveContainer" containerID="1fa14de6f339ce02988aa68b23310e5bd94bb48ca94e9e25e9769d3ef9ebc2e2" Oct 31 00:30:16.119970 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:30:16.119947 4927 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-957019-29890110-bpqlv_odh-ai-gateway-infra(0fda0711-f23d-409b-af43-23d6bbb8aac7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-957019-29890110-bpqlv" podUID="0fda0711-f23d-409b-af43-23d6bbb8aac7" Oct 31 00:30:16.236982 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:16.236950 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ddf61a-29890110-rfdvn"] Oct 31 00:30:16.371597 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:16.371435 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-957019-29890110-bpqlv"] Oct 31 00:30:16.414047 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:16.414019 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ddf61a-29890110-rfdvn" Oct 31 00:30:16.554192 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:16.554158 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b75ee4b6-1c0b-4a64-a4a9-14da52d5f7db-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b75ee4b6-1c0b-4a64-a4a9-14da52d5f7db-maas-api-service-ca\") pod \"b75ee4b6-1c0b-4a64-a4a9-14da52d5f7db\" (UID: \"b75ee4b6-1c0b-4a64-a4a9-14da52d5f7db\") " Oct 31 00:30:16.554396 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:16.554287 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b75ee4b6-1c0b-4a64-a4a9-14da52d5f7db-kube-api-access-9h7nm\" (UniqueName: \"kubernetes.io/projected/b75ee4b6-1c0b-4a64-a4a9-14da52d5f7db-kube-api-access-9h7nm\") pod \"b75ee4b6-1c0b-4a64-a4a9-14da52d5f7db\" (UID: \"b75ee4b6-1c0b-4a64-a4a9-14da52d5f7db\") " Oct 31 00:30:16.554597 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:16.554569 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b75ee4b6-1c0b-4a64-a4a9-14da52d5f7db-maas-api-service-ca" pod "b75ee4b6-1c0b-4a64-a4a9-14da52d5f7db" (UID: "b75ee4b6-1c0b-4a64-a4a9-14da52d5f7db"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:30:16.554794 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:16.554775 4927 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b75ee4b6-1c0b-4a64-a4a9-14da52d5f7db-maas-api-service-ca\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:16.556617 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:16.556580 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b75ee4b6-1c0b-4a64-a4a9-14da52d5f7db-kube-api-access-9h7nm" pod "b75ee4b6-1c0b-4a64-a4a9-14da52d5f7db" (UID: "b75ee4b6-1c0b-4a64-a4a9-14da52d5f7db"). InnerVolumeSpecName "kube-api-access-9h7nm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:30:16.655870 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:16.655788 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9h7nm\" (UniqueName: \"kubernetes.io/projected/b75ee4b6-1c0b-4a64-a4a9-14da52d5f7db-kube-api-access-9h7nm\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:17.127137 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:17.127110 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ddf61a-29890110-rfdvn" Oct 31 00:30:17.127137 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:17.127125 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ddf61a-29890110-rfdvn" event={"ID":"b75ee4b6-1c0b-4a64-a4a9-14da52d5f7db","Type":"ContainerDied","Data":"e7b00950900a25f049646a224df8f132a1405bc73dddee0fe4883c0bc772a701"} Oct 31 00:30:17.127667 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:17.127194 4927 scope.go:122] "RemoveContainer" containerID="7bc59ba55e2a87465225c41fdbafad571b9e88095081e5d72ccaf3a5f386cfa7" Oct 31 00:30:17.131097 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:17.131070 4927 generic.go:356] "Generic (PLEG): container finished" podID="8b01ad9a-b63a-40ea-9dfb-4ea15d74bc50" containerID="998e8ea34fe36b3bb509ade743e9a39e3fe83941577dbe35dd446c0c1257644a" exitCode=60 Oct 31 00:30:17.131203 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:17.131107 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-46b586-29890110-mvw2k" event={"ID":"8b01ad9a-b63a-40ea-9dfb-4ea15d74bc50","Type":"ContainerDied","Data":"998e8ea34fe36b3bb509ade743e9a39e3fe83941577dbe35dd446c0c1257644a"} Oct 31 00:30:17.131532 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:17.131514 4927 scope.go:122] "RemoveContainer" containerID="998e8ea34fe36b3bb509ade743e9a39e3fe83941577dbe35dd446c0c1257644a" Oct 31 00:30:17.131786 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:30:17.131767 4927 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-46b586-29890110-mvw2k_odh-ai-gateway-infra(8b01ad9a-b63a-40ea-9dfb-4ea15d74bc50)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-46b586-29890110-mvw2k" podUID="8b01ad9a-b63a-40ea-9dfb-4ea15d74bc50" Oct 31 00:30:17.137960 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:17.137941 4927 scope.go:122] "RemoveContainer" containerID="09ba4e35a398de95d41917b071e49917cb5c6c79a34817f133eb5ddf9c70ab35" Oct 31 00:30:17.164209 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:17.161901 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-957019-77d447ff94-88797"] Oct 31 00:30:17.164209 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:17.162138 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-957019-77d447ff94-88797" podUID="08cf4b69-cb92-439b-bd2f-66734dc9f641" containerName="maas-api" containerID="cri-o://3e33d5b602b753df19917df823673000655af48d843e6dcc6dbb908e9036f161" gracePeriod=30 Oct 31 00:30:17.267001 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:17.266966 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:30:17.285002 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:17.284976 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-957019-29890110-bpqlv" Oct 31 00:30:17.364636 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:17.364603 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0fda0711-f23d-409b-af43-23d6bbb8aac7-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0fda0711-f23d-409b-af43-23d6bbb8aac7-maas-api-service-ca\") pod \"0fda0711-f23d-409b-af43-23d6bbb8aac7\" (UID: \"0fda0711-f23d-409b-af43-23d6bbb8aac7\") " Oct 31 00:30:17.364805 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:17.364650 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0fda0711-f23d-409b-af43-23d6bbb8aac7-kube-api-access-5q77b\" (UniqueName: \"kubernetes.io/projected/0fda0711-f23d-409b-af43-23d6bbb8aac7-kube-api-access-5q77b\") pod \"0fda0711-f23d-409b-af43-23d6bbb8aac7\" (UID: \"0fda0711-f23d-409b-af43-23d6bbb8aac7\") " Oct 31 00:30:17.364952 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:17.364928 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0fda0711-f23d-409b-af43-23d6bbb8aac7-maas-api-service-ca" pod "0fda0711-f23d-409b-af43-23d6bbb8aac7" (UID: "0fda0711-f23d-409b-af43-23d6bbb8aac7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:30:17.365120 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:17.365102 4927 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0fda0711-f23d-409b-af43-23d6bbb8aac7-maas-api-service-ca\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:17.367297 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:17.367275 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0fda0711-f23d-409b-af43-23d6bbb8aac7-kube-api-access-5q77b" pod "0fda0711-f23d-409b-af43-23d6bbb8aac7" (UID: "0fda0711-f23d-409b-af43-23d6bbb8aac7"). InnerVolumeSpecName "kube-api-access-5q77b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:30:17.466123 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:17.466084 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5q77b\" (UniqueName: \"kubernetes.io/projected/0fda0711-f23d-409b-af43-23d6bbb8aac7-kube-api-access-5q77b\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:17.482219 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:17.482183 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-957019-77d447ff94-88797" Oct 31 00:30:17.567483 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:17.567440 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/08cf4b69-cb92-439b-bd2f-66734dc9f641-metrics-certs\" (UniqueName: \"kubernetes.io/secret/08cf4b69-cb92-439b-bd2f-66734dc9f641-metrics-certs\") pod \"08cf4b69-cb92-439b-bd2f-66734dc9f641\" (UID: \"08cf4b69-cb92-439b-bd2f-66734dc9f641\") " Oct 31 00:30:17.567680 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:17.567504 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/08cf4b69-cb92-439b-bd2f-66734dc9f641-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/08cf4b69-cb92-439b-bd2f-66734dc9f641-openshift-service-ca\") pod \"08cf4b69-cb92-439b-bd2f-66734dc9f641\" (UID: \"08cf4b69-cb92-439b-bd2f-66734dc9f641\") " Oct 31 00:30:17.567680 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:17.567558 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08cf4b69-cb92-439b-bd2f-66734dc9f641-kube-api-access-f5pvj\" (UniqueName: \"kubernetes.io/projected/08cf4b69-cb92-439b-bd2f-66734dc9f641-kube-api-access-f5pvj\") pod \"08cf4b69-cb92-439b-bd2f-66734dc9f641\" (UID: \"08cf4b69-cb92-439b-bd2f-66734dc9f641\") " Oct 31 00:30:17.567843 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:17.567723 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/08cf4b69-cb92-439b-bd2f-66734dc9f641-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/08cf4b69-cb92-439b-bd2f-66734dc9f641-maas-api-tls\") pod \"08cf4b69-cb92-439b-bd2f-66734dc9f641\" (UID: \"08cf4b69-cb92-439b-bd2f-66734dc9f641\") " Oct 31 00:30:17.567931 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:17.567904 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08cf4b69-cb92-439b-bd2f-66734dc9f641-openshift-service-ca" pod "08cf4b69-cb92-439b-bd2f-66734dc9f641" (UID: "08cf4b69-cb92-439b-bd2f-66734dc9f641"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:30:17.568063 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:17.568046 4927 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/08cf4b69-cb92-439b-bd2f-66734dc9f641-openshift-service-ca\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:17.570244 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:17.570200 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/08cf4b69-cb92-439b-bd2f-66734dc9f641-maas-api-tls" pod "08cf4b69-cb92-439b-bd2f-66734dc9f641" (UID: "08cf4b69-cb92-439b-bd2f-66734dc9f641"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:30:17.570591 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:17.570563 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/08cf4b69-cb92-439b-bd2f-66734dc9f641-metrics-certs" pod "08cf4b69-cb92-439b-bd2f-66734dc9f641" (UID: "08cf4b69-cb92-439b-bd2f-66734dc9f641"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:30:17.571070 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:17.571033 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08cf4b69-cb92-439b-bd2f-66734dc9f641-kube-api-access-f5pvj" pod "08cf4b69-cb92-439b-bd2f-66734dc9f641" (UID: "08cf4b69-cb92-439b-bd2f-66734dc9f641"). InnerVolumeSpecName "kube-api-access-f5pvj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:30:17.668925 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:17.668824 4927 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/08cf4b69-cb92-439b-bd2f-66734dc9f641-maas-api-tls\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:17.668925 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:17.668860 4927 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/08cf4b69-cb92-439b-bd2f-66734dc9f641-metrics-certs\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:17.668925 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:17.668893 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f5pvj\" (UniqueName: \"kubernetes.io/projected/08cf4b69-cb92-439b-bd2f-66734dc9f641-kube-api-access-f5pvj\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:17.670656 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:17.670598 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ddf61a-29890110-rfdvn"] Oct 31 00:30:17.673569 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:17.673535 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ddf61a-29890110-rfdvn"] Oct 31 00:30:18.141485 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:18.141450 4927 generic.go:356] "Generic (PLEG): container finished" podID="08cf4b69-cb92-439b-bd2f-66734dc9f641" containerID="3e33d5b602b753df19917df823673000655af48d843e6dcc6dbb908e9036f161" exitCode=0 Oct 31 00:30:18.141982 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:18.141511 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-957019-77d447ff94-88797" Oct 31 00:30:18.141982 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:18.141542 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-957019-77d447ff94-88797" event={"ID":"08cf4b69-cb92-439b-bd2f-66734dc9f641","Type":"ContainerDied","Data":"3e33d5b602b753df19917df823673000655af48d843e6dcc6dbb908e9036f161"} Oct 31 00:30:18.141982 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:18.141611 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-957019-77d447ff94-88797" event={"ID":"08cf4b69-cb92-439b-bd2f-66734dc9f641","Type":"ContainerDied","Data":"2678ad28b757546d7dad3d62f58fb88745e4eff0b7ddabb21c5282d3c0222a12"} Oct 31 00:30:18.141982 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:18.141645 4927 scope.go:122] "RemoveContainer" containerID="3e33d5b602b753df19917df823673000655af48d843e6dcc6dbb908e9036f161" Oct 31 00:30:18.144306 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:18.144288 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-957019-29890110-bpqlv" Oct 31 00:30:18.144453 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:18.144320 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-957019-29890110-bpqlv" event={"ID":"0fda0711-f23d-409b-af43-23d6bbb8aac7","Type":"ContainerDied","Data":"1e48883c0794ee022ddc92b10cad3541ff48467f9eb9dd9c3e974318628fc5e6"} Oct 31 00:30:18.155085 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:18.155067 4927 scope.go:122] "RemoveContainer" containerID="3e33d5b602b753df19917df823673000655af48d843e6dcc6dbb908e9036f161" Oct 31 00:30:18.155428 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:30:18.155406 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e33d5b602b753df19917df823673000655af48d843e6dcc6dbb908e9036f161\": container with ID starting with 3e33d5b602b753df19917df823673000655af48d843e6dcc6dbb908e9036f161 not found: ID does not exist" containerID="3e33d5b602b753df19917df823673000655af48d843e6dcc6dbb908e9036f161" Oct 31 00:30:18.155500 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:18.155435 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e33d5b602b753df19917df823673000655af48d843e6dcc6dbb908e9036f161"} err="failed to get container status \"3e33d5b602b753df19917df823673000655af48d843e6dcc6dbb908e9036f161\": rpc error: code = NotFound desc = could not find container \"3e33d5b602b753df19917df823673000655af48d843e6dcc6dbb908e9036f161\": container with ID starting with 3e33d5b602b753df19917df823673000655af48d843e6dcc6dbb908e9036f161 not found: ID does not exist" Oct 31 00:30:18.155500 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:18.155459 4927 scope.go:122] "RemoveContainer" containerID="1fa14de6f339ce02988aa68b23310e5bd94bb48ca94e9e25e9769d3ef9ebc2e2" Oct 31 00:30:18.314803 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:18.314768 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-957019-77d447ff94-88797"] Oct 31 00:30:18.319442 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:18.319402 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-957019-77d447ff94-88797"] Oct 31 00:30:18.433789 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:18.433756 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-957019-29890110-bpqlv"] Oct 31 00:30:18.438373 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:18.438345 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-957019-29890110-bpqlv"] Oct 31 00:30:18.562096 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:18.562069 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-46b586-29890110-mvw2k"] Oct 31 00:30:18.710446 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:18.710419 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-46b586-29890110-mvw2k" Oct 31 00:30:18.773746 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:18.773712 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:30:18.781710 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:18.781685 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b01ad9a-b63a-40ea-9dfb-4ea15d74bc50-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b01ad9a-b63a-40ea-9dfb-4ea15d74bc50-maas-api-service-ca\") pod \"8b01ad9a-b63a-40ea-9dfb-4ea15d74bc50\" (UID: \"8b01ad9a-b63a-40ea-9dfb-4ea15d74bc50\") " Oct 31 00:30:18.781824 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:18.781786 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b01ad9a-b63a-40ea-9dfb-4ea15d74bc50-kube-api-access-nssm5\" (UniqueName: \"kubernetes.io/projected/8b01ad9a-b63a-40ea-9dfb-4ea15d74bc50-kube-api-access-nssm5\") pod \"8b01ad9a-b63a-40ea-9dfb-4ea15d74bc50\" (UID: \"8b01ad9a-b63a-40ea-9dfb-4ea15d74bc50\") " Oct 31 00:30:18.782095 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:18.782070 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b01ad9a-b63a-40ea-9dfb-4ea15d74bc50-maas-api-service-ca" pod "8b01ad9a-b63a-40ea-9dfb-4ea15d74bc50" (UID: "8b01ad9a-b63a-40ea-9dfb-4ea15d74bc50"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:30:18.783812 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:18.783792 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b01ad9a-b63a-40ea-9dfb-4ea15d74bc50-kube-api-access-nssm5" pod "8b01ad9a-b63a-40ea-9dfb-4ea15d74bc50" (UID: "8b01ad9a-b63a-40ea-9dfb-4ea15d74bc50"). InnerVolumeSpecName "kube-api-access-nssm5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:30:18.883117 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:18.883080 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nssm5\" (UniqueName: \"kubernetes.io/projected/8b01ad9a-b63a-40ea-9dfb-4ea15d74bc50-kube-api-access-nssm5\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:18.883117 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:18.883107 4927 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b01ad9a-b63a-40ea-9dfb-4ea15d74bc50-maas-api-service-ca\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:19.011044 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:19.010960 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:30:19.093384 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:19.093322 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="08cf4b69-cb92-439b-bd2f-66734dc9f641" path="/var/lib/kubelet/pods/08cf4b69-cb92-439b-bd2f-66734dc9f641/volumes" Oct 31 00:30:19.093906 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:19.093890 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0fda0711-f23d-409b-af43-23d6bbb8aac7" path="/var/lib/kubelet/pods/0fda0711-f23d-409b-af43-23d6bbb8aac7/volumes" Oct 31 00:30:19.094268 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:19.094255 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b75ee4b6-1c0b-4a64-a4a9-14da52d5f7db" path="/var/lib/kubelet/pods/b75ee4b6-1c0b-4a64-a4a9-14da52d5f7db/volumes" Oct 31 00:30:19.152791 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:19.152757 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-46b586-29890110-mvw2k" Oct 31 00:30:19.153234 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:19.152784 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-46b586-29890110-mvw2k" event={"ID":"8b01ad9a-b63a-40ea-9dfb-4ea15d74bc50","Type":"ContainerDied","Data":"d344cf3643627c8116598e20a629b5ad4ef5193603b9a2455a40093ffead5a2b"} Oct 31 00:30:19.153234 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:19.152847 4927 scope.go:122] "RemoveContainer" containerID="998e8ea34fe36b3bb509ade743e9a39e3fe83941577dbe35dd446c0c1257644a" Oct 31 00:30:19.170584 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:19.170560 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:30:19.249000 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:19.248968 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-957019-tcgnw"] Oct 31 00:30:19.253548 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:19.253519 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-957019-tcgnw"] Oct 31 00:30:19.322791 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:19.322715 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-46b586-29890110-mvw2k"] Oct 31 00:30:19.327465 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:19.327441 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-46b586-29890110-mvw2k"] Oct 31 00:30:19.678479 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:30:19.678438 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sqv69" containerName="extractor" Oct 31 00:30:19.716500 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:19.716468 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:30:21.087826 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:21.087787 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="68687d7b-81bd-4129-af1a-be643b52aa58" path="/var/lib/kubelet/pods/68687d7b-81bd-4129-af1a-be643b52aa58/volumes" Oct 31 00:30:21.088437 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:21.088404 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b01ad9a-b63a-40ea-9dfb-4ea15d74bc50" path="/var/lib/kubelet/pods/8b01ad9a-b63a-40ea-9dfb-4ea15d74bc50/volumes" Oct 31 00:30:23.848176 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:23.848139 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:30:25.903578 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:25.903547 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:30:26.105894 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:26.105838 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:30:26.310527 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:26.310495 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-957019-openshift-default-59bc4d44f9-8xtcd"] Oct 31 00:30:26.310774 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:26.310751 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w0-957019-openshift-default-59bc4d44f9-8xtcd" podUID="4b623edb-855c-44b8-a0c1-abbd33925bd4" containerName="istio-proxy" containerID="cri-o://83c89c92e6e55d4905b445efe29c501f62cd664f6c65d4ea0bcba420518d3191" gracePeriod=30 Oct 31 00:30:26.340221 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:26.340186 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:30:26.818226 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:26.818185 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:30:27.376402 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:27.376353 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-46b586-bsx74"] Oct 31 00:30:27.377189 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:27.377166 4927 cpu_manager.go:414] "containerMap: removing container" podUID="0fda0711-f23d-409b-af43-23d6bbb8aac7" containerName="cleanup" Oct 31 00:30:27.377296 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:27.377189 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0fda0711-f23d-409b-af43-23d6bbb8aac7" containerName="cleanup" Oct 31 00:30:27.377296 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:27.377204 4927 cpu_manager.go:414] "containerMap: removing container" podUID="b75ee4b6-1c0b-4a64-a4a9-14da52d5f7db" containerName="cleanup" Oct 31 00:30:27.377296 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:27.377212 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b75ee4b6-1c0b-4a64-a4a9-14da52d5f7db" containerName="cleanup" Oct 31 00:30:27.377296 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:27.377232 4927 cpu_manager.go:414] "containerMap: removing container" podUID="0fda0711-f23d-409b-af43-23d6bbb8aac7" containerName="cleanup" Oct 31 00:30:27.377296 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:27.377240 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0fda0711-f23d-409b-af43-23d6bbb8aac7" containerName="cleanup" Oct 31 00:30:27.377296 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:27.377265 4927 cpu_manager.go:414] "containerMap: removing container" podUID="0fda0711-f23d-409b-af43-23d6bbb8aac7" containerName="cleanup" Oct 31 00:30:27.377296 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:27.377273 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0fda0711-f23d-409b-af43-23d6bbb8aac7" containerName="cleanup" Oct 31 00:30:27.377698 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:27.377306 4927 cpu_manager.go:414] "containerMap: removing container" podUID="8b01ad9a-b63a-40ea-9dfb-4ea15d74bc50" containerName="cleanup" Oct 31 00:30:27.377698 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:27.377314 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b01ad9a-b63a-40ea-9dfb-4ea15d74bc50" containerName="cleanup" Oct 31 00:30:27.377698 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:27.377326 4927 cpu_manager.go:414] "containerMap: removing container" podUID="8b01ad9a-b63a-40ea-9dfb-4ea15d74bc50" containerName="cleanup" Oct 31 00:30:27.377698 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:27.377349 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b01ad9a-b63a-40ea-9dfb-4ea15d74bc50" containerName="cleanup" Oct 31 00:30:27.377698 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:27.377359 4927 cpu_manager.go:414] "containerMap: removing container" podUID="358e61e1-8b72-4d73-b3ec-9016ebe766df" containerName="payload-pre-processing" Oct 31 00:30:27.377698 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:27.377366 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="358e61e1-8b72-4d73-b3ec-9016ebe766df" containerName="payload-pre-processing" Oct 31 00:30:27.377698 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:27.377389 4927 cpu_manager.go:414] "containerMap: removing container" podUID="b75ee4b6-1c0b-4a64-a4a9-14da52d5f7db" containerName="cleanup" Oct 31 00:30:27.377698 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:27.377396 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b75ee4b6-1c0b-4a64-a4a9-14da52d5f7db" containerName="cleanup" Oct 31 00:30:27.377698 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:27.377412 4927 cpu_manager.go:414] "containerMap: removing container" podUID="ed77f29b-ed8c-428a-a8f7-1c55892d64a8" containerName="payload-processing" Oct 31 00:30:27.377698 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:27.377420 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed77f29b-ed8c-428a-a8f7-1c55892d64a8" containerName="payload-processing" Oct 31 00:30:27.377698 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:27.377431 4927 cpu_manager.go:414] "containerMap: removing container" podUID="08cf4b69-cb92-439b-bd2f-66734dc9f641" containerName="maas-api" Oct 31 00:30:27.377698 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:27.377437 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08cf4b69-cb92-439b-bd2f-66734dc9f641" containerName="maas-api" Oct 31 00:30:27.377698 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:27.377564 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="0fda0711-f23d-409b-af43-23d6bbb8aac7" containerName="cleanup" Oct 31 00:30:27.377698 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:27.377579 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="b75ee4b6-1c0b-4a64-a4a9-14da52d5f7db" containerName="cleanup" Oct 31 00:30:27.377698 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:27.377594 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b01ad9a-b63a-40ea-9dfb-4ea15d74bc50" containerName="cleanup" Oct 31 00:30:27.377698 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:27.377603 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b01ad9a-b63a-40ea-9dfb-4ea15d74bc50" containerName="cleanup" Oct 31 00:30:27.377698 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:27.377611 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="358e61e1-8b72-4d73-b3ec-9016ebe766df" containerName="payload-pre-processing" Oct 31 00:30:27.377698 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:27.377623 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed77f29b-ed8c-428a-a8f7-1c55892d64a8" containerName="payload-processing" Oct 31 00:30:27.377698 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:27.377632 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="08cf4b69-cb92-439b-bd2f-66734dc9f641" containerName="maas-api" Oct 31 00:30:27.377698 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:27.377643 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="0fda0711-f23d-409b-af43-23d6bbb8aac7" containerName="cleanup" Oct 31 00:30:27.377698 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:27.377653 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="b75ee4b6-1c0b-4a64-a4a9-14da52d5f7db" containerName="cleanup" Oct 31 00:30:27.381519 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:27.381496 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-46b586-bsx74" Oct 31 00:30:27.443424 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:27.443392 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-46b586-bsx74"] Oct 31 00:30:27.447278 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:27.447251 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4fdzs\"" Oct 31 00:30:27.447278 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:27.447258 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:30:27.478428 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:27.478392 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6421ddc-b3e6-47d2-960e-b9716f234b3a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-46b586-bsx74\" (UID: \"c6421ddc-b3e6-47d2-960e-b9716f234b3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-46b586-bsx74" Oct 31 00:30:27.478591 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:27.478523 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7f4z\" (UniqueName: \"kubernetes.io/projected/c6421ddc-b3e6-47d2-960e-b9716f234b3a-kube-api-access-t7f4z\") pod \"maas-api-revoke-keys-e2e-worker-w0-46b586-bsx74\" (UID: \"c6421ddc-b3e6-47d2-960e-b9716f234b3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-46b586-bsx74" Oct 31 00:30:27.478591 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:27.478572 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6421ddc-b3e6-47d2-960e-b9716f234b3a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-46b586-bsx74\" (UID: \"c6421ddc-b3e6-47d2-960e-b9716f234b3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-46b586-bsx74" Oct 31 00:30:27.579170 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:27.579136 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7f4z\" (UniqueName: \"kubernetes.io/projected/c6421ddc-b3e6-47d2-960e-b9716f234b3a-kube-api-access-t7f4z\") pod \"maas-api-revoke-keys-e2e-worker-w0-46b586-bsx74\" (UID: \"c6421ddc-b3e6-47d2-960e-b9716f234b3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-46b586-bsx74" Oct 31 00:30:27.579396 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:27.579189 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6421ddc-b3e6-47d2-960e-b9716f234b3a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-46b586-bsx74\" (UID: \"c6421ddc-b3e6-47d2-960e-b9716f234b3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-46b586-bsx74" Oct 31 00:30:27.579396 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:27.579301 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6421ddc-b3e6-47d2-960e-b9716f234b3a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-46b586-bsx74\" (UID: \"c6421ddc-b3e6-47d2-960e-b9716f234b3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-46b586-bsx74" Oct 31 00:30:27.579954 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:27.579917 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6421ddc-b3e6-47d2-960e-b9716f234b3a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-46b586-bsx74\" (UID: \"c6421ddc-b3e6-47d2-960e-b9716f234b3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-46b586-bsx74" Oct 31 00:30:27.581637 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:27.581616 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6421ddc-b3e6-47d2-960e-b9716f234b3a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-46b586-bsx74\" (UID: \"c6421ddc-b3e6-47d2-960e-b9716f234b3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-46b586-bsx74" Oct 31 00:30:27.642964 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:27.642902 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7f4z\" (UniqueName: \"kubernetes.io/projected/c6421ddc-b3e6-47d2-960e-b9716f234b3a-kube-api-access-t7f4z\") pod \"maas-api-revoke-keys-e2e-worker-w0-46b586-bsx74\" (UID: \"c6421ddc-b3e6-47d2-960e-b9716f234b3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-46b586-bsx74" Oct 31 00:30:27.697830 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:27.697799 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-46b586-bsx74" Oct 31 00:30:28.081196 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:28.081170 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-46b586-bsx74"] Oct 31 00:30:28.082832 ip-10-0-141-180 kubenswrapper[4927]: W1031 00:30:28.082799 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6421ddc_b3e6_47d2_960e_b9716f234b3a.slice/crio-4a1ce3f02b1b0019e63a9dcac11bc881b41af184fa1b5859342d0dccc5a3d5cd WatchSource:0}: Error finding container 4a1ce3f02b1b0019e63a9dcac11bc881b41af184fa1b5859342d0dccc5a3d5cd: Status 404 returned error can't find the container with id 4a1ce3f02b1b0019e63a9dcac11bc881b41af184fa1b5859342d0dccc5a3d5cd Oct 31 00:30:28.220593 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:28.220559 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-46b586-bsx74" event={"ID":"c6421ddc-b3e6-47d2-960e-b9716f234b3a","Type":"ContainerStarted","Data":"4a1ce3f02b1b0019e63a9dcac11bc881b41af184fa1b5859342d0dccc5a3d5cd"} Oct 31 00:30:28.763715 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:28.763626 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:30:29.234423 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:29.234378 4927 generic.go:356] "Generic (PLEG): container finished" podID="c6421ddc-b3e6-47d2-960e-b9716f234b3a" containerID="0d80369c859d795b1b31f3bf521ebbd098d7394a31875fdcf0671b79df513690" exitCode=0 Oct 31 00:30:29.234643 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:29.234567 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-46b586-bsx74" event={"ID":"c6421ddc-b3e6-47d2-960e-b9716f234b3a","Type":"ContainerDied","Data":"0d80369c859d795b1b31f3bf521ebbd098d7394a31875fdcf0671b79df513690"} Oct 31 00:30:30.394887 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:30.394861 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-46b586-bsx74" Oct 31 00:30:30.505900 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:30.505813 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c6421ddc-b3e6-47d2-960e-b9716f234b3a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6421ddc-b3e6-47d2-960e-b9716f234b3a-maas-api-xks-ca\") pod \"c6421ddc-b3e6-47d2-960e-b9716f234b3a\" (UID: \"c6421ddc-b3e6-47d2-960e-b9716f234b3a\") " Oct 31 00:30:30.506088 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:30.505967 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6421ddc-b3e6-47d2-960e-b9716f234b3a-kube-api-access-t7f4z\" (UniqueName: \"kubernetes.io/projected/c6421ddc-b3e6-47d2-960e-b9716f234b3a-kube-api-access-t7f4z\") pod \"c6421ddc-b3e6-47d2-960e-b9716f234b3a\" (UID: \"c6421ddc-b3e6-47d2-960e-b9716f234b3a\") " Oct 31 00:30:30.506088 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:30.506013 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c6421ddc-b3e6-47d2-960e-b9716f234b3a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6421ddc-b3e6-47d2-960e-b9716f234b3a-maas-api-openshift-ca\") pod \"c6421ddc-b3e6-47d2-960e-b9716f234b3a\" (UID: \"c6421ddc-b3e6-47d2-960e-b9716f234b3a\") " Oct 31 00:30:30.506477 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:30.506445 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6421ddc-b3e6-47d2-960e-b9716f234b3a-maas-api-openshift-ca" pod "c6421ddc-b3e6-47d2-960e-b9716f234b3a" (UID: "c6421ddc-b3e6-47d2-960e-b9716f234b3a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:30:30.508021 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:30.507996 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6421ddc-b3e6-47d2-960e-b9716f234b3a-kube-api-access-t7f4z" pod "c6421ddc-b3e6-47d2-960e-b9716f234b3a" (UID: "c6421ddc-b3e6-47d2-960e-b9716f234b3a"). InnerVolumeSpecName "kube-api-access-t7f4z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:30:30.508151 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:30.508107 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6421ddc-b3e6-47d2-960e-b9716f234b3a-maas-api-xks-ca" pod "c6421ddc-b3e6-47d2-960e-b9716f234b3a" (UID: "c6421ddc-b3e6-47d2-960e-b9716f234b3a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:30:30.607862 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:30.607825 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t7f4z\" (UniqueName: \"kubernetes.io/projected/c6421ddc-b3e6-47d2-960e-b9716f234b3a-kube-api-access-t7f4z\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:30.607862 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:30.607859 4927 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6421ddc-b3e6-47d2-960e-b9716f234b3a-maas-api-openshift-ca\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:30.607862 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:30.607870 4927 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6421ddc-b3e6-47d2-960e-b9716f234b3a-maas-api-xks-ca\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:31.249474 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:31.249439 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-46b586-bsx74" Oct 31 00:30:31.249652 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:31.249470 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-46b586-bsx74" event={"ID":"c6421ddc-b3e6-47d2-960e-b9716f234b3a","Type":"ContainerDied","Data":"4a1ce3f02b1b0019e63a9dcac11bc881b41af184fa1b5859342d0dccc5a3d5cd"} Oct 31 00:30:31.249652 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:31.249523 4927 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4a1ce3f02b1b0019e63a9dcac11bc881b41af184fa1b5859342d0dccc5a3d5cd" Oct 31 00:30:31.581465 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:31.581435 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-957019-openshift-default-59bc4d44f9-8xtcd" Oct 31 00:30:31.722216 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:31.722132 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b623edb-855c-44b8-a0c1-abbd33925bd4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b623edb-855c-44b8-a0c1-abbd33925bd4-workload-certs\") pod \"4b623edb-855c-44b8-a0c1-abbd33925bd4\" (UID: \"4b623edb-855c-44b8-a0c1-abbd33925bd4\") " Oct 31 00:30:31.722391 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:31.722243 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4b623edb-855c-44b8-a0c1-abbd33925bd4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b623edb-855c-44b8-a0c1-abbd33925bd4-istio-podinfo\") pod \"4b623edb-855c-44b8-a0c1-abbd33925bd4\" (UID: \"4b623edb-855c-44b8-a0c1-abbd33925bd4\") " Oct 31 00:30:31.722391 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:31.722317 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b623edb-855c-44b8-a0c1-abbd33925bd4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b623edb-855c-44b8-a0c1-abbd33925bd4-credential-socket\") pod \"4b623edb-855c-44b8-a0c1-abbd33925bd4\" (UID: \"4b623edb-855c-44b8-a0c1-abbd33925bd4\") " Oct 31 00:30:31.722498 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:31.722455 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4b623edb-855c-44b8-a0c1-abbd33925bd4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b623edb-855c-44b8-a0c1-abbd33925bd4-istiod-ca-cert\") pod \"4b623edb-855c-44b8-a0c1-abbd33925bd4\" (UID: \"4b623edb-855c-44b8-a0c1-abbd33925bd4\") " Oct 31 00:30:31.722498 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:31.722474 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b623edb-855c-44b8-a0c1-abbd33925bd4-workload-certs" pod "4b623edb-855c-44b8-a0c1-abbd33925bd4" (UID: "4b623edb-855c-44b8-a0c1-abbd33925bd4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:30:31.722580 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:31.722507 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b623edb-855c-44b8-a0c1-abbd33925bd4-kube-api-access-5vflf\" (UniqueName: \"kubernetes.io/projected/4b623edb-855c-44b8-a0c1-abbd33925bd4-kube-api-access-5vflf\") pod \"4b623edb-855c-44b8-a0c1-abbd33925bd4\" (UID: \"4b623edb-855c-44b8-a0c1-abbd33925bd4\") " Oct 31 00:30:31.722580 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:31.722562 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b623edb-855c-44b8-a0c1-abbd33925bd4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b623edb-855c-44b8-a0c1-abbd33925bd4-istio-data\") pod \"4b623edb-855c-44b8-a0c1-abbd33925bd4\" (UID: \"4b623edb-855c-44b8-a0c1-abbd33925bd4\") " Oct 31 00:30:31.722687 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:31.722588 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b623edb-855c-44b8-a0c1-abbd33925bd4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b623edb-855c-44b8-a0c1-abbd33925bd4-workload-socket\") pod \"4b623edb-855c-44b8-a0c1-abbd33925bd4\" (UID: \"4b623edb-855c-44b8-a0c1-abbd33925bd4\") " Oct 31 00:30:31.722687 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:31.722603 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b623edb-855c-44b8-a0c1-abbd33925bd4-credential-socket" pod "4b623edb-855c-44b8-a0c1-abbd33925bd4" (UID: "4b623edb-855c-44b8-a0c1-abbd33925bd4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:30:31.722687 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:31.722621 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b623edb-855c-44b8-a0c1-abbd33925bd4-istio-token\" (UniqueName: \"kubernetes.io/projected/4b623edb-855c-44b8-a0c1-abbd33925bd4-istio-token\") pod \"4b623edb-855c-44b8-a0c1-abbd33925bd4\" (UID: \"4b623edb-855c-44b8-a0c1-abbd33925bd4\") " Oct 31 00:30:31.722687 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:31.722677 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b623edb-855c-44b8-a0c1-abbd33925bd4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b623edb-855c-44b8-a0c1-abbd33925bd4-istio-envoy\") pod \"4b623edb-855c-44b8-a0c1-abbd33925bd4\" (UID: \"4b623edb-855c-44b8-a0c1-abbd33925bd4\") " Oct 31 00:30:31.722908 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:31.722829 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b623edb-855c-44b8-a0c1-abbd33925bd4-istiod-ca-cert" pod "4b623edb-855c-44b8-a0c1-abbd33925bd4" (UID: "4b623edb-855c-44b8-a0c1-abbd33925bd4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:30:31.723040 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:31.723010 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b623edb-855c-44b8-a0c1-abbd33925bd4-workload-socket" pod "4b623edb-855c-44b8-a0c1-abbd33925bd4" (UID: "4b623edb-855c-44b8-a0c1-abbd33925bd4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:30:31.723040 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:31.722833 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b623edb-855c-44b8-a0c1-abbd33925bd4-istio-data" pod "4b623edb-855c-44b8-a0c1-abbd33925bd4" (UID: "4b623edb-855c-44b8-a0c1-abbd33925bd4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:30:31.723264 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:31.723240 4927 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b623edb-855c-44b8-a0c1-abbd33925bd4-workload-certs\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:31.723379 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:31.723272 4927 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b623edb-855c-44b8-a0c1-abbd33925bd4-credential-socket\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:31.723379 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:31.723288 4927 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b623edb-855c-44b8-a0c1-abbd33925bd4-istiod-ca-cert\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:31.723379 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:31.723303 4927 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b623edb-855c-44b8-a0c1-abbd33925bd4-istio-data\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:31.723379 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:31.723317 4927 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b623edb-855c-44b8-a0c1-abbd33925bd4-workload-socket\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:31.725117 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:31.725086 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4b623edb-855c-44b8-a0c1-abbd33925bd4-istio-podinfo" pod "4b623edb-855c-44b8-a0c1-abbd33925bd4" (UID: "4b623edb-855c-44b8-a0c1-abbd33925bd4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:30:31.725117 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:31.725101 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b623edb-855c-44b8-a0c1-abbd33925bd4-istio-token" pod "4b623edb-855c-44b8-a0c1-abbd33925bd4" (UID: "4b623edb-855c-44b8-a0c1-abbd33925bd4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:30:31.725516 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:31.725494 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b623edb-855c-44b8-a0c1-abbd33925bd4-kube-api-access-5vflf" pod "4b623edb-855c-44b8-a0c1-abbd33925bd4" (UID: "4b623edb-855c-44b8-a0c1-abbd33925bd4"). InnerVolumeSpecName "kube-api-access-5vflf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:30:31.725592 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:31.725538 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b623edb-855c-44b8-a0c1-abbd33925bd4-istio-envoy" pod "4b623edb-855c-44b8-a0c1-abbd33925bd4" (UID: "4b623edb-855c-44b8-a0c1-abbd33925bd4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:30:31.824198 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:31.824166 4927 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b623edb-855c-44b8-a0c1-abbd33925bd4-istio-podinfo\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:31.824198 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:31.824194 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5vflf\" (UniqueName: \"kubernetes.io/projected/4b623edb-855c-44b8-a0c1-abbd33925bd4-kube-api-access-5vflf\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:31.824423 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:31.824207 4927 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b623edb-855c-44b8-a0c1-abbd33925bd4-istio-token\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:31.824423 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:31.824216 4927 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b623edb-855c-44b8-a0c1-abbd33925bd4-istio-envoy\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:31.843273 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:31.843244 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-46b586/simulator-w0-46b586-kserve-67588b8dbd-z8xmn"] Oct 31 00:30:31.964581 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:31.964546 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:30:32.258446 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:32.258409 4927 generic.go:356] "Generic (PLEG): container finished" podID="4b623edb-855c-44b8-a0c1-abbd33925bd4" containerID="83c89c92e6e55d4905b445efe29c501f62cd664f6c65d4ea0bcba420518d3191" exitCode=0 Oct 31 00:30:32.258624 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:32.258464 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-957019-openshift-default-59bc4d44f9-8xtcd" Oct 31 00:30:32.258624 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:32.258489 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-957019-openshift-default-59bc4d44f9-8xtcd" event={"ID":"4b623edb-855c-44b8-a0c1-abbd33925bd4","Type":"ContainerDied","Data":"83c89c92e6e55d4905b445efe29c501f62cd664f6c65d4ea0bcba420518d3191"} Oct 31 00:30:32.258624 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:32.258548 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-957019-openshift-default-59bc4d44f9-8xtcd" event={"ID":"4b623edb-855c-44b8-a0c1-abbd33925bd4","Type":"ContainerDied","Data":"0b96aaed85980a6afd72a71d0a227b0d09800adf26b12ab03c9982553e0767bb"} Oct 31 00:30:32.258624 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:32.258571 4927 scope.go:122] "RemoveContainer" containerID="83c89c92e6e55d4905b445efe29c501f62cd664f6c65d4ea0bcba420518d3191" Oct 31 00:30:32.269592 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:32.269572 4927 scope.go:122] "RemoveContainer" containerID="cdcb32181d652df3402395552e0ae0898b41240322995bcf0f51e7ff004c2707" Oct 31 00:30:32.278807 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:32.278783 4927 scope.go:122] "RemoveContainer" containerID="83c89c92e6e55d4905b445efe29c501f62cd664f6c65d4ea0bcba420518d3191" Oct 31 00:30:32.279068 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:30:32.279048 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"83c89c92e6e55d4905b445efe29c501f62cd664f6c65d4ea0bcba420518d3191\": container with ID starting with 83c89c92e6e55d4905b445efe29c501f62cd664f6c65d4ea0bcba420518d3191 not found: ID does not exist" containerID="83c89c92e6e55d4905b445efe29c501f62cd664f6c65d4ea0bcba420518d3191" Oct 31 00:30:32.279119 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:32.279078 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83c89c92e6e55d4905b445efe29c501f62cd664f6c65d4ea0bcba420518d3191"} err="failed to get container status \"83c89c92e6e55d4905b445efe29c501f62cd664f6c65d4ea0bcba420518d3191\": rpc error: code = NotFound desc = could not find container \"83c89c92e6e55d4905b445efe29c501f62cd664f6c65d4ea0bcba420518d3191\": container with ID starting with 83c89c92e6e55d4905b445efe29c501f62cd664f6c65d4ea0bcba420518d3191 not found: ID does not exist" Oct 31 00:30:32.279119 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:32.279096 4927 scope.go:122] "RemoveContainer" containerID="cdcb32181d652df3402395552e0ae0898b41240322995bcf0f51e7ff004c2707" Oct 31 00:30:32.279363 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:30:32.279346 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cdcb32181d652df3402395552e0ae0898b41240322995bcf0f51e7ff004c2707\": container with ID starting with cdcb32181d652df3402395552e0ae0898b41240322995bcf0f51e7ff004c2707 not found: ID does not exist" containerID="cdcb32181d652df3402395552e0ae0898b41240322995bcf0f51e7ff004c2707" Oct 31 00:30:32.279430 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:32.279366 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cdcb32181d652df3402395552e0ae0898b41240322995bcf0f51e7ff004c2707"} err="failed to get container status \"cdcb32181d652df3402395552e0ae0898b41240322995bcf0f51e7ff004c2707\": rpc error: code = NotFound desc = could not find container \"cdcb32181d652df3402395552e0ae0898b41240322995bcf0f51e7ff004c2707\": container with ID starting with cdcb32181d652df3402395552e0ae0898b41240322995bcf0f51e7ff004c2707 not found: ID does not exist" Oct 31 00:30:32.430778 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:32.430749 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-957019-openshift-default-59bc4d44f9-8xtcd"] Oct 31 00:30:32.481800 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:32.481769 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-957019-openshift-default-59bc4d44f9-8xtcd"] Oct 31 00:30:33.086638 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:33.086604 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b623edb-855c-44b8-a0c1-abbd33925bd4" path="/var/lib/kubelet/pods/4b623edb-855c-44b8-a0c1-abbd33925bd4/volumes" Oct 31 00:30:33.571496 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:33.571460 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:30:34.841713 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:34.841672 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:30:38.220971 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:38.220923 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:30:38.942504 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:38.942470 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-46b586-669f464d79-fsqx4"] Oct 31 00:30:39.189853 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:39.189815 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:30:39.375294 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:39.375194 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:30:42.829863 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:42.829811 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:30:43.943598 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:43.943521 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-46b586-669f464d79-fsqx4" podUID="fea168a8-e4b2-46e2-8180-6910723a1982" containerName="payload-pre-processing" containerID="cri-o://288590708f982a09b365e63976837e3c6f890f81f277361dc85950fb338e2173" gracePeriod=25 Oct 31 00:30:44.993448 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:44.993411 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-46b586-669f464d79-fsqx4" podUID="fea168a8-e4b2-46e2-8180-6910723a1982" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 00:30:46.863425 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:46.863386 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-46b586/simulator-w0-46b586-kserve-67588b8dbd-z8xmn" podUID="9a25e252-223d-41f2-abf7-632067c77e0b" containerName="main" containerID="cri-o://5fc487df72f1e3d46eb58939813b297b3c070c8821cfd9ee0a2a6f85ec2c70a6" gracePeriod=45 Oct 31 00:30:47.125503 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:47.125441 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-46b586/simulator-w0-46b586-kserve-67588b8dbd-z8xmn" Oct 31 00:30:47.181410 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:47.181381 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9a25e252-223d-41f2-abf7-632067c77e0b-tls-certs\" (UniqueName: \"kubernetes.io/secret/9a25e252-223d-41f2-abf7-632067c77e0b-tls-certs\") pod \"9a25e252-223d-41f2-abf7-632067c77e0b\" (UID: \"9a25e252-223d-41f2-abf7-632067c77e0b\") " Oct 31 00:30:47.181549 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:47.181428 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a25e252-223d-41f2-abf7-632067c77e0b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9a25e252-223d-41f2-abf7-632067c77e0b-tmp-dir\") pod \"9a25e252-223d-41f2-abf7-632067c77e0b\" (UID: \"9a25e252-223d-41f2-abf7-632067c77e0b\") " Oct 31 00:30:47.181549 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:47.181468 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a25e252-223d-41f2-abf7-632067c77e0b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9a25e252-223d-41f2-abf7-632067c77e0b-model-cache\") pod \"9a25e252-223d-41f2-abf7-632067c77e0b\" (UID: \"9a25e252-223d-41f2-abf7-632067c77e0b\") " Oct 31 00:30:47.181549 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:47.181494 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a25e252-223d-41f2-abf7-632067c77e0b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9a25e252-223d-41f2-abf7-632067c77e0b-dshm\") pod \"9a25e252-223d-41f2-abf7-632067c77e0b\" (UID: \"9a25e252-223d-41f2-abf7-632067c77e0b\") " Oct 31 00:30:47.181549 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:47.181509 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a25e252-223d-41f2-abf7-632067c77e0b-home\" (UniqueName: \"kubernetes.io/empty-dir/9a25e252-223d-41f2-abf7-632067c77e0b-home\") pod \"9a25e252-223d-41f2-abf7-632067c77e0b\" (UID: \"9a25e252-223d-41f2-abf7-632067c77e0b\") " Oct 31 00:30:47.181796 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:47.181557 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a25e252-223d-41f2-abf7-632067c77e0b-kube-api-access-vc5g2\" (UniqueName: \"kubernetes.io/projected/9a25e252-223d-41f2-abf7-632067c77e0b-kube-api-access-vc5g2\") pod \"9a25e252-223d-41f2-abf7-632067c77e0b\" (UID: \"9a25e252-223d-41f2-abf7-632067c77e0b\") " Oct 31 00:30:47.181796 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:47.181718 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a25e252-223d-41f2-abf7-632067c77e0b-model-cache" pod "9a25e252-223d-41f2-abf7-632067c77e0b" (UID: "9a25e252-223d-41f2-abf7-632067c77e0b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:30:47.181928 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:47.181784 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a25e252-223d-41f2-abf7-632067c77e0b-tmp-dir" pod "9a25e252-223d-41f2-abf7-632067c77e0b" (UID: "9a25e252-223d-41f2-abf7-632067c77e0b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:30:47.181928 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:47.181849 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a25e252-223d-41f2-abf7-632067c77e0b-home" pod "9a25e252-223d-41f2-abf7-632067c77e0b" (UID: "9a25e252-223d-41f2-abf7-632067c77e0b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:30:47.182037 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:47.182013 4927 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9a25e252-223d-41f2-abf7-632067c77e0b-tmp-dir\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:47.182086 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:47.182034 4927 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9a25e252-223d-41f2-abf7-632067c77e0b-model-cache\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:47.182086 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:47.182051 4927 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9a25e252-223d-41f2-abf7-632067c77e0b-home\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:47.183776 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:47.183749 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a25e252-223d-41f2-abf7-632067c77e0b-dshm" pod "9a25e252-223d-41f2-abf7-632067c77e0b" (UID: "9a25e252-223d-41f2-abf7-632067c77e0b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:30:47.183887 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:47.183778 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a25e252-223d-41f2-abf7-632067c77e0b-kube-api-access-vc5g2" pod "9a25e252-223d-41f2-abf7-632067c77e0b" (UID: "9a25e252-223d-41f2-abf7-632067c77e0b"). InnerVolumeSpecName "kube-api-access-vc5g2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:30:47.183887 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:47.183818 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a25e252-223d-41f2-abf7-632067c77e0b-tls-certs" pod "9a25e252-223d-41f2-abf7-632067c77e0b" (UID: "9a25e252-223d-41f2-abf7-632067c77e0b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:30:47.283224 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:47.283184 4927 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9a25e252-223d-41f2-abf7-632067c77e0b-tls-certs\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:47.283224 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:47.283217 4927 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9a25e252-223d-41f2-abf7-632067c77e0b-dshm\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:47.283224 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:47.283229 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vc5g2\" (UniqueName: \"kubernetes.io/projected/9a25e252-223d-41f2-abf7-632067c77e0b-kube-api-access-vc5g2\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:47.359347 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:47.359291 4927 generic.go:356] "Generic (PLEG): container finished" podID="9a25e252-223d-41f2-abf7-632067c77e0b" containerID="5fc487df72f1e3d46eb58939813b297b3c070c8821cfd9ee0a2a6f85ec2c70a6" exitCode=0 Oct 31 00:30:47.359509 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:47.359368 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-46b586/simulator-w0-46b586-kserve-67588b8dbd-z8xmn" Oct 31 00:30:47.359509 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:47.359388 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-46b586/simulator-w0-46b586-kserve-67588b8dbd-z8xmn" event={"ID":"9a25e252-223d-41f2-abf7-632067c77e0b","Type":"ContainerDied","Data":"5fc487df72f1e3d46eb58939813b297b3c070c8821cfd9ee0a2a6f85ec2c70a6"} Oct 31 00:30:47.359509 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:47.359438 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-46b586/simulator-w0-46b586-kserve-67588b8dbd-z8xmn" event={"ID":"9a25e252-223d-41f2-abf7-632067c77e0b","Type":"ContainerDied","Data":"357a6258b7eb20df4d89943a06535cf9c4b20ecaa34582e2e0ab20eb8dfba36f"} Oct 31 00:30:47.359509 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:47.359457 4927 scope.go:122] "RemoveContainer" containerID="5fc487df72f1e3d46eb58939813b297b3c070c8821cfd9ee0a2a6f85ec2c70a6" Oct 31 00:30:47.369285 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:47.369265 4927 scope.go:122] "RemoveContainer" containerID="5fc487df72f1e3d46eb58939813b297b3c070c8821cfd9ee0a2a6f85ec2c70a6" Oct 31 00:30:47.369596 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:30:47.369574 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5fc487df72f1e3d46eb58939813b297b3c070c8821cfd9ee0a2a6f85ec2c70a6\": container with ID starting with 5fc487df72f1e3d46eb58939813b297b3c070c8821cfd9ee0a2a6f85ec2c70a6 not found: ID does not exist" containerID="5fc487df72f1e3d46eb58939813b297b3c070c8821cfd9ee0a2a6f85ec2c70a6" Oct 31 00:30:47.369666 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:47.369603 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5fc487df72f1e3d46eb58939813b297b3c070c8821cfd9ee0a2a6f85ec2c70a6"} err="failed to get container status \"5fc487df72f1e3d46eb58939813b297b3c070c8821cfd9ee0a2a6f85ec2c70a6\": rpc error: code = NotFound desc = could not find container \"5fc487df72f1e3d46eb58939813b297b3c070c8821cfd9ee0a2a6f85ec2c70a6\": container with ID starting with 5fc487df72f1e3d46eb58939813b297b3c070c8821cfd9ee0a2a6f85ec2c70a6 not found: ID does not exist" Oct 31 00:30:47.399880 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:47.399791 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:30:47.528787 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:47.528753 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-46b586/simulator-w0-46b586-kserve-67588b8dbd-z8xmn"] Oct 31 00:30:47.534113 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:47.534080 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-46b586/simulator-w0-46b586-kserve-67588b8dbd-z8xmn"] Oct 31 00:30:49.086525 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:49.086495 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a25e252-223d-41f2-abf7-632067c77e0b" path="/var/lib/kubelet/pods/9a25e252-223d-41f2-abf7-632067c77e0b/volumes" Oct 31 00:30:49.677980 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:30:49.677940 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sqv69" containerName="extractor" Oct 31 00:30:50.786105 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:50.786069 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-37abc0/test-model-37abc0-kserve-5f4c8bbbb7-5959k"] Oct 31 00:30:50.832808 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:50.832755 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:30:51.299494 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:51.299457 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ddf61a-9vr6p"] Oct 31 00:30:51.299938 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:51.299926 4927 cpu_manager.go:414] "containerMap: removing container" podUID="8b01ad9a-b63a-40ea-9dfb-4ea15d74bc50" containerName="cleanup" Oct 31 00:30:51.299986 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:51.299940 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b01ad9a-b63a-40ea-9dfb-4ea15d74bc50" containerName="cleanup" Oct 31 00:30:51.299986 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:51.299959 4927 cpu_manager.go:414] "containerMap: removing container" podUID="4b623edb-855c-44b8-a0c1-abbd33925bd4" containerName="istio-proxy" Oct 31 00:30:51.299986 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:51.299965 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b623edb-855c-44b8-a0c1-abbd33925bd4" containerName="istio-proxy" Oct 31 00:30:51.299986 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:51.299978 4927 cpu_manager.go:414] "containerMap: removing container" podUID="b75ee4b6-1c0b-4a64-a4a9-14da52d5f7db" containerName="cleanup" Oct 31 00:30:51.299986 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:51.299984 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b75ee4b6-1c0b-4a64-a4a9-14da52d5f7db" containerName="cleanup" Oct 31 00:30:51.300148 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:51.300003 4927 cpu_manager.go:414] "containerMap: removing container" podUID="c6421ddc-b3e6-47d2-960e-b9716f234b3a" containerName="revoke-keys" Oct 31 00:30:51.300148 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:51.300008 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6421ddc-b3e6-47d2-960e-b9716f234b3a" containerName="revoke-keys" Oct 31 00:30:51.300148 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:51.300015 4927 cpu_manager.go:414] "containerMap: removing container" podUID="9a25e252-223d-41f2-abf7-632067c77e0b" containerName="main" Oct 31 00:30:51.300148 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:51.300020 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a25e252-223d-41f2-abf7-632067c77e0b" containerName="main" Oct 31 00:30:51.300148 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:51.300089 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="0fda0711-f23d-409b-af43-23d6bbb8aac7" containerName="cleanup" Oct 31 00:30:51.300148 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:51.300101 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6421ddc-b3e6-47d2-960e-b9716f234b3a" containerName="revoke-keys" Oct 31 00:30:51.300148 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:51.300109 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a25e252-223d-41f2-abf7-632067c77e0b" containerName="main" Oct 31 00:30:51.300148 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:51.300115 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b623edb-855c-44b8-a0c1-abbd33925bd4" containerName="istio-proxy" Oct 31 00:30:51.300148 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:51.300122 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b01ad9a-b63a-40ea-9dfb-4ea15d74bc50" containerName="cleanup" Oct 31 00:30:51.300148 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:51.300131 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="b75ee4b6-1c0b-4a64-a4a9-14da52d5f7db" containerName="cleanup" Oct 31 00:30:51.304970 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:51.304949 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ddf61a-9vr6p" Oct 31 00:30:51.359750 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:51.359712 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ddf61a-9vr6p"] Oct 31 00:30:51.360120 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:51.360064 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:30:51.360282 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:51.360262 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4fdzs\"" Oct 31 00:30:51.429184 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:51.429144 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b078e00-ce6f-4c16-8e3b-3355ef379b8a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-ddf61a-9vr6p\" (UID: \"4b078e00-ce6f-4c16-8e3b-3355ef379b8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ddf61a-9vr6p" Oct 31 00:30:51.429184 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:51.429187 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rz9ps\" (UniqueName: \"kubernetes.io/projected/4b078e00-ce6f-4c16-8e3b-3355ef379b8a-kube-api-access-rz9ps\") pod \"maas-api-revoke-keys-e2e-inf-a-ddf61a-9vr6p\" (UID: \"4b078e00-ce6f-4c16-8e3b-3355ef379b8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ddf61a-9vr6p" Oct 31 00:30:51.429457 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:51.429291 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b078e00-ce6f-4c16-8e3b-3355ef379b8a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-ddf61a-9vr6p\" (UID: \"4b078e00-ce6f-4c16-8e3b-3355ef379b8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ddf61a-9vr6p" Oct 31 00:30:51.530346 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:51.530302 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b078e00-ce6f-4c16-8e3b-3355ef379b8a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-ddf61a-9vr6p\" (UID: \"4b078e00-ce6f-4c16-8e3b-3355ef379b8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ddf61a-9vr6p" Oct 31 00:30:51.530559 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:51.530382 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rz9ps\" (UniqueName: \"kubernetes.io/projected/4b078e00-ce6f-4c16-8e3b-3355ef379b8a-kube-api-access-rz9ps\") pod \"maas-api-revoke-keys-e2e-inf-a-ddf61a-9vr6p\" (UID: \"4b078e00-ce6f-4c16-8e3b-3355ef379b8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ddf61a-9vr6p" Oct 31 00:30:51.530559 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:51.530457 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b078e00-ce6f-4c16-8e3b-3355ef379b8a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-ddf61a-9vr6p\" (UID: \"4b078e00-ce6f-4c16-8e3b-3355ef379b8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ddf61a-9vr6p" Oct 31 00:30:51.531152 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:51.531122 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b078e00-ce6f-4c16-8e3b-3355ef379b8a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-ddf61a-9vr6p\" (UID: \"4b078e00-ce6f-4c16-8e3b-3355ef379b8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ddf61a-9vr6p" Oct 31 00:30:51.532688 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:51.532670 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b078e00-ce6f-4c16-8e3b-3355ef379b8a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-ddf61a-9vr6p\" (UID: \"4b078e00-ce6f-4c16-8e3b-3355ef379b8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ddf61a-9vr6p" Oct 31 00:30:51.577965 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:51.577881 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c93075a7-6dfxg"] Oct 31 00:30:51.578395 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:51.578381 4927 cpu_manager.go:414] "containerMap: removing container" podUID="4b623edb-855c-44b8-a0c1-abbd33925bd4" containerName="istio-proxy" Oct 31 00:30:51.578447 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:51.578397 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b623edb-855c-44b8-a0c1-abbd33925bd4" containerName="istio-proxy" Oct 31 00:30:51.578492 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:51.578482 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b623edb-855c-44b8-a0c1-abbd33925bd4" containerName="istio-proxy" Oct 31 00:30:51.581949 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:51.581933 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c93075a7-6dfxg" Oct 31 00:30:51.592525 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:51.592503 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rz9ps\" (UniqueName: \"kubernetes.io/projected/4b078e00-ce6f-4c16-8e3b-3355ef379b8a-kube-api-access-rz9ps\") pod \"maas-api-revoke-keys-e2e-inf-a-ddf61a-9vr6p\" (UID: \"4b078e00-ce6f-4c16-8e3b-3355ef379b8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ddf61a-9vr6p" Oct 31 00:30:51.621857 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:51.621833 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ddf61a-9vr6p" Oct 31 00:30:51.631319 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:51.631287 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgfkv\" (UniqueName: \"kubernetes.io/projected/bd7ab0e9-13cb-45b7-8a57-0f82d4895093-kube-api-access-xgfkv\") pod \"maas-api-revoke-sub-c93075a7-6dfxg\" (UID: \"bd7ab0e9-13cb-45b7-8a57-0f82d4895093\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c93075a7-6dfxg" Oct 31 00:30:51.631471 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:51.631381 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd7ab0e9-13cb-45b7-8a57-0f82d4895093-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c93075a7-6dfxg\" (UID: \"bd7ab0e9-13cb-45b7-8a57-0f82d4895093\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c93075a7-6dfxg" Oct 31 00:30:51.631471 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:51.631463 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd7ab0e9-13cb-45b7-8a57-0f82d4895093-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c93075a7-6dfxg\" (UID: \"bd7ab0e9-13cb-45b7-8a57-0f82d4895093\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c93075a7-6dfxg" Oct 31 00:30:51.637920 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:51.637300 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c93075a7-6dfxg"] Oct 31 00:30:51.732186 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:51.732148 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd7ab0e9-13cb-45b7-8a57-0f82d4895093-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c93075a7-6dfxg\" (UID: \"bd7ab0e9-13cb-45b7-8a57-0f82d4895093\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c93075a7-6dfxg" Oct 31 00:30:51.732415 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:51.732227 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgfkv\" (UniqueName: \"kubernetes.io/projected/bd7ab0e9-13cb-45b7-8a57-0f82d4895093-kube-api-access-xgfkv\") pod \"maas-api-revoke-sub-c93075a7-6dfxg\" (UID: \"bd7ab0e9-13cb-45b7-8a57-0f82d4895093\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c93075a7-6dfxg" Oct 31 00:30:51.732415 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:51.732272 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd7ab0e9-13cb-45b7-8a57-0f82d4895093-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c93075a7-6dfxg\" (UID: \"bd7ab0e9-13cb-45b7-8a57-0f82d4895093\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c93075a7-6dfxg" Oct 31 00:30:51.732861 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:51.732830 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd7ab0e9-13cb-45b7-8a57-0f82d4895093-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c93075a7-6dfxg\" (UID: \"bd7ab0e9-13cb-45b7-8a57-0f82d4895093\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c93075a7-6dfxg" Oct 31 00:30:51.734756 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:51.734728 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd7ab0e9-13cb-45b7-8a57-0f82d4895093-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c93075a7-6dfxg\" (UID: \"bd7ab0e9-13cb-45b7-8a57-0f82d4895093\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c93075a7-6dfxg" Oct 31 00:30:51.797588 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:51.797552 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgfkv\" (UniqueName: \"kubernetes.io/projected/bd7ab0e9-13cb-45b7-8a57-0f82d4895093-kube-api-access-xgfkv\") pod \"maas-api-revoke-sub-c93075a7-6dfxg\" (UID: \"bd7ab0e9-13cb-45b7-8a57-0f82d4895093\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c93075a7-6dfxg" Oct 31 00:30:51.895629 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:51.895560 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c93075a7-6dfxg" Oct 31 00:30:52.010078 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:52.010050 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ddf61a-9vr6p"] Oct 31 00:30:52.011251 ip-10-0-141-180 kubenswrapper[4927]: W1031 00:30:52.011224 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b078e00_ce6f_4c16_8e3b_3355ef379b8a.slice/crio-c12148ac51f262191a100b8367df9b3c91cba9e2feafa9726716e599fd4e0ab5 WatchSource:0}: Error finding container c12148ac51f262191a100b8367df9b3c91cba9e2feafa9726716e599fd4e0ab5: Status 404 returned error can't find the container with id c12148ac51f262191a100b8367df9b3c91cba9e2feafa9726716e599fd4e0ab5 Oct 31 00:30:52.024040 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:52.024012 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b940a211-vfr92"] Oct 31 00:30:52.029270 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:52.029239 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b940a211-vfr92" Oct 31 00:30:52.121065 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:52.121031 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b940a211-vfr92"] Oct 31 00:30:52.135655 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:52.135624 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9tzx4\" (UniqueName: \"kubernetes.io/projected/443e190a-3873-4532-92da-5324fb3ddf18-kube-api-access-9tzx4\") pod \"maas-api-revoke-sub-b940a211-vfr92\" (UID: \"443e190a-3873-4532-92da-5324fb3ddf18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b940a211-vfr92" Oct 31 00:30:52.135825 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:52.135770 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/443e190a-3873-4532-92da-5324fb3ddf18-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b940a211-vfr92\" (UID: \"443e190a-3873-4532-92da-5324fb3ddf18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b940a211-vfr92" Oct 31 00:30:52.135903 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:52.135859 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/443e190a-3873-4532-92da-5324fb3ddf18-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b940a211-vfr92\" (UID: \"443e190a-3873-4532-92da-5324fb3ddf18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b940a211-vfr92" Oct 31 00:30:52.237475 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:52.237447 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/443e190a-3873-4532-92da-5324fb3ddf18-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b940a211-vfr92\" (UID: \"443e190a-3873-4532-92da-5324fb3ddf18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b940a211-vfr92" Oct 31 00:30:52.237640 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:52.237512 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9tzx4\" (UniqueName: \"kubernetes.io/projected/443e190a-3873-4532-92da-5324fb3ddf18-kube-api-access-9tzx4\") pod \"maas-api-revoke-sub-b940a211-vfr92\" (UID: \"443e190a-3873-4532-92da-5324fb3ddf18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b940a211-vfr92" Oct 31 00:30:52.237640 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:52.237573 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/443e190a-3873-4532-92da-5324fb3ddf18-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b940a211-vfr92\" (UID: \"443e190a-3873-4532-92da-5324fb3ddf18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b940a211-vfr92" Oct 31 00:30:52.238184 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:52.238162 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/443e190a-3873-4532-92da-5324fb3ddf18-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b940a211-vfr92\" (UID: \"443e190a-3873-4532-92da-5324fb3ddf18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b940a211-vfr92" Oct 31 00:30:52.239810 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:52.239787 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/443e190a-3873-4532-92da-5324fb3ddf18-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b940a211-vfr92\" (UID: \"443e190a-3873-4532-92da-5324fb3ddf18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b940a211-vfr92" Oct 31 00:30:52.278877 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:52.278852 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c93075a7-6dfxg"] Oct 31 00:30:52.300522 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:52.300494 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9tzx4\" (UniqueName: \"kubernetes.io/projected/443e190a-3873-4532-92da-5324fb3ddf18-kube-api-access-9tzx4\") pod \"maas-api-revoke-sub-b940a211-vfr92\" (UID: \"443e190a-3873-4532-92da-5324fb3ddf18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b940a211-vfr92" Oct 31 00:30:52.309570 ip-10-0-141-180 kubenswrapper[4927]: W1031 00:30:52.309542 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd7ab0e9_13cb_45b7_8a57_0f82d4895093.slice/crio-7877c637a21d7eecc944c409c367a5e29a68819f968fccc3e943377dcadf2578 WatchSource:0}: Error finding container 7877c637a21d7eecc944c409c367a5e29a68819f968fccc3e943377dcadf2578: Status 404 returned error can't find the container with id 7877c637a21d7eecc944c409c367a5e29a68819f968fccc3e943377dcadf2578 Oct 31 00:30:52.355097 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:52.355068 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b940a211-vfr92" Oct 31 00:30:52.396489 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:52.395868 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ddf61a-9vr6p" event={"ID":"4b078e00-ce6f-4c16-8e3b-3355ef379b8a","Type":"ContainerStarted","Data":"3ae428042f01c0e007043ffcf8edbd2765401d8237bbe76f40f1d46e9c5848fc"} Oct 31 00:30:52.396489 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:52.395935 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ddf61a-9vr6p" event={"ID":"4b078e00-ce6f-4c16-8e3b-3355ef379b8a","Type":"ContainerStarted","Data":"c12148ac51f262191a100b8367df9b3c91cba9e2feafa9726716e599fd4e0ab5"} Oct 31 00:30:52.398470 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:52.398443 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c93075a7-6dfxg" event={"ID":"bd7ab0e9-13cb-45b7-8a57-0f82d4895093","Type":"ContainerStarted","Data":"7877c637a21d7eecc944c409c367a5e29a68819f968fccc3e943377dcadf2578"} Oct 31 00:30:52.490558 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:52.490459 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:30:52.527215 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:52.526681 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ddf61a-9vr6p" podStartSLOduration=1.526662083 podStartE2EDuration="1.526662083s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:30:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:30:52.523280493 +0000 UTC m=+2965.559611315" watchObservedRunningTime="2026-10-31 00:30:52.526662083 +0000 UTC m=+2965.562992913" Oct 31 00:30:52.738424 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:52.738395 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b940a211-vfr92"] Oct 31 00:30:52.739500 ip-10-0-141-180 kubenswrapper[4927]: W1031 00:30:52.739472 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod443e190a_3873_4532_92da_5324fb3ddf18.slice/crio-0ed678baff1d60f158294f600aca7e2656b9ca15ec7173ec72bc2bb066bb421f WatchSource:0}: Error finding container 0ed678baff1d60f158294f600aca7e2656b9ca15ec7173ec72bc2bb066bb421f: Status 404 returned error can't find the container with id 0ed678baff1d60f158294f600aca7e2656b9ca15ec7173ec72bc2bb066bb421f Oct 31 00:30:53.406215 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:53.406181 4927 generic.go:356] "Generic (PLEG): container finished" podID="4b078e00-ce6f-4c16-8e3b-3355ef379b8a" containerID="3ae428042f01c0e007043ffcf8edbd2765401d8237bbe76f40f1d46e9c5848fc" exitCode=0 Oct 31 00:30:53.406215 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:53.406210 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ddf61a-9vr6p" event={"ID":"4b078e00-ce6f-4c16-8e3b-3355ef379b8a","Type":"ContainerDied","Data":"3ae428042f01c0e007043ffcf8edbd2765401d8237bbe76f40f1d46e9c5848fc"} Oct 31 00:30:53.407984 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:53.407958 4927 generic.go:356] "Generic (PLEG): container finished" podID="bd7ab0e9-13cb-45b7-8a57-0f82d4895093" containerID="f04a85df51ad55a3594b00a5fd8efe5c2a78aa3c1df675ee335f7147a10d3748" exitCode=0 Oct 31 00:30:53.408119 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:53.408033 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c93075a7-6dfxg" event={"ID":"bd7ab0e9-13cb-45b7-8a57-0f82d4895093","Type":"ContainerDied","Data":"f04a85df51ad55a3594b00a5fd8efe5c2a78aa3c1df675ee335f7147a10d3748"} Oct 31 00:30:53.409634 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:53.409607 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b940a211-vfr92" event={"ID":"443e190a-3873-4532-92da-5324fb3ddf18","Type":"ContainerStarted","Data":"27b24f6894c7d6e9fc5ee17f849524d8952a27c86b292bb053b06cc7a0faecf9"} Oct 31 00:30:53.409756 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:53.409650 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b940a211-vfr92" event={"ID":"443e190a-3873-4532-92da-5324fb3ddf18","Type":"ContainerStarted","Data":"0ed678baff1d60f158294f600aca7e2656b9ca15ec7173ec72bc2bb066bb421f"} Oct 31 00:30:53.762692 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:53.762624 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b940a211-vfr92" podStartSLOduration=2.762608938 podStartE2EDuration="2.762608938s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:30:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:30:53.760375489 +0000 UTC m=+2966.796706311" watchObservedRunningTime="2026-10-31 00:30:53.762608938 +0000 UTC m=+2966.798939761" Oct 31 00:30:54.418746 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:54.418714 4927 generic.go:356] "Generic (PLEG): container finished" podID="443e190a-3873-4532-92da-5324fb3ddf18" containerID="27b24f6894c7d6e9fc5ee17f849524d8952a27c86b292bb053b06cc7a0faecf9" exitCode=0 Oct 31 00:30:54.419185 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:54.418799 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b940a211-vfr92" event={"ID":"443e190a-3873-4532-92da-5324fb3ddf18","Type":"ContainerDied","Data":"27b24f6894c7d6e9fc5ee17f849524d8952a27c86b292bb053b06cc7a0faecf9"} Oct 31 00:30:54.585739 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:54.585708 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c93075a7-6dfxg" Oct 31 00:30:54.618231 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:54.618193 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ddf61a-9vr6p" Oct 31 00:30:54.663983 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:54.663946 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bd7ab0e9-13cb-45b7-8a57-0f82d4895093-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd7ab0e9-13cb-45b7-8a57-0f82d4895093-maas-api-xks-ca\") pod \"bd7ab0e9-13cb-45b7-8a57-0f82d4895093\" (UID: \"bd7ab0e9-13cb-45b7-8a57-0f82d4895093\") " Oct 31 00:30:54.664182 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:54.664040 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bd7ab0e9-13cb-45b7-8a57-0f82d4895093-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd7ab0e9-13cb-45b7-8a57-0f82d4895093-maas-api-openshift-ca\") pod \"bd7ab0e9-13cb-45b7-8a57-0f82d4895093\" (UID: \"bd7ab0e9-13cb-45b7-8a57-0f82d4895093\") " Oct 31 00:30:54.664182 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:54.664085 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd7ab0e9-13cb-45b7-8a57-0f82d4895093-kube-api-access-xgfkv\" (UniqueName: \"kubernetes.io/projected/bd7ab0e9-13cb-45b7-8a57-0f82d4895093-kube-api-access-xgfkv\") pod \"bd7ab0e9-13cb-45b7-8a57-0f82d4895093\" (UID: \"bd7ab0e9-13cb-45b7-8a57-0f82d4895093\") " Oct 31 00:30:54.664481 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:54.664454 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd7ab0e9-13cb-45b7-8a57-0f82d4895093-maas-api-openshift-ca" pod "bd7ab0e9-13cb-45b7-8a57-0f82d4895093" (UID: "bd7ab0e9-13cb-45b7-8a57-0f82d4895093"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:30:54.666211 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:54.666179 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bd7ab0e9-13cb-45b7-8a57-0f82d4895093-maas-api-xks-ca" pod "bd7ab0e9-13cb-45b7-8a57-0f82d4895093" (UID: "bd7ab0e9-13cb-45b7-8a57-0f82d4895093"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:30:54.666358 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:54.666248 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd7ab0e9-13cb-45b7-8a57-0f82d4895093-kube-api-access-xgfkv" pod "bd7ab0e9-13cb-45b7-8a57-0f82d4895093" (UID: "bd7ab0e9-13cb-45b7-8a57-0f82d4895093"). InnerVolumeSpecName "kube-api-access-xgfkv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:30:54.765056 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:54.765016 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b078e00-ce6f-4c16-8e3b-3355ef379b8a-kube-api-access-rz9ps\" (UniqueName: \"kubernetes.io/projected/4b078e00-ce6f-4c16-8e3b-3355ef379b8a-kube-api-access-rz9ps\") pod \"4b078e00-ce6f-4c16-8e3b-3355ef379b8a\" (UID: \"4b078e00-ce6f-4c16-8e3b-3355ef379b8a\") " Oct 31 00:30:54.765268 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:54.765092 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4b078e00-ce6f-4c16-8e3b-3355ef379b8a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b078e00-ce6f-4c16-8e3b-3355ef379b8a-maas-api-xks-ca\") pod \"4b078e00-ce6f-4c16-8e3b-3355ef379b8a\" (UID: \"4b078e00-ce6f-4c16-8e3b-3355ef379b8a\") " Oct 31 00:30:54.765268 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:54.765152 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4b078e00-ce6f-4c16-8e3b-3355ef379b8a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b078e00-ce6f-4c16-8e3b-3355ef379b8a-maas-api-openshift-ca\") pod \"4b078e00-ce6f-4c16-8e3b-3355ef379b8a\" (UID: \"4b078e00-ce6f-4c16-8e3b-3355ef379b8a\") " Oct 31 00:30:54.765506 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:54.765492 4927 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd7ab0e9-13cb-45b7-8a57-0f82d4895093-maas-api-openshift-ca\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:54.765586 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:54.765509 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xgfkv\" (UniqueName: \"kubernetes.io/projected/bd7ab0e9-13cb-45b7-8a57-0f82d4895093-kube-api-access-xgfkv\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:54.765586 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:54.765519 4927 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd7ab0e9-13cb-45b7-8a57-0f82d4895093-maas-api-xks-ca\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:54.765697 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:54.765621 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b078e00-ce6f-4c16-8e3b-3355ef379b8a-maas-api-openshift-ca" pod "4b078e00-ce6f-4c16-8e3b-3355ef379b8a" (UID: "4b078e00-ce6f-4c16-8e3b-3355ef379b8a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:30:54.767195 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:54.767172 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4b078e00-ce6f-4c16-8e3b-3355ef379b8a-maas-api-xks-ca" pod "4b078e00-ce6f-4c16-8e3b-3355ef379b8a" (UID: "4b078e00-ce6f-4c16-8e3b-3355ef379b8a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:30:54.767285 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:54.767226 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b078e00-ce6f-4c16-8e3b-3355ef379b8a-kube-api-access-rz9ps" pod "4b078e00-ce6f-4c16-8e3b-3355ef379b8a" (UID: "4b078e00-ce6f-4c16-8e3b-3355ef379b8a"). InnerVolumeSpecName "kube-api-access-rz9ps". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:30:54.866921 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:54.866882 4927 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b078e00-ce6f-4c16-8e3b-3355ef379b8a-maas-api-xks-ca\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:54.866921 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:54.866915 4927 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b078e00-ce6f-4c16-8e3b-3355ef379b8a-maas-api-openshift-ca\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:54.866921 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:54.866926 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rz9ps\" (UniqueName: \"kubernetes.io/projected/4b078e00-ce6f-4c16-8e3b-3355ef379b8a-kube-api-access-rz9ps\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:54.993047 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:54.992956 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-46b586-669f464d79-fsqx4" podUID="fea168a8-e4b2-46e2-8180-6910723a1982" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 00:30:55.428063 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:55.428032 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ddf61a-9vr6p" Oct 31 00:30:55.428063 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:55.428050 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ddf61a-9vr6p" event={"ID":"4b078e00-ce6f-4c16-8e3b-3355ef379b8a","Type":"ContainerDied","Data":"c12148ac51f262191a100b8367df9b3c91cba9e2feafa9726716e599fd4e0ab5"} Oct 31 00:30:55.428667 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:55.428107 4927 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c12148ac51f262191a100b8367df9b3c91cba9e2feafa9726716e599fd4e0ab5" Oct 31 00:30:55.429534 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:55.429511 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c93075a7-6dfxg" Oct 31 00:30:55.429656 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:55.429538 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c93075a7-6dfxg" event={"ID":"bd7ab0e9-13cb-45b7-8a57-0f82d4895093","Type":"ContainerDied","Data":"7877c637a21d7eecc944c409c367a5e29a68819f968fccc3e943377dcadf2578"} Oct 31 00:30:55.429656 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:55.429596 4927 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7877c637a21d7eecc944c409c367a5e29a68819f968fccc3e943377dcadf2578" Oct 31 00:30:55.557741 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:55.557705 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b940a211-vfr92" Oct 31 00:30:55.675136 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:55.675103 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/443e190a-3873-4532-92da-5324fb3ddf18-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/443e190a-3873-4532-92da-5324fb3ddf18-maas-api-openshift-ca\") pod \"443e190a-3873-4532-92da-5324fb3ddf18\" (UID: \"443e190a-3873-4532-92da-5324fb3ddf18\") " Oct 31 00:30:55.675322 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:55.675165 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/443e190a-3873-4532-92da-5324fb3ddf18-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/443e190a-3873-4532-92da-5324fb3ddf18-maas-api-xks-ca\") pod \"443e190a-3873-4532-92da-5324fb3ddf18\" (UID: \"443e190a-3873-4532-92da-5324fb3ddf18\") " Oct 31 00:30:55.675442 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:55.675370 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/443e190a-3873-4532-92da-5324fb3ddf18-kube-api-access-9tzx4\" (UniqueName: \"kubernetes.io/projected/443e190a-3873-4532-92da-5324fb3ddf18-kube-api-access-9tzx4\") pod \"443e190a-3873-4532-92da-5324fb3ddf18\" (UID: \"443e190a-3873-4532-92da-5324fb3ddf18\") " Oct 31 00:30:55.675560 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:55.675536 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/443e190a-3873-4532-92da-5324fb3ddf18-maas-api-openshift-ca" pod "443e190a-3873-4532-92da-5324fb3ddf18" (UID: "443e190a-3873-4532-92da-5324fb3ddf18"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:30:55.675700 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:55.675677 4927 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/443e190a-3873-4532-92da-5324fb3ddf18-maas-api-openshift-ca\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:55.678407 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:55.678343 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/443e190a-3873-4532-92da-5324fb3ddf18-maas-api-xks-ca" pod "443e190a-3873-4532-92da-5324fb3ddf18" (UID: "443e190a-3873-4532-92da-5324fb3ddf18"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:30:55.678407 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:55.678353 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/443e190a-3873-4532-92da-5324fb3ddf18-kube-api-access-9tzx4" pod "443e190a-3873-4532-92da-5324fb3ddf18" (UID: "443e190a-3873-4532-92da-5324fb3ddf18"). InnerVolumeSpecName "kube-api-access-9tzx4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:30:55.776830 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:55.776797 4927 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/443e190a-3873-4532-92da-5324fb3ddf18-maas-api-xks-ca\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:55.776830 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:55.776825 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9tzx4\" (UniqueName: \"kubernetes.io/projected/443e190a-3873-4532-92da-5324fb3ddf18-kube-api-access-9tzx4\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:56.439819 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:56.439780 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b940a211-vfr92" Oct 31 00:30:56.440364 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:56.439821 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b940a211-vfr92" event={"ID":"443e190a-3873-4532-92da-5324fb3ddf18","Type":"ContainerDied","Data":"0ed678baff1d60f158294f600aca7e2656b9ca15ec7173ec72bc2bb066bb421f"} Oct 31 00:30:56.440364 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:56.439878 4927 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0ed678baff1d60f158294f600aca7e2656b9ca15ec7173ec72bc2bb066bb421f" Oct 31 00:30:57.305265 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:30:57.305232 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:31:01.900471 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:01.900433 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:31:03.114717 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:03.114680 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:31:03.306270 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:03.306230 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:31:04.257032 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:04.257005 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-46b586-669f464d79-fsqx4" Oct 31 00:31:04.374342 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:04.374307 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fea168a8-e4b2-46e2-8180-6910723a1982-kube-api-access-b7d25\" (UniqueName: \"kubernetes.io/projected/fea168a8-e4b2-46e2-8180-6910723a1982-kube-api-access-b7d25\") pod \"fea168a8-e4b2-46e2-8180-6910723a1982\" (UID: \"fea168a8-e4b2-46e2-8180-6910723a1982\") " Oct 31 00:31:04.374532 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:04.374435 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fea168a8-e4b2-46e2-8180-6910723a1982-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fea168a8-e4b2-46e2-8180-6910723a1982-plugins-config-volume\") pod \"fea168a8-e4b2-46e2-8180-6910723a1982\" (UID: \"fea168a8-e4b2-46e2-8180-6910723a1982\") " Oct 31 00:31:04.374912 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:04.374843 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fea168a8-e4b2-46e2-8180-6910723a1982-plugins-config-volume" pod "fea168a8-e4b2-46e2-8180-6910723a1982" (UID: "fea168a8-e4b2-46e2-8180-6910723a1982"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:31:04.382112 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:04.382061 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fea168a8-e4b2-46e2-8180-6910723a1982-kube-api-access-b7d25" pod "fea168a8-e4b2-46e2-8180-6910723a1982" (UID: "fea168a8-e4b2-46e2-8180-6910723a1982"). InnerVolumeSpecName "kube-api-access-b7d25". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:31:04.475874 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:04.475824 4927 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fea168a8-e4b2-46e2-8180-6910723a1982-plugins-config-volume\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:31:04.475874 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:04.475868 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b7d25\" (UniqueName: \"kubernetes.io/projected/fea168a8-e4b2-46e2-8180-6910723a1982-kube-api-access-b7d25\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:31:04.495955 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:04.495870 4927 generic.go:356] "Generic (PLEG): container finished" podID="fea168a8-e4b2-46e2-8180-6910723a1982" containerID="288590708f982a09b365e63976837e3c6f890f81f277361dc85950fb338e2173" exitCode=0 Oct 31 00:31:04.495955 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:04.495918 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-46b586-669f464d79-fsqx4" event={"ID":"fea168a8-e4b2-46e2-8180-6910723a1982","Type":"ContainerDied","Data":"288590708f982a09b365e63976837e3c6f890f81f277361dc85950fb338e2173"} Oct 31 00:31:04.495955 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:04.495927 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-46b586-669f464d79-fsqx4" Oct 31 00:31:04.496247 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:04.495962 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-46b586-669f464d79-fsqx4" event={"ID":"fea168a8-e4b2-46e2-8180-6910723a1982","Type":"ContainerDied","Data":"73dcdd56c3920557bcd6d8d74ad71522b086d38a38c10d64d83abcb6ef53a479"} Oct 31 00:31:04.496247 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:04.495983 4927 scope.go:122] "RemoveContainer" containerID="288590708f982a09b365e63976837e3c6f890f81f277361dc85950fb338e2173" Oct 31 00:31:04.506514 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:04.506493 4927 scope.go:122] "RemoveContainer" containerID="288590708f982a09b365e63976837e3c6f890f81f277361dc85950fb338e2173" Oct 31 00:31:04.506806 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:31:04.506783 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"288590708f982a09b365e63976837e3c6f890f81f277361dc85950fb338e2173\": container with ID starting with 288590708f982a09b365e63976837e3c6f890f81f277361dc85950fb338e2173 not found: ID does not exist" containerID="288590708f982a09b365e63976837e3c6f890f81f277361dc85950fb338e2173" Oct 31 00:31:04.506861 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:04.506817 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"288590708f982a09b365e63976837e3c6f890f81f277361dc85950fb338e2173"} err="failed to get container status \"288590708f982a09b365e63976837e3c6f890f81f277361dc85950fb338e2173\": rpc error: code = NotFound desc = could not find container \"288590708f982a09b365e63976837e3c6f890f81f277361dc85950fb338e2173\": container with ID starting with 288590708f982a09b365e63976837e3c6f890f81f277361dc85950fb338e2173 not found: ID does not exist" Oct 31 00:31:04.687814 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:04.687780 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-46b586-669f464d79-fsqx4"] Oct 31 00:31:04.713285 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:04.713247 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-46b586-669f464d79-fsqx4"] Oct 31 00:31:05.085915 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:05.085881 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fea168a8-e4b2-46e2-8180-6910723a1982" path="/var/lib/kubelet/pods/fea168a8-e4b2-46e2-8180-6910723a1982/volumes" Oct 31 00:31:05.806256 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:05.806192 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-37abc0/test-model-37abc0-kserve-5f4c8bbbb7-5959k" podUID="46532fb5-6817-4bf2-bb01-7b03eda6810a" containerName="main" containerID="cri-o://a2ad8efaa4c1b787fd64ba1901edc4ccb646b28661f27bcd59e466becfe4be88" gracePeriod=45 Oct 31 00:31:05.824945 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:05.824916 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-37abc0/test-model-37abc0-kserve-5f4c8bbbb7-5959k" podUID="46532fb5-6817-4bf2-bb01-7b03eda6810a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.249:8000/ready\": dial tcp 10.132.0.249:8000: connect: connection refused" Oct 31 00:31:06.070160 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:06.070080 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-37abc0/test-model-37abc0-kserve-5f4c8bbbb7-5959k" Oct 31 00:31:06.193087 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:06.193052 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46532fb5-6817-4bf2-bb01-7b03eda6810a-home\" (UniqueName: \"kubernetes.io/empty-dir/46532fb5-6817-4bf2-bb01-7b03eda6810a-home\") pod \"46532fb5-6817-4bf2-bb01-7b03eda6810a\" (UID: \"46532fb5-6817-4bf2-bb01-7b03eda6810a\") " Oct 31 00:31:06.193398 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:06.193376 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/46532fb5-6817-4bf2-bb01-7b03eda6810a-tls-certs\" (UniqueName: \"kubernetes.io/secret/46532fb5-6817-4bf2-bb01-7b03eda6810a-tls-certs\") pod \"46532fb5-6817-4bf2-bb01-7b03eda6810a\" (UID: \"46532fb5-6817-4bf2-bb01-7b03eda6810a\") " Oct 31 00:31:06.193497 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:06.193477 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46532fb5-6817-4bf2-bb01-7b03eda6810a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/46532fb5-6817-4bf2-bb01-7b03eda6810a-model-cache\") pod \"46532fb5-6817-4bf2-bb01-7b03eda6810a\" (UID: \"46532fb5-6817-4bf2-bb01-7b03eda6810a\") " Oct 31 00:31:06.193559 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:06.193528 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46532fb5-6817-4bf2-bb01-7b03eda6810a-kube-api-access-q8zh5\" (UniqueName: \"kubernetes.io/projected/46532fb5-6817-4bf2-bb01-7b03eda6810a-kube-api-access-q8zh5\") pod \"46532fb5-6817-4bf2-bb01-7b03eda6810a\" (UID: \"46532fb5-6817-4bf2-bb01-7b03eda6810a\") " Oct 31 00:31:06.193621 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:06.193565 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46532fb5-6817-4bf2-bb01-7b03eda6810a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/46532fb5-6817-4bf2-bb01-7b03eda6810a-dshm\") pod \"46532fb5-6817-4bf2-bb01-7b03eda6810a\" (UID: \"46532fb5-6817-4bf2-bb01-7b03eda6810a\") " Oct 31 00:31:06.193621 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:06.193606 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46532fb5-6817-4bf2-bb01-7b03eda6810a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/46532fb5-6817-4bf2-bb01-7b03eda6810a-tmp-dir\") pod \"46532fb5-6817-4bf2-bb01-7b03eda6810a\" (UID: \"46532fb5-6817-4bf2-bb01-7b03eda6810a\") " Oct 31 00:31:06.193763 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:06.193647 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46532fb5-6817-4bf2-bb01-7b03eda6810a-home" pod "46532fb5-6817-4bf2-bb01-7b03eda6810a" (UID: "46532fb5-6817-4bf2-bb01-7b03eda6810a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:31:06.193871 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:06.193845 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46532fb5-6817-4bf2-bb01-7b03eda6810a-model-cache" pod "46532fb5-6817-4bf2-bb01-7b03eda6810a" (UID: "46532fb5-6817-4bf2-bb01-7b03eda6810a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:31:06.194016 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:06.193999 4927 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/46532fb5-6817-4bf2-bb01-7b03eda6810a-home\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:31:06.194081 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:06.194022 4927 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/46532fb5-6817-4bf2-bb01-7b03eda6810a-model-cache\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:31:06.194323 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:06.194295 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46532fb5-6817-4bf2-bb01-7b03eda6810a-tmp-dir" pod "46532fb5-6817-4bf2-bb01-7b03eda6810a" (UID: "46532fb5-6817-4bf2-bb01-7b03eda6810a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:31:06.196430 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:06.196400 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46532fb5-6817-4bf2-bb01-7b03eda6810a-kube-api-access-q8zh5" pod "46532fb5-6817-4bf2-bb01-7b03eda6810a" (UID: "46532fb5-6817-4bf2-bb01-7b03eda6810a"). InnerVolumeSpecName "kube-api-access-q8zh5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:31:06.196558 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:06.196474 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/46532fb5-6817-4bf2-bb01-7b03eda6810a-tls-certs" pod "46532fb5-6817-4bf2-bb01-7b03eda6810a" (UID: "46532fb5-6817-4bf2-bb01-7b03eda6810a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:31:06.196627 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:06.196597 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46532fb5-6817-4bf2-bb01-7b03eda6810a-dshm" pod "46532fb5-6817-4bf2-bb01-7b03eda6810a" (UID: "46532fb5-6817-4bf2-bb01-7b03eda6810a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:31:06.295185 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:06.295057 4927 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/46532fb5-6817-4bf2-bb01-7b03eda6810a-tls-certs\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:31:06.295185 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:06.295091 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q8zh5\" (UniqueName: \"kubernetes.io/projected/46532fb5-6817-4bf2-bb01-7b03eda6810a-kube-api-access-q8zh5\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:31:06.295185 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:06.295104 4927 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/46532fb5-6817-4bf2-bb01-7b03eda6810a-dshm\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:31:06.295185 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:06.295113 4927 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/46532fb5-6817-4bf2-bb01-7b03eda6810a-tmp-dir\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:31:06.510931 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:06.510898 4927 generic.go:356] "Generic (PLEG): container finished" podID="46532fb5-6817-4bf2-bb01-7b03eda6810a" containerID="a2ad8efaa4c1b787fd64ba1901edc4ccb646b28661f27bcd59e466becfe4be88" exitCode=0 Oct 31 00:31:06.511119 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:06.510943 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-37abc0/test-model-37abc0-kserve-5f4c8bbbb7-5959k" Oct 31 00:31:06.511119 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:06.510987 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-37abc0/test-model-37abc0-kserve-5f4c8bbbb7-5959k" event={"ID":"46532fb5-6817-4bf2-bb01-7b03eda6810a","Type":"ContainerDied","Data":"a2ad8efaa4c1b787fd64ba1901edc4ccb646b28661f27bcd59e466becfe4be88"} Oct 31 00:31:06.511119 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:06.511039 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-37abc0/test-model-37abc0-kserve-5f4c8bbbb7-5959k" event={"ID":"46532fb5-6817-4bf2-bb01-7b03eda6810a","Type":"ContainerDied","Data":"5803f62a54736e3b1fe5877987cc61f0332691e9000bb8e5e99d150b474841f5"} Oct 31 00:31:06.511119 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:06.511062 4927 scope.go:122] "RemoveContainer" containerID="a2ad8efaa4c1b787fd64ba1901edc4ccb646b28661f27bcd59e466becfe4be88" Oct 31 00:31:06.522085 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:06.522063 4927 scope.go:122] "RemoveContainer" containerID="a2ad8efaa4c1b787fd64ba1901edc4ccb646b28661f27bcd59e466becfe4be88" Oct 31 00:31:06.522397 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:31:06.522371 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2ad8efaa4c1b787fd64ba1901edc4ccb646b28661f27bcd59e466becfe4be88\": container with ID starting with a2ad8efaa4c1b787fd64ba1901edc4ccb646b28661f27bcd59e466becfe4be88 not found: ID does not exist" containerID="a2ad8efaa4c1b787fd64ba1901edc4ccb646b28661f27bcd59e466becfe4be88" Oct 31 00:31:06.522486 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:06.522410 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2ad8efaa4c1b787fd64ba1901edc4ccb646b28661f27bcd59e466becfe4be88"} err="failed to get container status \"a2ad8efaa4c1b787fd64ba1901edc4ccb646b28661f27bcd59e466becfe4be88\": rpc error: code = NotFound desc = could not find container \"a2ad8efaa4c1b787fd64ba1901edc4ccb646b28661f27bcd59e466becfe4be88\": container with ID starting with a2ad8efaa4c1b787fd64ba1901edc4ccb646b28661f27bcd59e466becfe4be88 not found: ID does not exist" Oct 31 00:31:06.608391 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:06.608358 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:31:06.681490 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:06.681453 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-37abc0/test-model-37abc0-kserve-5f4c8bbbb7-5959k"] Oct 31 00:31:06.686028 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:06.685997 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-37abc0/test-model-37abc0-kserve-5f4c8bbbb7-5959k"] Oct 31 00:31:07.086512 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:07.086481 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46532fb5-6817-4bf2-bb01-7b03eda6810a" path="/var/lib/kubelet/pods/46532fb5-6817-4bf2-bb01-7b03eda6810a/volumes" Oct 31 00:31:09.298610 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:09.298572 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:31:09.453397 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:09.453316 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:31:09.940507 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:09.940467 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:31:10.619304 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:10.619265 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:31:11.718968 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:11.718936 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:31:12.100072 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:12.099969 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:31:12.344446 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:12.344407 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:31:12.540805 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:12.540769 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-46b586-bsx74"] Oct 31 00:31:12.590044 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:12.590006 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-46b586-bsx74"] Oct 31 00:31:13.005606 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:13.005572 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:31:13.085322 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:13.085292 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c6421ddc-b3e6-47d2-960e-b9716f234b3a" path="/var/lib/kubelet/pods/c6421ddc-b3e6-47d2-960e-b9716f234b3a/volumes" Oct 31 00:31:13.267285 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:13.267203 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:31:13.552694 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:13.552603 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:31:13.709830 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:13.709786 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:31:13.843698 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:13.843594 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ddf61a-9vr6p"] Oct 31 00:31:13.892743 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:13.892708 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ddf61a-9vr6p"] Oct 31 00:31:14.209074 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:14.209025 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:31:15.085661 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:15.085619 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b078e00-ce6f-4c16-8e3b-3355ef379b8a" path="/var/lib/kubelet/pods/4b078e00-ce6f-4c16-8e3b-3355ef379b8a/volumes" Oct 31 00:31:16.430663 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:16.430622 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:31:18.654664 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:18.654627 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:31:19.677763 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:31:19.677726 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sqv69" containerName="extractor" Oct 31 00:31:19.691717 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:19.691687 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:31:19.892367 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:19.892252 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:31:20.098473 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:20.098354 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:31:21.054283 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:21.054249 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:31:21.209875 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:21.209826 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:31:21.404966 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:21.404889 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:31:21.408557 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:21.408528 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-ddf61a-openshift-default-f57f54bc-4gvfk"] Oct 31 00:31:21.408812 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:21.408777 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-ddf61a-openshift-default-f57f54bc-4gvfk" podUID="8a91e846-1ef4-4063-860e-46cdbf225ced" containerName="istio-proxy" containerID="cri-o://5f78ac7307756be3414509c8fea905737c9432c9e9f25c7f910b02b7bcb9c51e" gracePeriod=30 Oct 31 00:31:22.234356 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:22.234311 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-37abc0-lb5gt"] Oct 31 00:31:22.234802 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:22.234787 4927 cpu_manager.go:414] "containerMap: removing container" podUID="443e190a-3873-4532-92da-5324fb3ddf18" containerName="revoke-keys" Oct 31 00:31:22.234853 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:22.234804 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="443e190a-3873-4532-92da-5324fb3ddf18" containerName="revoke-keys" Oct 31 00:31:22.234853 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:22.234825 4927 cpu_manager.go:414] "containerMap: removing container" podUID="bd7ab0e9-13cb-45b7-8a57-0f82d4895093" containerName="revoke-keys" Oct 31 00:31:22.234853 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:22.234831 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd7ab0e9-13cb-45b7-8a57-0f82d4895093" containerName="revoke-keys" Oct 31 00:31:22.234964 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:22.234852 4927 cpu_manager.go:414] "containerMap: removing container" podUID="46532fb5-6817-4bf2-bb01-7b03eda6810a" containerName="main" Oct 31 00:31:22.234964 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:22.234861 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46532fb5-6817-4bf2-bb01-7b03eda6810a" containerName="main" Oct 31 00:31:22.234964 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:22.234879 4927 cpu_manager.go:414] "containerMap: removing container" podUID="fea168a8-e4b2-46e2-8180-6910723a1982" containerName="payload-pre-processing" Oct 31 00:31:22.234964 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:22.234883 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fea168a8-e4b2-46e2-8180-6910723a1982" containerName="payload-pre-processing" Oct 31 00:31:22.234964 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:22.234893 4927 cpu_manager.go:414] "containerMap: removing container" podUID="4b078e00-ce6f-4c16-8e3b-3355ef379b8a" containerName="revoke-keys" Oct 31 00:31:22.234964 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:22.234897 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b078e00-ce6f-4c16-8e3b-3355ef379b8a" containerName="revoke-keys" Oct 31 00:31:22.235151 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:22.234970 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd7ab0e9-13cb-45b7-8a57-0f82d4895093" containerName="revoke-keys" Oct 31 00:31:22.235151 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:22.234983 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="46532fb5-6817-4bf2-bb01-7b03eda6810a" containerName="main" Oct 31 00:31:22.235151 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:22.234992 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="fea168a8-e4b2-46e2-8180-6910723a1982" containerName="payload-pre-processing" Oct 31 00:31:22.235151 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:22.234998 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b078e00-ce6f-4c16-8e3b-3355ef379b8a" containerName="revoke-keys" Oct 31 00:31:22.235151 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:22.235004 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="443e190a-3873-4532-92da-5324fb3ddf18" containerName="revoke-keys" Oct 31 00:31:22.239781 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:22.239762 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-37abc0-lb5gt" Oct 31 00:31:22.292692 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:22.292656 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-37abc0-lb5gt"] Oct 31 00:31:22.301011 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:22.300985 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:31:22.301323 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:22.301295 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4fdzs\"" Oct 31 00:31:22.357781 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:22.357750 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2nkkb\" (UniqueName: \"kubernetes.io/projected/61edd758-f5e7-4ca0-a1df-ba7161712831-kube-api-access-2nkkb\") pod \"maas-api-revoke-keys-e2e-inf-b-37abc0-lb5gt\" (UID: \"61edd758-f5e7-4ca0-a1df-ba7161712831\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-37abc0-lb5gt" Oct 31 00:31:22.357954 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:22.357798 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61edd758-f5e7-4ca0-a1df-ba7161712831-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-37abc0-lb5gt\" (UID: \"61edd758-f5e7-4ca0-a1df-ba7161712831\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-37abc0-lb5gt" Oct 31 00:31:22.357954 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:22.357820 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61edd758-f5e7-4ca0-a1df-ba7161712831-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-37abc0-lb5gt\" (UID: \"61edd758-f5e7-4ca0-a1df-ba7161712831\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-37abc0-lb5gt" Oct 31 00:31:22.458684 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:22.458647 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61edd758-f5e7-4ca0-a1df-ba7161712831-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-37abc0-lb5gt\" (UID: \"61edd758-f5e7-4ca0-a1df-ba7161712831\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-37abc0-lb5gt" Oct 31 00:31:22.458684 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:22.458684 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61edd758-f5e7-4ca0-a1df-ba7161712831-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-37abc0-lb5gt\" (UID: \"61edd758-f5e7-4ca0-a1df-ba7161712831\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-37abc0-lb5gt" Oct 31 00:31:22.458910 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:22.458785 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2nkkb\" (UniqueName: \"kubernetes.io/projected/61edd758-f5e7-4ca0-a1df-ba7161712831-kube-api-access-2nkkb\") pod \"maas-api-revoke-keys-e2e-inf-b-37abc0-lb5gt\" (UID: \"61edd758-f5e7-4ca0-a1df-ba7161712831\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-37abc0-lb5gt" Oct 31 00:31:22.459358 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:22.459308 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61edd758-f5e7-4ca0-a1df-ba7161712831-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-37abc0-lb5gt\" (UID: \"61edd758-f5e7-4ca0-a1df-ba7161712831\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-37abc0-lb5gt" Oct 31 00:31:22.461151 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:22.461121 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61edd758-f5e7-4ca0-a1df-ba7161712831-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-37abc0-lb5gt\" (UID: \"61edd758-f5e7-4ca0-a1df-ba7161712831\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-37abc0-lb5gt" Oct 31 00:31:22.524701 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:22.524634 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2nkkb\" (UniqueName: \"kubernetes.io/projected/61edd758-f5e7-4ca0-a1df-ba7161712831-kube-api-access-2nkkb\") pod \"maas-api-revoke-keys-e2e-inf-b-37abc0-lb5gt\" (UID: \"61edd758-f5e7-4ca0-a1df-ba7161712831\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-37abc0-lb5gt" Oct 31 00:31:22.552640 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:22.552609 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-37abc0-lb5gt" Oct 31 00:31:22.931768 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:22.931735 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-37abc0-lb5gt"] Oct 31 00:31:22.933484 ip-10-0-141-180 kubenswrapper[4927]: W1031 00:31:22.933459 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61edd758_f5e7_4ca0_a1df_ba7161712831.slice/crio-87522efc17119cf8d125697d7b70a2f5832f51fc43b4c1b2efdb158d011022b0 WatchSource:0}: Error finding container 87522efc17119cf8d125697d7b70a2f5832f51fc43b4c1b2efdb158d011022b0: Status 404 returned error can't find the container with id 87522efc17119cf8d125697d7b70a2f5832f51fc43b4c1b2efdb158d011022b0 Oct 31 00:31:23.118365 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:23.118306 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:31:23.294107 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:23.294069 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:31:23.472676 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:23.472588 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:31:23.636216 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:23.636184 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-37abc0-lb5gt" event={"ID":"61edd758-f5e7-4ca0-a1df-ba7161712831","Type":"ContainerStarted","Data":"0ba35cb0714250ebc84d5c6163ce3349ce8d3baf1e41ffeb76cc5d797a164e6c"} Oct 31 00:31:23.636405 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:23.636233 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-37abc0-lb5gt" event={"ID":"61edd758-f5e7-4ca0-a1df-ba7161712831","Type":"ContainerStarted","Data":"87522efc17119cf8d125697d7b70a2f5832f51fc43b4c1b2efdb158d011022b0"} Oct 31 00:31:23.765995 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:23.765885 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-37abc0-lb5gt" podStartSLOduration=1.7658698419999999 podStartE2EDuration="1.765869842s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:31:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:31:23.763385693 +0000 UTC m=+2996.799716514" watchObservedRunningTime="2026-10-31 00:31:23.765869842 +0000 UTC m=+2996.802200660" Oct 31 00:31:24.644906 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:24.644869 4927 generic.go:356] "Generic (PLEG): container finished" podID="61edd758-f5e7-4ca0-a1df-ba7161712831" containerID="0ba35cb0714250ebc84d5c6163ce3349ce8d3baf1e41ffeb76cc5d797a164e6c" exitCode=0 Oct 31 00:31:24.645352 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:24.644912 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-37abc0-lb5gt" event={"ID":"61edd758-f5e7-4ca0-a1df-ba7161712831","Type":"ContainerDied","Data":"0ba35cb0714250ebc84d5c6163ce3349ce8d3baf1e41ffeb76cc5d797a164e6c"} Oct 31 00:31:25.796962 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:25.796936 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-37abc0-lb5gt" Oct 31 00:31:25.893394 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:25.893363 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/61edd758-f5e7-4ca0-a1df-ba7161712831-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61edd758-f5e7-4ca0-a1df-ba7161712831-maas-api-openshift-ca\") pod \"61edd758-f5e7-4ca0-a1df-ba7161712831\" (UID: \"61edd758-f5e7-4ca0-a1df-ba7161712831\") " Oct 31 00:31:25.893613 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:25.893432 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/61edd758-f5e7-4ca0-a1df-ba7161712831-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61edd758-f5e7-4ca0-a1df-ba7161712831-maas-api-xks-ca\") pod \"61edd758-f5e7-4ca0-a1df-ba7161712831\" (UID: \"61edd758-f5e7-4ca0-a1df-ba7161712831\") " Oct 31 00:31:25.893613 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:25.893511 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61edd758-f5e7-4ca0-a1df-ba7161712831-kube-api-access-2nkkb\" (UniqueName: \"kubernetes.io/projected/61edd758-f5e7-4ca0-a1df-ba7161712831-kube-api-access-2nkkb\") pod \"61edd758-f5e7-4ca0-a1df-ba7161712831\" (UID: \"61edd758-f5e7-4ca0-a1df-ba7161712831\") " Oct 31 00:31:25.893835 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:25.893807 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61edd758-f5e7-4ca0-a1df-ba7161712831-maas-api-openshift-ca" pod "61edd758-f5e7-4ca0-a1df-ba7161712831" (UID: "61edd758-f5e7-4ca0-a1df-ba7161712831"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:31:25.895641 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:25.895617 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/61edd758-f5e7-4ca0-a1df-ba7161712831-maas-api-xks-ca" pod "61edd758-f5e7-4ca0-a1df-ba7161712831" (UID: "61edd758-f5e7-4ca0-a1df-ba7161712831"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:31:25.895754 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:25.895674 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61edd758-f5e7-4ca0-a1df-ba7161712831-kube-api-access-2nkkb" pod "61edd758-f5e7-4ca0-a1df-ba7161712831" (UID: "61edd758-f5e7-4ca0-a1df-ba7161712831"). InnerVolumeSpecName "kube-api-access-2nkkb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:31:25.995099 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:25.995059 4927 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61edd758-f5e7-4ca0-a1df-ba7161712831-maas-api-openshift-ca\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:31:25.995099 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:25.995095 4927 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61edd758-f5e7-4ca0-a1df-ba7161712831-maas-api-xks-ca\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:31:25.995099 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:25.995106 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2nkkb\" (UniqueName: \"kubernetes.io/projected/61edd758-f5e7-4ca0-a1df-ba7161712831-kube-api-access-2nkkb\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:31:26.655720 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:26.655690 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-ddf61a-openshift-default-f57f54bc-4gvfk" Oct 31 00:31:26.658979 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:26.658951 4927 generic.go:356] "Generic (PLEG): container finished" podID="8a91e846-1ef4-4063-860e-46cdbf225ced" containerID="5f78ac7307756be3414509c8fea905737c9432c9e9f25c7f910b02b7bcb9c51e" exitCode=0 Oct 31 00:31:26.659103 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:26.658992 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-ddf61a-openshift-default-f57f54bc-4gvfk" Oct 31 00:31:26.659103 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:26.659027 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ddf61a-openshift-default-f57f54bc-4gvfk" event={"ID":"8a91e846-1ef4-4063-860e-46cdbf225ced","Type":"ContainerDied","Data":"5f78ac7307756be3414509c8fea905737c9432c9e9f25c7f910b02b7bcb9c51e"} Oct 31 00:31:26.659103 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:26.659078 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ddf61a-openshift-default-f57f54bc-4gvfk" event={"ID":"8a91e846-1ef4-4063-860e-46cdbf225ced","Type":"ContainerDied","Data":"5bcb39b4f13615bb376a993efcf7899a29402c386aded5bb45dd3316b316a884"} Oct 31 00:31:26.659103 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:26.659091 4927 scope.go:122] "RemoveContainer" containerID="5f78ac7307756be3414509c8fea905737c9432c9e9f25c7f910b02b7bcb9c51e" Oct 31 00:31:26.660510 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:26.660491 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-37abc0-lb5gt" Oct 31 00:31:26.660629 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:26.660528 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-37abc0-lb5gt" event={"ID":"61edd758-f5e7-4ca0-a1df-ba7161712831","Type":"ContainerDied","Data":"87522efc17119cf8d125697d7b70a2f5832f51fc43b4c1b2efdb158d011022b0"} Oct 31 00:31:26.660629 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:26.660570 4927 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="87522efc17119cf8d125697d7b70a2f5832f51fc43b4c1b2efdb158d011022b0" Oct 31 00:31:26.669053 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:26.669030 4927 scope.go:122] "RemoveContainer" containerID="75c18b5577f51dca639c1901a93d0bd173d4a1772389f80ba0fe87b53cab3d52" Oct 31 00:31:26.678096 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:26.678078 4927 scope.go:122] "RemoveContainer" containerID="5f78ac7307756be3414509c8fea905737c9432c9e9f25c7f910b02b7bcb9c51e" Oct 31 00:31:26.678340 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:31:26.678310 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f78ac7307756be3414509c8fea905737c9432c9e9f25c7f910b02b7bcb9c51e\": container with ID starting with 5f78ac7307756be3414509c8fea905737c9432c9e9f25c7f910b02b7bcb9c51e not found: ID does not exist" containerID="5f78ac7307756be3414509c8fea905737c9432c9e9f25c7f910b02b7bcb9c51e" Oct 31 00:31:26.678395 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:26.678346 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f78ac7307756be3414509c8fea905737c9432c9e9f25c7f910b02b7bcb9c51e"} err="failed to get container status \"5f78ac7307756be3414509c8fea905737c9432c9e9f25c7f910b02b7bcb9c51e\": rpc error: code = NotFound desc = could not find container \"5f78ac7307756be3414509c8fea905737c9432c9e9f25c7f910b02b7bcb9c51e\": container with ID starting with 5f78ac7307756be3414509c8fea905737c9432c9e9f25c7f910b02b7bcb9c51e not found: ID does not exist" Oct 31 00:31:26.678395 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:26.678364 4927 scope.go:122] "RemoveContainer" containerID="75c18b5577f51dca639c1901a93d0bd173d4a1772389f80ba0fe87b53cab3d52" Oct 31 00:31:26.678580 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:31:26.678567 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75c18b5577f51dca639c1901a93d0bd173d4a1772389f80ba0fe87b53cab3d52\": container with ID starting with 75c18b5577f51dca639c1901a93d0bd173d4a1772389f80ba0fe87b53cab3d52 not found: ID does not exist" containerID="75c18b5577f51dca639c1901a93d0bd173d4a1772389f80ba0fe87b53cab3d52" Oct 31 00:31:26.678622 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:26.678583 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75c18b5577f51dca639c1901a93d0bd173d4a1772389f80ba0fe87b53cab3d52"} err="failed to get container status \"75c18b5577f51dca639c1901a93d0bd173d4a1772389f80ba0fe87b53cab3d52\": rpc error: code = NotFound desc = could not find container \"75c18b5577f51dca639c1901a93d0bd173d4a1772389f80ba0fe87b53cab3d52\": container with ID starting with 75c18b5577f51dca639c1901a93d0bd173d4a1772389f80ba0fe87b53cab3d52 not found: ID does not exist" Oct 31 00:31:26.702943 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:26.702918 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a91e846-1ef4-4063-860e-46cdbf225ced-kube-api-access-s2g5r\" (UniqueName: \"kubernetes.io/projected/8a91e846-1ef4-4063-860e-46cdbf225ced-kube-api-access-s2g5r\") pod \"8a91e846-1ef4-4063-860e-46cdbf225ced\" (UID: \"8a91e846-1ef4-4063-860e-46cdbf225ced\") " Oct 31 00:31:26.703034 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:26.702962 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a91e846-1ef4-4063-860e-46cdbf225ced-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a91e846-1ef4-4063-860e-46cdbf225ced-istio-envoy\") pod \"8a91e846-1ef4-4063-860e-46cdbf225ced\" (UID: \"8a91e846-1ef4-4063-860e-46cdbf225ced\") " Oct 31 00:31:26.703034 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:26.702983 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a91e846-1ef4-4063-860e-46cdbf225ced-istio-token\" (UniqueName: \"kubernetes.io/projected/8a91e846-1ef4-4063-860e-46cdbf225ced-istio-token\") pod \"8a91e846-1ef4-4063-860e-46cdbf225ced\" (UID: \"8a91e846-1ef4-4063-860e-46cdbf225ced\") " Oct 31 00:31:26.703034 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:26.703016 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a91e846-1ef4-4063-860e-46cdbf225ced-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a91e846-1ef4-4063-860e-46cdbf225ced-credential-socket\") pod \"8a91e846-1ef4-4063-860e-46cdbf225ced\" (UID: \"8a91e846-1ef4-4063-860e-46cdbf225ced\") " Oct 31 00:31:26.703208 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:26.703054 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a91e846-1ef4-4063-860e-46cdbf225ced-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a91e846-1ef4-4063-860e-46cdbf225ced-workload-socket\") pod \"8a91e846-1ef4-4063-860e-46cdbf225ced\" (UID: \"8a91e846-1ef4-4063-860e-46cdbf225ced\") " Oct 31 00:31:26.703208 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:26.703088 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a91e846-1ef4-4063-860e-46cdbf225ced-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a91e846-1ef4-4063-860e-46cdbf225ced-istiod-ca-cert\") pod \"8a91e846-1ef4-4063-860e-46cdbf225ced\" (UID: \"8a91e846-1ef4-4063-860e-46cdbf225ced\") " Oct 31 00:31:26.703208 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:26.703115 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a91e846-1ef4-4063-860e-46cdbf225ced-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a91e846-1ef4-4063-860e-46cdbf225ced-istio-data\") pod \"8a91e846-1ef4-4063-860e-46cdbf225ced\" (UID: \"8a91e846-1ef4-4063-860e-46cdbf225ced\") " Oct 31 00:31:26.703208 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:26.703196 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a91e846-1ef4-4063-860e-46cdbf225ced-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a91e846-1ef4-4063-860e-46cdbf225ced-workload-certs\") pod \"8a91e846-1ef4-4063-860e-46cdbf225ced\" (UID: \"8a91e846-1ef4-4063-860e-46cdbf225ced\") " Oct 31 00:31:26.703456 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:26.703236 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8a91e846-1ef4-4063-860e-46cdbf225ced-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a91e846-1ef4-4063-860e-46cdbf225ced-istio-podinfo\") pod \"8a91e846-1ef4-4063-860e-46cdbf225ced\" (UID: \"8a91e846-1ef4-4063-860e-46cdbf225ced\") " Oct 31 00:31:26.703456 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:26.703391 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a91e846-1ef4-4063-860e-46cdbf225ced-credential-socket" pod "8a91e846-1ef4-4063-860e-46cdbf225ced" (UID: "8a91e846-1ef4-4063-860e-46cdbf225ced"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:31:26.703456 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:26.703404 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a91e846-1ef4-4063-860e-46cdbf225ced-workload-socket" pod "8a91e846-1ef4-4063-860e-46cdbf225ced" (UID: "8a91e846-1ef4-4063-860e-46cdbf225ced"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:31:26.703631 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:26.703521 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a91e846-1ef4-4063-860e-46cdbf225ced-istio-data" pod "8a91e846-1ef4-4063-860e-46cdbf225ced" (UID: "8a91e846-1ef4-4063-860e-46cdbf225ced"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:31:26.703631 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:26.703619 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a91e846-1ef4-4063-860e-46cdbf225ced-workload-certs" pod "8a91e846-1ef4-4063-860e-46cdbf225ced" (UID: "8a91e846-1ef4-4063-860e-46cdbf225ced"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:31:26.703747 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:26.703708 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a91e846-1ef4-4063-860e-46cdbf225ced-istiod-ca-cert" pod "8a91e846-1ef4-4063-860e-46cdbf225ced" (UID: "8a91e846-1ef4-4063-860e-46cdbf225ced"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:31:26.703747 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:26.703716 4927 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a91e846-1ef4-4063-860e-46cdbf225ced-workload-certs\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:31:26.703747 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:26.703742 4927 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a91e846-1ef4-4063-860e-46cdbf225ced-credential-socket\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:31:26.703992 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:26.703752 4927 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a91e846-1ef4-4063-860e-46cdbf225ced-workload-socket\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:31:26.703992 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:26.703762 4927 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a91e846-1ef4-4063-860e-46cdbf225ced-istio-data\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:31:26.705300 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:26.705280 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a91e846-1ef4-4063-860e-46cdbf225ced-istio-token" pod "8a91e846-1ef4-4063-860e-46cdbf225ced" (UID: "8a91e846-1ef4-4063-860e-46cdbf225ced"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:31:26.705433 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:26.705310 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8a91e846-1ef4-4063-860e-46cdbf225ced-istio-podinfo" pod "8a91e846-1ef4-4063-860e-46cdbf225ced" (UID: "8a91e846-1ef4-4063-860e-46cdbf225ced"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:31:26.705596 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:26.705577 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a91e846-1ef4-4063-860e-46cdbf225ced-istio-envoy" pod "8a91e846-1ef4-4063-860e-46cdbf225ced" (UID: "8a91e846-1ef4-4063-860e-46cdbf225ced"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:31:26.705661 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:26.705604 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a91e846-1ef4-4063-860e-46cdbf225ced-kube-api-access-s2g5r" pod "8a91e846-1ef4-4063-860e-46cdbf225ced" (UID: "8a91e846-1ef4-4063-860e-46cdbf225ced"). InnerVolumeSpecName "kube-api-access-s2g5r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:31:26.805212 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:26.805178 4927 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a91e846-1ef4-4063-860e-46cdbf225ced-istiod-ca-cert\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:31:26.805212 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:26.805207 4927 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a91e846-1ef4-4063-860e-46cdbf225ced-istio-podinfo\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:31:26.805212 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:26.805218 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s2g5r\" (UniqueName: \"kubernetes.io/projected/8a91e846-1ef4-4063-860e-46cdbf225ced-kube-api-access-s2g5r\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:31:26.805681 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:26.805228 4927 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a91e846-1ef4-4063-860e-46cdbf225ced-istio-envoy\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:31:26.805681 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:26.805237 4927 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a91e846-1ef4-4063-860e-46cdbf225ced-istio-token\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:31:27.088979 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:27.088943 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-ddf61a-openshift-default-f57f54bc-4gvfk"] Oct 31 00:31:27.088979 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:27.088983 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-ddf61a-openshift-default-f57f54bc-4gvfk"] Oct 31 00:31:28.420798 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:28.420764 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:31:28.747725 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:28.747653 4927 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-pqt2c_6382dfb5-a363-47a2-bc79-0b96f7bec0c3/console-operator/3.log" Oct 31 00:31:29.085858 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:29.085783 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a91e846-1ef4-4063-860e-46cdbf225ced" path="/var/lib/kubelet/pods/8a91e846-1ef4-4063-860e-46cdbf225ced/volumes" Oct 31 00:31:29.587975 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:29.587935 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-mz295"] Oct 31 00:31:29.588443 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:29.588147 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-mz295" podUID="c76ae81c-b5d0-4c9b-af8f-0a95d2aabf9f" containerName="istio-proxy" containerID="cri-o://f2735651c5eb3c78efea40369c5e52ff8edef4177ea7da229ac47810cbac400a" gracePeriod=30 Oct 31 00:31:29.766262 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:29.766232 4927 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-pqt2c_6382dfb5-a363-47a2-bc79-0b96f7bec0c3/console-operator/3.log" Oct 31 00:31:31.071458 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:31.071418 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:31:32.985515 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:32.985478 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:31:33.517207 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:33.517171 4927 scope.go:122] "RemoveContainer" containerID="e5e8167b7df2e3a35b2b4a67942ca163ec632caafc80cfb20f9422771ce42371" Oct 31 00:31:33.994430 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:33.994387 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:31:34.157655 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:34.154597 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:31:34.715749 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:34.715718 4927 generic.go:356] "Generic (PLEG): container finished" podID="c76ae81c-b5d0-4c9b-af8f-0a95d2aabf9f" containerID="f2735651c5eb3c78efea40369c5e52ff8edef4177ea7da229ac47810cbac400a" exitCode=0 Oct 31 00:31:34.715925 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:34.715793 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-mz295" event={"ID":"c76ae81c-b5d0-4c9b-af8f-0a95d2aabf9f","Type":"ContainerDied","Data":"f2735651c5eb3c78efea40369c5e52ff8edef4177ea7da229ac47810cbac400a"} Oct 31 00:31:34.869272 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:34.869205 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-mz295" Oct 31 00:31:34.983954 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:34.983914 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c76ae81c-b5d0-4c9b-af8f-0a95d2aabf9f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c76ae81c-b5d0-4c9b-af8f-0a95d2aabf9f-workload-socket\") pod \"c76ae81c-b5d0-4c9b-af8f-0a95d2aabf9f\" (UID: \"c76ae81c-b5d0-4c9b-af8f-0a95d2aabf9f\") " Oct 31 00:31:34.984149 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:34.984014 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c76ae81c-b5d0-4c9b-af8f-0a95d2aabf9f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c76ae81c-b5d0-4c9b-af8f-0a95d2aabf9f-istio-podinfo\") pod \"c76ae81c-b5d0-4c9b-af8f-0a95d2aabf9f\" (UID: \"c76ae81c-b5d0-4c9b-af8f-0a95d2aabf9f\") " Oct 31 00:31:34.984149 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:34.984066 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c76ae81c-b5d0-4c9b-af8f-0a95d2aabf9f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c76ae81c-b5d0-4c9b-af8f-0a95d2aabf9f-istiod-ca-cert\") pod \"c76ae81c-b5d0-4c9b-af8f-0a95d2aabf9f\" (UID: \"c76ae81c-b5d0-4c9b-af8f-0a95d2aabf9f\") " Oct 31 00:31:34.984149 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:34.984098 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c76ae81c-b5d0-4c9b-af8f-0a95d2aabf9f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c76ae81c-b5d0-4c9b-af8f-0a95d2aabf9f-workload-certs\") pod \"c76ae81c-b5d0-4c9b-af8f-0a95d2aabf9f\" (UID: \"c76ae81c-b5d0-4c9b-af8f-0a95d2aabf9f\") " Oct 31 00:31:34.984149 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:34.984140 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c76ae81c-b5d0-4c9b-af8f-0a95d2aabf9f-istio-token\" (UniqueName: \"kubernetes.io/projected/c76ae81c-b5d0-4c9b-af8f-0a95d2aabf9f-istio-token\") pod \"c76ae81c-b5d0-4c9b-af8f-0a95d2aabf9f\" (UID: \"c76ae81c-b5d0-4c9b-af8f-0a95d2aabf9f\") " Oct 31 00:31:34.984399 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:34.984169 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c76ae81c-b5d0-4c9b-af8f-0a95d2aabf9f-kube-api-access-6wgbs\" (UniqueName: \"kubernetes.io/projected/c76ae81c-b5d0-4c9b-af8f-0a95d2aabf9f-kube-api-access-6wgbs\") pod \"c76ae81c-b5d0-4c9b-af8f-0a95d2aabf9f\" (UID: \"c76ae81c-b5d0-4c9b-af8f-0a95d2aabf9f\") " Oct 31 00:31:34.984399 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:34.984189 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c76ae81c-b5d0-4c9b-af8f-0a95d2aabf9f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c76ae81c-b5d0-4c9b-af8f-0a95d2aabf9f-istio-data\") pod \"c76ae81c-b5d0-4c9b-af8f-0a95d2aabf9f\" (UID: \"c76ae81c-b5d0-4c9b-af8f-0a95d2aabf9f\") " Oct 31 00:31:34.984399 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:34.984212 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c76ae81c-b5d0-4c9b-af8f-0a95d2aabf9f-workload-socket" pod "c76ae81c-b5d0-4c9b-af8f-0a95d2aabf9f" (UID: "c76ae81c-b5d0-4c9b-af8f-0a95d2aabf9f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:31:34.984399 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:34.984224 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c76ae81c-b5d0-4c9b-af8f-0a95d2aabf9f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c76ae81c-b5d0-4c9b-af8f-0a95d2aabf9f-istio-envoy\") pod \"c76ae81c-b5d0-4c9b-af8f-0a95d2aabf9f\" (UID: \"c76ae81c-b5d0-4c9b-af8f-0a95d2aabf9f\") " Oct 31 00:31:34.984399 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:34.984361 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c76ae81c-b5d0-4c9b-af8f-0a95d2aabf9f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c76ae81c-b5d0-4c9b-af8f-0a95d2aabf9f-credential-socket\") pod \"c76ae81c-b5d0-4c9b-af8f-0a95d2aabf9f\" (UID: \"c76ae81c-b5d0-4c9b-af8f-0a95d2aabf9f\") " Oct 31 00:31:34.984674 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:34.984547 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c76ae81c-b5d0-4c9b-af8f-0a95d2aabf9f-istiod-ca-cert" pod "c76ae81c-b5d0-4c9b-af8f-0a95d2aabf9f" (UID: "c76ae81c-b5d0-4c9b-af8f-0a95d2aabf9f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:31:34.984731 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:34.984715 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c76ae81c-b5d0-4c9b-af8f-0a95d2aabf9f-credential-socket" pod "c76ae81c-b5d0-4c9b-af8f-0a95d2aabf9f" (UID: "c76ae81c-b5d0-4c9b-af8f-0a95d2aabf9f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:31:34.984839 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:34.984777 4927 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c76ae81c-b5d0-4c9b-af8f-0a95d2aabf9f-istiod-ca-cert\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:31:34.984839 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:34.984805 4927 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c76ae81c-b5d0-4c9b-af8f-0a95d2aabf9f-workload-socket\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:31:34.985047 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:34.984868 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c76ae81c-b5d0-4c9b-af8f-0a95d2aabf9f-workload-certs" pod "c76ae81c-b5d0-4c9b-af8f-0a95d2aabf9f" (UID: "c76ae81c-b5d0-4c9b-af8f-0a95d2aabf9f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:31:34.985047 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:34.985026 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c76ae81c-b5d0-4c9b-af8f-0a95d2aabf9f-istio-data" pod "c76ae81c-b5d0-4c9b-af8f-0a95d2aabf9f" (UID: "c76ae81c-b5d0-4c9b-af8f-0a95d2aabf9f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:31:34.991011 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:34.990968 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c76ae81c-b5d0-4c9b-af8f-0a95d2aabf9f-istio-envoy" pod "c76ae81c-b5d0-4c9b-af8f-0a95d2aabf9f" (UID: "c76ae81c-b5d0-4c9b-af8f-0a95d2aabf9f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:31:34.991906 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:34.991873 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c76ae81c-b5d0-4c9b-af8f-0a95d2aabf9f-istio-podinfo" pod "c76ae81c-b5d0-4c9b-af8f-0a95d2aabf9f" (UID: "c76ae81c-b5d0-4c9b-af8f-0a95d2aabf9f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:31:34.992707 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:34.992582 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c76ae81c-b5d0-4c9b-af8f-0a95d2aabf9f-kube-api-access-6wgbs" pod "c76ae81c-b5d0-4c9b-af8f-0a95d2aabf9f" (UID: "c76ae81c-b5d0-4c9b-af8f-0a95d2aabf9f"). InnerVolumeSpecName "kube-api-access-6wgbs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:31:34.992707 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:34.992599 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c76ae81c-b5d0-4c9b-af8f-0a95d2aabf9f-istio-token" pod "c76ae81c-b5d0-4c9b-af8f-0a95d2aabf9f" (UID: "c76ae81c-b5d0-4c9b-af8f-0a95d2aabf9f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:31:35.085567 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:35.085534 4927 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c76ae81c-b5d0-4c9b-af8f-0a95d2aabf9f-credential-socket\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:31:35.085567 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:35.085561 4927 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c76ae81c-b5d0-4c9b-af8f-0a95d2aabf9f-istio-podinfo\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:31:35.085567 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:35.085571 4927 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c76ae81c-b5d0-4c9b-af8f-0a95d2aabf9f-workload-certs\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:31:35.085822 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:35.085580 4927 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c76ae81c-b5d0-4c9b-af8f-0a95d2aabf9f-istio-token\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:31:35.085822 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:35.085589 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6wgbs\" (UniqueName: \"kubernetes.io/projected/c76ae81c-b5d0-4c9b-af8f-0a95d2aabf9f-kube-api-access-6wgbs\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:31:35.085822 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:35.085599 4927 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c76ae81c-b5d0-4c9b-af8f-0a95d2aabf9f-istio-data\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:31:35.085822 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:35.085608 4927 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c76ae81c-b5d0-4c9b-af8f-0a95d2aabf9f-istio-envoy\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:31:35.207025 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:35.206984 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:31:35.370026 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:35.369987 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:31:35.724532 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:35.724498 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-mz295" Oct 31 00:31:35.724738 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:35.724531 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-mz295" event={"ID":"c76ae81c-b5d0-4c9b-af8f-0a95d2aabf9f","Type":"ContainerDied","Data":"0e947dc1eb51d43213a3cb3682f5637c72a42f2f6b2ab978b8edc90aa3ea0071"} Oct 31 00:31:35.724738 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:35.724593 4927 scope.go:122] "RemoveContainer" containerID="f2735651c5eb3c78efea40369c5e52ff8edef4177ea7da229ac47810cbac400a" Oct 31 00:31:35.895092 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:35.895058 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-mz295"] Oct 31 00:31:35.946270 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:35.946234 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-mz295"] Oct 31 00:31:37.084964 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:37.084890 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c76ae81c-b5d0-4c9b-af8f-0a95d2aabf9f" path="/var/lib/kubelet/pods/c76ae81c-b5d0-4c9b-af8f-0a95d2aabf9f/volumes" Oct 31 00:31:37.577659 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:37.577621 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:31:41.587298 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:41.587261 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:31:41.849028 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:41.848952 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:31:42.184811 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:42.184775 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:31:46.185580 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:46.185541 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:31:47.689867 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:47.689833 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:31:47.892564 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:47.892528 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:31:48.052181 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:48.052096 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:31:48.150507 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:48.150472 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-37abc0-lb5gt"] Oct 31 00:31:48.201692 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:48.201657 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-37abc0-lb5gt"] Oct 31 00:31:48.592471 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:48.592436 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:31:49.086099 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:49.086061 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61edd758-f5e7-4ca0-a1df-ba7161712831" path="/var/lib/kubelet/pods/61edd758-f5e7-4ca0-a1df-ba7161712831/volumes" Oct 31 00:31:49.677168 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:31:49.677123 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sqv69" containerName="extractor" Oct 31 00:31:52.816887 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:52.816837 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:31:54.741539 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:54.741487 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:31:54.962593 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:54.962556 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:31:55.131540 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:55.131451 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-37abc0-openshift-default-65b888884f-pqjfl"] Oct 31 00:31:55.131727 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:55.131693 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-37abc0-openshift-default-65b888884f-pqjfl" podUID="84baa184-a53a-4f9f-8597-fd7c95e04d0d" containerName="istio-proxy" containerID="cri-o://98a82c5790d247e201ae11d6fb373788135e95863729e3435f0fa5b6ccd122b6" gracePeriod=30 Oct 31 00:31:55.171461 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:55.171423 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:31:55.324087 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:55.324046 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:31:55.585689 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:55.585654 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:31:56.118537 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:56.118497 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-5c3374-76h6v"] Oct 31 00:31:56.119036 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:56.118914 4927 cpu_manager.go:414] "containerMap: removing container" podUID="8a91e846-1ef4-4063-860e-46cdbf225ced" containerName="istio-proxy" Oct 31 00:31:56.119036 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:56.118926 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a91e846-1ef4-4063-860e-46cdbf225ced" containerName="istio-proxy" Oct 31 00:31:56.119036 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:56.118945 4927 cpu_manager.go:414] "containerMap: removing container" podUID="c76ae81c-b5d0-4c9b-af8f-0a95d2aabf9f" containerName="istio-proxy" Oct 31 00:31:56.119036 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:56.118950 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c76ae81c-b5d0-4c9b-af8f-0a95d2aabf9f" containerName="istio-proxy" Oct 31 00:31:56.119036 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:56.118964 4927 cpu_manager.go:414] "containerMap: removing container" podUID="61edd758-f5e7-4ca0-a1df-ba7161712831" containerName="revoke-keys" Oct 31 00:31:56.119036 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:56.118970 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61edd758-f5e7-4ca0-a1df-ba7161712831" containerName="revoke-keys" Oct 31 00:31:56.119036 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:56.119038 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="61edd758-f5e7-4ca0-a1df-ba7161712831" containerName="revoke-keys" Oct 31 00:31:56.119324 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:56.119052 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a91e846-1ef4-4063-860e-46cdbf225ced" containerName="istio-proxy" Oct 31 00:31:56.119324 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:56.119061 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a91e846-1ef4-4063-860e-46cdbf225ced" containerName="istio-proxy" Oct 31 00:31:56.119324 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:56.119069 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="c76ae81c-b5d0-4c9b-af8f-0a95d2aabf9f" containerName="istio-proxy" Oct 31 00:31:56.122425 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:56.122409 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-5c3374-76h6v" Oct 31 00:31:56.178143 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:56.178111 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:31:56.187949 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:56.187917 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7aec19d-427d-4215-bdda-6b5ee3ee6157-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-5c3374-76h6v\" (UID: \"a7aec19d-427d-4215-bdda-6b5ee3ee6157\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-5c3374-76h6v" Oct 31 00:31:56.188121 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:56.187985 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbvxg\" (UniqueName: \"kubernetes.io/projected/a7aec19d-427d-4215-bdda-6b5ee3ee6157-kube-api-access-dbvxg\") pod \"maas-api-revoke-keys-e2e-worker-w1-5c3374-76h6v\" (UID: \"a7aec19d-427d-4215-bdda-6b5ee3ee6157\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-5c3374-76h6v" Oct 31 00:31:56.188194 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:56.188162 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7aec19d-427d-4215-bdda-6b5ee3ee6157-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-5c3374-76h6v\" (UID: \"a7aec19d-427d-4215-bdda-6b5ee3ee6157\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-5c3374-76h6v" Oct 31 00:31:56.229275 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:56.229243 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4fdzs\"" Oct 31 00:31:56.232131 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:56.232103 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-5c3374-76h6v"] Oct 31 00:31:56.288966 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:56.288935 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7aec19d-427d-4215-bdda-6b5ee3ee6157-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-5c3374-76h6v\" (UID: \"a7aec19d-427d-4215-bdda-6b5ee3ee6157\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-5c3374-76h6v" Oct 31 00:31:56.289132 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:56.289020 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7aec19d-427d-4215-bdda-6b5ee3ee6157-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-5c3374-76h6v\" (UID: \"a7aec19d-427d-4215-bdda-6b5ee3ee6157\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-5c3374-76h6v" Oct 31 00:31:56.289132 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:56.289064 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbvxg\" (UniqueName: \"kubernetes.io/projected/a7aec19d-427d-4215-bdda-6b5ee3ee6157-kube-api-access-dbvxg\") pod \"maas-api-revoke-keys-e2e-worker-w1-5c3374-76h6v\" (UID: \"a7aec19d-427d-4215-bdda-6b5ee3ee6157\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-5c3374-76h6v" Oct 31 00:31:56.289791 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:56.289771 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7aec19d-427d-4215-bdda-6b5ee3ee6157-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-5c3374-76h6v\" (UID: \"a7aec19d-427d-4215-bdda-6b5ee3ee6157\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-5c3374-76h6v" Oct 31 00:31:56.291372 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:56.291355 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7aec19d-427d-4215-bdda-6b5ee3ee6157-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-5c3374-76h6v\" (UID: \"a7aec19d-427d-4215-bdda-6b5ee3ee6157\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-5c3374-76h6v" Oct 31 00:31:56.351527 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:56.351496 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbvxg\" (UniqueName: \"kubernetes.io/projected/a7aec19d-427d-4215-bdda-6b5ee3ee6157-kube-api-access-dbvxg\") pod \"maas-api-revoke-keys-e2e-worker-w1-5c3374-76h6v\" (UID: \"a7aec19d-427d-4215-bdda-6b5ee3ee6157\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-5c3374-76h6v" Oct 31 00:31:56.436138 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:56.436104 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-5c3374-76h6v" Oct 31 00:31:56.815398 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:56.815372 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-5c3374-76h6v"] Oct 31 00:31:56.817441 ip-10-0-141-180 kubenswrapper[4927]: W1031 00:31:56.817410 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7aec19d_427d_4215_bdda_6b5ee3ee6157.slice/crio-be22b3e90522b8795cd551d44f24eff0dfa47fdf5aa33cf6ce8840e06330f03f WatchSource:0}: Error finding container be22b3e90522b8795cd551d44f24eff0dfa47fdf5aa33cf6ce8840e06330f03f: Status 404 returned error can't find the container with id be22b3e90522b8795cd551d44f24eff0dfa47fdf5aa33cf6ce8840e06330f03f Oct 31 00:31:56.871443 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:56.871413 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-5c3374-76h6v" event={"ID":"a7aec19d-427d-4215-bdda-6b5ee3ee6157","Type":"ContainerStarted","Data":"be22b3e90522b8795cd551d44f24eff0dfa47fdf5aa33cf6ce8840e06330f03f"} Oct 31 00:31:57.502166 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:57.502131 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:31:57.879294 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:57.879214 4927 generic.go:356] "Generic (PLEG): container finished" podID="a7aec19d-427d-4215-bdda-6b5ee3ee6157" containerID="d7c747cdf240a8304fa94db7b1f3b468917aa96a8436afc6cbf1d3726ebb5441" exitCode=0 Oct 31 00:31:57.879510 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:57.879301 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-5c3374-76h6v" event={"ID":"a7aec19d-427d-4215-bdda-6b5ee3ee6157","Type":"ContainerDied","Data":"d7c747cdf240a8304fa94db7b1f3b468917aa96a8436afc6cbf1d3726ebb5441"} Oct 31 00:31:59.022488 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:59.022454 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-5c3374-76h6v" Oct 31 00:31:59.117050 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:59.117013 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7aec19d-427d-4215-bdda-6b5ee3ee6157-kube-api-access-dbvxg\" (UniqueName: \"kubernetes.io/projected/a7aec19d-427d-4215-bdda-6b5ee3ee6157-kube-api-access-dbvxg\") pod \"a7aec19d-427d-4215-bdda-6b5ee3ee6157\" (UID: \"a7aec19d-427d-4215-bdda-6b5ee3ee6157\") " Oct 31 00:31:59.117258 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:59.117105 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a7aec19d-427d-4215-bdda-6b5ee3ee6157-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7aec19d-427d-4215-bdda-6b5ee3ee6157-maas-api-xks-ca\") pod \"a7aec19d-427d-4215-bdda-6b5ee3ee6157\" (UID: \"a7aec19d-427d-4215-bdda-6b5ee3ee6157\") " Oct 31 00:31:59.117258 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:59.117236 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a7aec19d-427d-4215-bdda-6b5ee3ee6157-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7aec19d-427d-4215-bdda-6b5ee3ee6157-maas-api-openshift-ca\") pod \"a7aec19d-427d-4215-bdda-6b5ee3ee6157\" (UID: \"a7aec19d-427d-4215-bdda-6b5ee3ee6157\") " Oct 31 00:31:59.118525 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:59.118492 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7aec19d-427d-4215-bdda-6b5ee3ee6157-maas-api-openshift-ca" pod "a7aec19d-427d-4215-bdda-6b5ee3ee6157" (UID: "a7aec19d-427d-4215-bdda-6b5ee3ee6157"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:31:59.120370 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:59.120305 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7aec19d-427d-4215-bdda-6b5ee3ee6157-kube-api-access-dbvxg" pod "a7aec19d-427d-4215-bdda-6b5ee3ee6157" (UID: "a7aec19d-427d-4215-bdda-6b5ee3ee6157"). InnerVolumeSpecName "kube-api-access-dbvxg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:31:59.120945 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:59.120917 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a7aec19d-427d-4215-bdda-6b5ee3ee6157-maas-api-xks-ca" pod "a7aec19d-427d-4215-bdda-6b5ee3ee6157" (UID: "a7aec19d-427d-4215-bdda-6b5ee3ee6157"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:31:59.218557 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:59.218519 4927 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7aec19d-427d-4215-bdda-6b5ee3ee6157-maas-api-openshift-ca\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:31:59.218557 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:59.218551 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dbvxg\" (UniqueName: \"kubernetes.io/projected/a7aec19d-427d-4215-bdda-6b5ee3ee6157-kube-api-access-dbvxg\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:31:59.218557 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:59.218562 4927 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7aec19d-427d-4215-bdda-6b5ee3ee6157-maas-api-xks-ca\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:31:59.893772 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:59.893743 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-5c3374-76h6v" Oct 31 00:31:59.893959 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:59.893747 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-5c3374-76h6v" event={"ID":"a7aec19d-427d-4215-bdda-6b5ee3ee6157","Type":"ContainerDied","Data":"be22b3e90522b8795cd551d44f24eff0dfa47fdf5aa33cf6ce8840e06330f03f"} Oct 31 00:31:59.893959 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:31:59.893884 4927 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="be22b3e90522b8795cd551d44f24eff0dfa47fdf5aa33cf6ce8840e06330f03f" Oct 31 00:32:00.389953 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:32:00.389929 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-37abc0-openshift-default-65b888884f-pqjfl" Oct 31 00:32:00.430091 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:32:00.430068 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84baa184-a53a-4f9f-8597-fd7c95e04d0d-istio-token\" (UniqueName: \"kubernetes.io/projected/84baa184-a53a-4f9f-8597-fd7c95e04d0d-istio-token\") pod \"84baa184-a53a-4f9f-8597-fd7c95e04d0d\" (UID: \"84baa184-a53a-4f9f-8597-fd7c95e04d0d\") " Oct 31 00:32:00.430216 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:32:00.430123 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84baa184-a53a-4f9f-8597-fd7c95e04d0d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84baa184-a53a-4f9f-8597-fd7c95e04d0d-istiod-ca-cert\") pod \"84baa184-a53a-4f9f-8597-fd7c95e04d0d\" (UID: \"84baa184-a53a-4f9f-8597-fd7c95e04d0d\") " Oct 31 00:32:00.430216 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:32:00.430152 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84baa184-a53a-4f9f-8597-fd7c95e04d0d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84baa184-a53a-4f9f-8597-fd7c95e04d0d-credential-socket\") pod \"84baa184-a53a-4f9f-8597-fd7c95e04d0d\" (UID: \"84baa184-a53a-4f9f-8597-fd7c95e04d0d\") " Oct 31 00:32:00.430304 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:32:00.430256 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84baa184-a53a-4f9f-8597-fd7c95e04d0d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84baa184-a53a-4f9f-8597-fd7c95e04d0d-workload-socket\") pod \"84baa184-a53a-4f9f-8597-fd7c95e04d0d\" (UID: \"84baa184-a53a-4f9f-8597-fd7c95e04d0d\") " Oct 31 00:32:00.430384 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:32:00.430308 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84baa184-a53a-4f9f-8597-fd7c95e04d0d-kube-api-access-8tb6t\" (UniqueName: \"kubernetes.io/projected/84baa184-a53a-4f9f-8597-fd7c95e04d0d-kube-api-access-8tb6t\") pod \"84baa184-a53a-4f9f-8597-fd7c95e04d0d\" (UID: \"84baa184-a53a-4f9f-8597-fd7c95e04d0d\") " Oct 31 00:32:00.430384 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:32:00.430371 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84baa184-a53a-4f9f-8597-fd7c95e04d0d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84baa184-a53a-4f9f-8597-fd7c95e04d0d-istio-envoy\") pod \"84baa184-a53a-4f9f-8597-fd7c95e04d0d\" (UID: \"84baa184-a53a-4f9f-8597-fd7c95e04d0d\") " Oct 31 00:32:00.430520 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:32:00.430416 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/84baa184-a53a-4f9f-8597-fd7c95e04d0d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84baa184-a53a-4f9f-8597-fd7c95e04d0d-istio-podinfo\") pod \"84baa184-a53a-4f9f-8597-fd7c95e04d0d\" (UID: \"84baa184-a53a-4f9f-8597-fd7c95e04d0d\") " Oct 31 00:32:00.430520 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:32:00.430433 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84baa184-a53a-4f9f-8597-fd7c95e04d0d-credential-socket" pod "84baa184-a53a-4f9f-8597-fd7c95e04d0d" (UID: "84baa184-a53a-4f9f-8597-fd7c95e04d0d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:32:00.430634 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:32:00.430511 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84baa184-a53a-4f9f-8597-fd7c95e04d0d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84baa184-a53a-4f9f-8597-fd7c95e04d0d-workload-certs\") pod \"84baa184-a53a-4f9f-8597-fd7c95e04d0d\" (UID: \"84baa184-a53a-4f9f-8597-fd7c95e04d0d\") " Oct 31 00:32:00.430634 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:32:00.430561 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84baa184-a53a-4f9f-8597-fd7c95e04d0d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84baa184-a53a-4f9f-8597-fd7c95e04d0d-istio-data\") pod \"84baa184-a53a-4f9f-8597-fd7c95e04d0d\" (UID: \"84baa184-a53a-4f9f-8597-fd7c95e04d0d\") " Oct 31 00:32:00.430634 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:32:00.430581 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84baa184-a53a-4f9f-8597-fd7c95e04d0d-workload-socket" pod "84baa184-a53a-4f9f-8597-fd7c95e04d0d" (UID: "84baa184-a53a-4f9f-8597-fd7c95e04d0d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:32:00.430634 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:32:00.430610 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84baa184-a53a-4f9f-8597-fd7c95e04d0d-istiod-ca-cert" pod "84baa184-a53a-4f9f-8597-fd7c95e04d0d" (UID: "84baa184-a53a-4f9f-8597-fd7c95e04d0d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:32:00.430858 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:32:00.430796 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84baa184-a53a-4f9f-8597-fd7c95e04d0d-workload-certs" pod "84baa184-a53a-4f9f-8597-fd7c95e04d0d" (UID: "84baa184-a53a-4f9f-8597-fd7c95e04d0d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:32:00.430923 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:32:00.430884 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84baa184-a53a-4f9f-8597-fd7c95e04d0d-istio-data" pod "84baa184-a53a-4f9f-8597-fd7c95e04d0d" (UID: "84baa184-a53a-4f9f-8597-fd7c95e04d0d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:32:00.430923 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:32:00.430901 4927 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84baa184-a53a-4f9f-8597-fd7c95e04d0d-workload-certs\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:32:00.431035 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:32:00.430922 4927 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84baa184-a53a-4f9f-8597-fd7c95e04d0d-istiod-ca-cert\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:32:00.431035 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:32:00.430937 4927 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84baa184-a53a-4f9f-8597-fd7c95e04d0d-credential-socket\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:32:00.431035 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:32:00.430951 4927 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84baa184-a53a-4f9f-8597-fd7c95e04d0d-workload-socket\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:32:00.432625 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:32:00.432594 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/84baa184-a53a-4f9f-8597-fd7c95e04d0d-istio-podinfo" pod "84baa184-a53a-4f9f-8597-fd7c95e04d0d" (UID: "84baa184-a53a-4f9f-8597-fd7c95e04d0d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:32:00.432745 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:32:00.432628 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84baa184-a53a-4f9f-8597-fd7c95e04d0d-istio-token" pod "84baa184-a53a-4f9f-8597-fd7c95e04d0d" (UID: "84baa184-a53a-4f9f-8597-fd7c95e04d0d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:32:00.432886 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:32:00.432864 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84baa184-a53a-4f9f-8597-fd7c95e04d0d-istio-envoy" pod "84baa184-a53a-4f9f-8597-fd7c95e04d0d" (UID: "84baa184-a53a-4f9f-8597-fd7c95e04d0d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:32:00.432935 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:32:00.432917 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84baa184-a53a-4f9f-8597-fd7c95e04d0d-kube-api-access-8tb6t" pod "84baa184-a53a-4f9f-8597-fd7c95e04d0d" (UID: "84baa184-a53a-4f9f-8597-fd7c95e04d0d"). InnerVolumeSpecName "kube-api-access-8tb6t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:32:00.531488 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:32:00.531451 4927 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84baa184-a53a-4f9f-8597-fd7c95e04d0d-istio-data\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:32:00.531488 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:32:00.531485 4927 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84baa184-a53a-4f9f-8597-fd7c95e04d0d-istio-token\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:32:00.531712 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:32:00.531501 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8tb6t\" (UniqueName: \"kubernetes.io/projected/84baa184-a53a-4f9f-8597-fd7c95e04d0d-kube-api-access-8tb6t\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:32:00.531712 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:32:00.531517 4927 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84baa184-a53a-4f9f-8597-fd7c95e04d0d-istio-envoy\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:32:00.531712 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:32:00.531526 4927 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84baa184-a53a-4f9f-8597-fd7c95e04d0d-istio-podinfo\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:32:00.553296 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:32:00.553256 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-5c3374/premium-w1-5c3374-kserve-6c56d4dcd-llwcl"] Oct 31 00:32:00.670279 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:32:00.670242 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:32:00.901354 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:32:00.901252 4927 generic.go:356] "Generic (PLEG): container finished" podID="84baa184-a53a-4f9f-8597-fd7c95e04d0d" containerID="98a82c5790d247e201ae11d6fb373788135e95863729e3435f0fa5b6ccd122b6" exitCode=0 Oct 31 00:32:00.901354 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:32:00.901304 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-37abc0-openshift-default-65b888884f-pqjfl" Oct 31 00:32:00.901582 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:32:00.901353 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-37abc0-openshift-default-65b888884f-pqjfl" event={"ID":"84baa184-a53a-4f9f-8597-fd7c95e04d0d","Type":"ContainerDied","Data":"98a82c5790d247e201ae11d6fb373788135e95863729e3435f0fa5b6ccd122b6"} Oct 31 00:32:00.901582 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:32:00.901402 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-37abc0-openshift-default-65b888884f-pqjfl" event={"ID":"84baa184-a53a-4f9f-8597-fd7c95e04d0d","Type":"ContainerDied","Data":"2d079763b5f56d836caf12656cc34019e9d9d0e3df1d94081507ace72c9f18c3"} Oct 31 00:32:00.901582 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:32:00.901427 4927 scope.go:122] "RemoveContainer" containerID="98a82c5790d247e201ae11d6fb373788135e95863729e3435f0fa5b6ccd122b6" Oct 31 00:32:00.912737 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:32:00.912715 4927 scope.go:122] "RemoveContainer" containerID="15a2819b990fb45280a7b8b986029e7fb518a58ba0c3030b8b70906d3f091550" Oct 31 00:32:00.922201 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:32:00.922182 4927 scope.go:122] "RemoveContainer" containerID="98a82c5790d247e201ae11d6fb373788135e95863729e3435f0fa5b6ccd122b6" Oct 31 00:32:00.922493 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:32:00.922475 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"98a82c5790d247e201ae11d6fb373788135e95863729e3435f0fa5b6ccd122b6\": container with ID starting with 98a82c5790d247e201ae11d6fb373788135e95863729e3435f0fa5b6ccd122b6 not found: ID does not exist" containerID="98a82c5790d247e201ae11d6fb373788135e95863729e3435f0fa5b6ccd122b6" Oct 31 00:32:00.922552 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:32:00.922504 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98a82c5790d247e201ae11d6fb373788135e95863729e3435f0fa5b6ccd122b6"} err="failed to get container status \"98a82c5790d247e201ae11d6fb373788135e95863729e3435f0fa5b6ccd122b6\": rpc error: code = NotFound desc = could not find container \"98a82c5790d247e201ae11d6fb373788135e95863729e3435f0fa5b6ccd122b6\": container with ID starting with 98a82c5790d247e201ae11d6fb373788135e95863729e3435f0fa5b6ccd122b6 not found: ID does not exist" Oct 31 00:32:00.922552 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:32:00.922522 4927 scope.go:122] "RemoveContainer" containerID="15a2819b990fb45280a7b8b986029e7fb518a58ba0c3030b8b70906d3f091550" Oct 31 00:32:00.922756 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:32:00.922740 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15a2819b990fb45280a7b8b986029e7fb518a58ba0c3030b8b70906d3f091550\": container with ID starting with 15a2819b990fb45280a7b8b986029e7fb518a58ba0c3030b8b70906d3f091550 not found: ID does not exist" containerID="15a2819b990fb45280a7b8b986029e7fb518a58ba0c3030b8b70906d3f091550" Oct 31 00:32:00.922799 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:32:00.922760 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15a2819b990fb45280a7b8b986029e7fb518a58ba0c3030b8b70906d3f091550"} err="failed to get container status \"15a2819b990fb45280a7b8b986029e7fb518a58ba0c3030b8b70906d3f091550\": rpc error: code = NotFound desc = could not find container \"15a2819b990fb45280a7b8b986029e7fb518a58ba0c3030b8b70906d3f091550\": container with ID starting with 15a2819b990fb45280a7b8b986029e7fb518a58ba0c3030b8b70906d3f091550 not found: ID does not exist" Oct 31 00:32:01.072450 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:32:01.072416 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-37abc0-openshift-default-65b888884f-pqjfl"] Oct 31 00:32:01.076730 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:32:01.076702 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-37abc0-openshift-default-65b888884f-pqjfl"] Oct 31 00:32:01.085733 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:32:01.085704 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84baa184-a53a-4f9f-8597-fd7c95e04d0d" path="/var/lib/kubelet/pods/84baa184-a53a-4f9f-8597-fd7c95e04d0d/volumes" Oct 31 00:32:02.401857 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:32:02.401818 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:32:07.004497 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:32:07.004456 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:32:07.506078 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:32:07.506047 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-5c3374-7969595c9d-gzhtd"] Oct 31 00:32:07.604951 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:32:07.604914 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-5c3374-f56fc6f79-7b9pp"] Oct 31 00:32:07.853852 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:32:07.853777 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:32:08.007102 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:32:08.007049 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:32:11.021086 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:32:11.021049 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5014c66a-mg9r4"] Oct 31 00:32:11.021549 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:32:11.021533 4927 cpu_manager.go:414] "containerMap: removing container" podUID="a7aec19d-427d-4215-bdda-6b5ee3ee6157" containerName="revoke-keys" Oct 31 00:32:11.021598 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:32:11.021550 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7aec19d-427d-4215-bdda-6b5ee3ee6157" containerName="revoke-keys" Oct 31 00:32:11.021598 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:32:11.021566 4927 cpu_manager.go:414] "containerMap: removing container" podUID="8a91e846-1ef4-4063-860e-46cdbf225ced" containerName="istio-proxy" Oct 31 00:32:11.021598 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:32:11.021572 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a91e846-1ef4-4063-860e-46cdbf225ced" containerName="istio-proxy" Oct 31 00:32:11.021598 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:32:11.021583 4927 cpu_manager.go:414] "containerMap: removing container" podUID="84baa184-a53a-4f9f-8597-fd7c95e04d0d" containerName="istio-proxy" Oct 31 00:32:11.021598 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:32:11.021587 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84baa184-a53a-4f9f-8597-fd7c95e04d0d" containerName="istio-proxy" Oct 31 00:32:11.021770 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:32:11.021656 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7aec19d-427d-4215-bdda-6b5ee3ee6157" containerName="revoke-keys" Oct 31 00:32:11.021770 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:32:11.021665 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="84baa184-a53a-4f9f-8597-fd7c95e04d0d" containerName="istio-proxy" Oct 31 00:32:11.021770 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:32:11.021674 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="84baa184-a53a-4f9f-8597-fd7c95e04d0d" containerName="istio-proxy" Oct 31 00:32:11.024780 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:32:11.024764 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5014c66a-mg9r4" Oct 31 00:32:11.086451 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:32:11.086420 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5014c66a-mg9r4"] Oct 31 00:32:11.128321 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:32:11.128285 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:32:11.128508 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:32:11.128350 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4fdzs\"" Oct 31 00:32:11.135177 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:32:11.135147 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-szr2j\" (UniqueName: \"kubernetes.io/projected/d0c77474-1554-4f73-84ef-524855a5160a-kube-api-access-szr2j\") pod \"maas-api-revoke-sub-5014c66a-mg9r4\" (UID: \"d0c77474-1554-4f73-84ef-524855a5160a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5014c66a-mg9r4" Oct 31 00:32:11.135370 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:32:11.135311 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0c77474-1554-4f73-84ef-524855a5160a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5014c66a-mg9r4\" (UID: \"d0c77474-1554-4f73-84ef-524855a5160a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5014c66a-mg9r4" Oct 31 00:32:11.135463 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:32:11.135453 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0c77474-1554-4f73-84ef-524855a5160a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5014c66a-mg9r4\" (UID: \"d0c77474-1554-4f73-84ef-524855a5160a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5014c66a-mg9r4" Oct 31 00:32:11.236703 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:32:11.236670 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0c77474-1554-4f73-84ef-524855a5160a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5014c66a-mg9r4\" (UID: \"d0c77474-1554-4f73-84ef-524855a5160a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5014c66a-mg9r4" Oct 31 00:32:11.236878 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:32:11.236724 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0c77474-1554-4f73-84ef-524855a5160a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5014c66a-mg9r4\" (UID: \"d0c77474-1554-4f73-84ef-524855a5160a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5014c66a-mg9r4" Oct 31 00:32:11.236878 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:32:11.236778 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szr2j\" (UniqueName: \"kubernetes.io/projected/d0c77474-1554-4f73-84ef-524855a5160a-kube-api-access-szr2j\") pod \"maas-api-revoke-sub-5014c66a-mg9r4\" (UID: \"d0c77474-1554-4f73-84ef-524855a5160a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5014c66a-mg9r4" Oct 31 00:32:11.237529 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:32:11.237511 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0c77474-1554-4f73-84ef-524855a5160a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5014c66a-mg9r4\" (UID: \"d0c77474-1554-4f73-84ef-524855a5160a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5014c66a-mg9r4" Oct 31 00:32:11.239086 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:32:11.239062 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0c77474-1554-4f73-84ef-524855a5160a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5014c66a-mg9r4\" (UID: \"d0c77474-1554-4f73-84ef-524855a5160a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5014c66a-mg9r4" Oct 31 00:32:11.300687 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:32:11.300620 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-szr2j\" (UniqueName: \"kubernetes.io/projected/d0c77474-1554-4f73-84ef-524855a5160a-kube-api-access-szr2j\") pod \"maas-api-revoke-sub-5014c66a-mg9r4\" (UID: \"d0c77474-1554-4f73-84ef-524855a5160a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5014c66a-mg9r4" Oct 31 00:32:11.337561 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:32:11.337531 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5014c66a-mg9r4" Oct 31 00:32:11.717647 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:32:11.717608 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5014c66a-mg9r4"] Oct 31 00:32:11.719056 ip-10-0-141-180 kubenswrapper[4927]: W1031 00:32:11.719026 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0c77474_1554_4f73_84ef_524855a5160a.slice/crio-477abc5c12c33c9aa55c4b6f47b4d4f422e83d79518682be611fb165e2083292 WatchSource:0}: Error finding container 477abc5c12c33c9aa55c4b6f47b4d4f422e83d79518682be611fb165e2083292: Status 404 returned error can't find the container with id 477abc5c12c33c9aa55c4b6f47b4d4f422e83d79518682be611fb165e2083292 Oct 31 00:32:11.974051 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:32:11.973971 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5014c66a-mg9r4" event={"ID":"d0c77474-1554-4f73-84ef-524855a5160a","Type":"ContainerStarted","Data":"477abc5c12c33c9aa55c4b6f47b4d4f422e83d79518682be611fb165e2083292"} Oct 31 00:32:12.507116 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:32:12.507063 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w1-5c3374-7969595c9d-gzhtd" podUID="2a65bb94-e7dd-4240-af0c-4ce7c24bc599" containerName="payload-processing" containerID="cri-o://ad51b490904e9d709e16972f3397ee5aacb4b22fd5ecccd4ac695bf49d1c7695" gracePeriod=25 Oct 31 00:32:12.605265 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:32:12.605223 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5c3374-f56fc6f79-7b9pp" podUID="f5c1c1eb-bce5-4ef6-8246-c0638f5c8b6b" containerName="payload-pre-processing" containerID="cri-o://c672e5ecdf6d33493ca13caa3595740645f4f791d76e997b3f60a01689a07573" gracePeriod=25 Oct 31 00:32:12.767852 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:32:12.767782 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-5c3374-7969595c9d-gzhtd" Oct 31 00:32:12.854127 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:32:12.854095 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a65bb94-e7dd-4240-af0c-4ce7c24bc599-kube-api-access-8qn57\" (UniqueName: \"kubernetes.io/projected/2a65bb94-e7dd-4240-af0c-4ce7c24bc599-kube-api-access-8qn57\") pod \"2a65bb94-e7dd-4240-af0c-4ce7c24bc599\" (UID: \"2a65bb94-e7dd-4240-af0c-4ce7c24bc599\") " Oct 31 00:32:12.854288 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:32:12.854140 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a65bb94-e7dd-4240-af0c-4ce7c24bc599-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a65bb94-e7dd-4240-af0c-4ce7c24bc599-plugins-config-volume\") pod \"2a65bb94-e7dd-4240-af0c-4ce7c24bc599\" (UID: \"2a65bb94-e7dd-4240-af0c-4ce7c24bc599\") " Oct 31 00:32:12.854586 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:32:12.854552 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a65bb94-e7dd-4240-af0c-4ce7c24bc599-plugins-config-volume" pod "2a65bb94-e7dd-4240-af0c-4ce7c24bc599" (UID: "2a65bb94-e7dd-4240-af0c-4ce7c24bc599"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:32:12.856379 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:32:12.856353 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a65bb94-e7dd-4240-af0c-4ce7c24bc599-kube-api-access-8qn57" pod "2a65bb94-e7dd-4240-af0c-4ce7c24bc599" (UID: "2a65bb94-e7dd-4240-af0c-4ce7c24bc599"). InnerVolumeSpecName "kube-api-access-8qn57". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:32:12.863956 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:32:12.863933 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5c3374-f56fc6f79-7b9pp" Oct 31 00:32:12.955716 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:32:12.955684 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5c1c1eb-bce5-4ef6-8246-c0638f5c8b6b-kube-api-access-xz9mx\" (UniqueName: \"kubernetes.io/projected/f5c1c1eb-bce5-4ef6-8246-c0638f5c8b6b-kube-api-access-xz9mx\") pod \"f5c1c1eb-bce5-4ef6-8246-c0638f5c8b6b\" (UID: \"f5c1c1eb-bce5-4ef6-8246-c0638f5c8b6b\") " Oct 31 00:32:12.955893 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:32:12.955822 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f5c1c1eb-bce5-4ef6-8246-c0638f5c8b6b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5c1c1eb-bce5-4ef6-8246-c0638f5c8b6b-plugins-config-volume\") pod \"f5c1c1eb-bce5-4ef6-8246-c0638f5c8b6b\" (UID: \"f5c1c1eb-bce5-4ef6-8246-c0638f5c8b6b\") " Oct 31 00:32:12.956031 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:32:12.956019 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8qn57\" (UniqueName: \"kubernetes.io/projected/2a65bb94-e7dd-4240-af0c-4ce7c24bc599-kube-api-access-8qn57\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:32:12.956069 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:32:12.956032 4927 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a65bb94-e7dd-4240-af0c-4ce7c24bc599-plugins-config-volume\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:32:12.956178 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:32:12.956158 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5c1c1eb-bce5-4ef6-8246-c0638f5c8b6b-plugins-config-volume" pod "f5c1c1eb-bce5-4ef6-8246-c0638f5c8b6b" (UID: "f5c1c1eb-bce5-4ef6-8246-c0638f5c8b6b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:32:12.957769 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:32:12.957747 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5c1c1eb-bce5-4ef6-8246-c0638f5c8b6b-kube-api-access-xz9mx" pod "f5c1c1eb-bce5-4ef6-8246-c0638f5c8b6b" (UID: "f5c1c1eb-bce5-4ef6-8246-c0638f5c8b6b"). InnerVolumeSpecName "kube-api-access-xz9mx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:32:12.981710 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:32:12.981680 4927 generic.go:356] "Generic (PLEG): container finished" podID="f5c1c1eb-bce5-4ef6-8246-c0638f5c8b6b" containerID="c672e5ecdf6d33493ca13caa3595740645f4f791d76e997b3f60a01689a07573" exitCode=0 Oct 31 00:32:12.981866 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:32:12.981725 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5c3374-f56fc6f79-7b9pp" Oct 31 00:32:12.981866 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:32:12.981770 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5c3374-f56fc6f79-7b9pp" event={"ID":"f5c1c1eb-bce5-4ef6-8246-c0638f5c8b6b","Type":"ContainerDied","Data":"c672e5ecdf6d33493ca13caa3595740645f4f791d76e997b3f60a01689a07573"} Oct 31 00:32:12.981866 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:32:12.981829 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5c3374-f56fc6f79-7b9pp" event={"ID":"f5c1c1eb-bce5-4ef6-8246-c0638f5c8b6b","Type":"ContainerDied","Data":"b23f7b1a3caf94a6985bc5d55abc05ba7b6bf1e8df6271fd86e359f82377aae3"} Oct 31 00:32:12.981866 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:32:12.981858 4927 scope.go:122] "RemoveContainer" containerID="c672e5ecdf6d33493ca13caa3595740645f4f791d76e997b3f60a01689a07573" Oct 31 00:32:12.983634 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:32:12.983610 4927 generic.go:356] "Generic (PLEG): container finished" podID="d0c77474-1554-4f73-84ef-524855a5160a" containerID="af6517da8f87f0cd9fdd668c6516e20bf89109e715d7dc97aebc3811149c7c72" exitCode=0 Oct 31 00:32:12.983742 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:32:12.983685 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5014c66a-mg9r4" event={"ID":"d0c77474-1554-4f73-84ef-524855a5160a","Type":"ContainerDied","Data":"af6517da8f87f0cd9fdd668c6516e20bf89109e715d7dc97aebc3811149c7c72"} Oct 31 00:32:12.985233 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:32:12.985207 4927 generic.go:356] "Generic (PLEG): container finished" podID="2a65bb94-e7dd-4240-af0c-4ce7c24bc599" containerID="ad51b490904e9d709e16972f3397ee5aacb4b22fd5ecccd4ac695bf49d1c7695" exitCode=0 Oct 31 00:32:12.985315 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:32:12.985240 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-5c3374-7969595c9d-gzhtd" Oct 31 00:32:12.985315 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:32:12.985244 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-5c3374-7969595c9d-gzhtd" event={"ID":"2a65bb94-e7dd-4240-af0c-4ce7c24bc599","Type":"ContainerDied","Data":"ad51b490904e9d709e16972f3397ee5aacb4b22fd5ecccd4ac695bf49d1c7695"} Oct 31 00:32:12.985315 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:32:12.985283 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-5c3374-7969595c9d-gzhtd" event={"ID":"2a65bb94-e7dd-4240-af0c-4ce7c24bc599","Type":"ContainerDied","Data":"91a24e6a206b34c1e0747f2d580f16d228404b3dc22b0123043ce3a787d79fb7"} Oct 31 00:32:12.993521 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:32:12.993502 4927 scope.go:122] "RemoveContainer" containerID="c672e5ecdf6d33493ca13caa3595740645f4f791d76e997b3f60a01689a07573" Oct 31 00:32:12.993791 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:32:12.993768 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c672e5ecdf6d33493ca13caa3595740645f4f791d76e997b3f60a01689a07573\": container with ID starting with c672e5ecdf6d33493ca13caa3595740645f4f791d76e997b3f60a01689a07573 not found: ID does not exist" containerID="c672e5ecdf6d33493ca13caa3595740645f4f791d76e997b3f60a01689a07573" Oct 31 00:32:12.993890 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:32:12.993803 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c672e5ecdf6d33493ca13caa3595740645f4f791d76e997b3f60a01689a07573"} err="failed to get container status \"c672e5ecdf6d33493ca13caa3595740645f4f791d76e997b3f60a01689a07573\": rpc error: code = NotFound desc = could not find container \"c672e5ecdf6d33493ca13caa3595740645f4f791d76e997b3f60a01689a07573\": container with ID starting with c672e5ecdf6d33493ca13caa3595740645f4f791d76e997b3f60a01689a07573 not found: ID does not exist" Oct 31 00:32:12.993890 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:32:12.993828 4927 scope.go:122] "RemoveContainer" containerID="ad51b490904e9d709e16972f3397ee5aacb4b22fd5ecccd4ac695bf49d1c7695" Oct 31 00:32:13.003557 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:32:13.003536 4927 scope.go:122] "RemoveContainer" containerID="ad51b490904e9d709e16972f3397ee5aacb4b22fd5ecccd4ac695bf49d1c7695" Oct 31 00:32:13.003841 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:32:13.003820 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad51b490904e9d709e16972f3397ee5aacb4b22fd5ecccd4ac695bf49d1c7695\": container with ID starting with ad51b490904e9d709e16972f3397ee5aacb4b22fd5ecccd4ac695bf49d1c7695 not found: ID does not exist" containerID="ad51b490904e9d709e16972f3397ee5aacb4b22fd5ecccd4ac695bf49d1c7695" Oct 31 00:32:13.003921 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:32:13.003859 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad51b490904e9d709e16972f3397ee5aacb4b22fd5ecccd4ac695bf49d1c7695"} err="failed to get container status \"ad51b490904e9d709e16972f3397ee5aacb4b22fd5ecccd4ac695bf49d1c7695\": rpc error: code = NotFound desc = could not find container \"ad51b490904e9d709e16972f3397ee5aacb4b22fd5ecccd4ac695bf49d1c7695\": container with ID starting with ad51b490904e9d709e16972f3397ee5aacb4b22fd5ecccd4ac695bf49d1c7695 not found: ID does not exist" Oct 31 00:32:13.057005 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:32:13.056920 4927 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5c1c1eb-bce5-4ef6-8246-c0638f5c8b6b-plugins-config-volume\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:32:13.057005 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:32:13.056953 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xz9mx\" (UniqueName: \"kubernetes.io/projected/f5c1c1eb-bce5-4ef6-8246-c0638f5c8b6b-kube-api-access-xz9mx\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:32:13.121931 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:32:13.121896 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-5c3374-f56fc6f79-7b9pp"] Oct 31 00:32:13.125733 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:32:13.125710 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-5c3374-f56fc6f79-7b9pp"] Oct 31 00:32:13.240481 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:32:13.240447 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-5c3374-7969595c9d-gzhtd"] Oct 31 00:32:13.245862 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:32:13.245833 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-5c3374-7969595c9d-gzhtd"] Oct 31 00:32:14.151406 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:32:14.151374 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5014c66a-mg9r4" Oct 31 00:32:14.268300 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:32:14.268209 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d0c77474-1554-4f73-84ef-524855a5160a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0c77474-1554-4f73-84ef-524855a5160a-maas-api-openshift-ca\") pod \"d0c77474-1554-4f73-84ef-524855a5160a\" (UID: \"d0c77474-1554-4f73-84ef-524855a5160a\") " Oct 31 00:32:14.268498 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:32:14.268318 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0c77474-1554-4f73-84ef-524855a5160a-kube-api-access-szr2j\" (UniqueName: \"kubernetes.io/projected/d0c77474-1554-4f73-84ef-524855a5160a-kube-api-access-szr2j\") pod \"d0c77474-1554-4f73-84ef-524855a5160a\" (UID: \"d0c77474-1554-4f73-84ef-524855a5160a\") " Oct 31 00:32:14.268498 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:32:14.268405 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d0c77474-1554-4f73-84ef-524855a5160a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0c77474-1554-4f73-84ef-524855a5160a-maas-api-xks-ca\") pod \"d0c77474-1554-4f73-84ef-524855a5160a\" (UID: \"d0c77474-1554-4f73-84ef-524855a5160a\") " Oct 31 00:32:14.268659 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:32:14.268635 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0c77474-1554-4f73-84ef-524855a5160a-maas-api-openshift-ca" pod "d0c77474-1554-4f73-84ef-524855a5160a" (UID: "d0c77474-1554-4f73-84ef-524855a5160a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:32:14.268768 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:32:14.268749 4927 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0c77474-1554-4f73-84ef-524855a5160a-maas-api-openshift-ca\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:32:14.270879 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:32:14.270852 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0c77474-1554-4f73-84ef-524855a5160a-kube-api-access-szr2j" pod "d0c77474-1554-4f73-84ef-524855a5160a" (UID: "d0c77474-1554-4f73-84ef-524855a5160a"). InnerVolumeSpecName "kube-api-access-szr2j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:32:14.273178 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:32:14.273153 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0c77474-1554-4f73-84ef-524855a5160a-maas-api-xks-ca" pod "d0c77474-1554-4f73-84ef-524855a5160a" (UID: "d0c77474-1554-4f73-84ef-524855a5160a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:32:14.369449 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:32:14.369411 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-szr2j\" (UniqueName: \"kubernetes.io/projected/d0c77474-1554-4f73-84ef-524855a5160a-kube-api-access-szr2j\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:32:14.369449 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:32:14.369442 4927 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0c77474-1554-4f73-84ef-524855a5160a-maas-api-xks-ca\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:32:15.019353 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:32:15.018820 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5014c66a-mg9r4" event={"ID":"d0c77474-1554-4f73-84ef-524855a5160a","Type":"ContainerDied","Data":"477abc5c12c33c9aa55c4b6f47b4d4f422e83d79518682be611fb165e2083292"} Oct 31 00:32:15.019353 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:32:15.018885 4927 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="477abc5c12c33c9aa55c4b6f47b4d4f422e83d79518682be611fb165e2083292" Oct 31 00:32:15.019353 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:32:15.018957 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5014c66a-mg9r4" Oct 31 00:32:15.086171 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:32:15.086133 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a65bb94-e7dd-4240-af0c-4ce7c24bc599" path="/var/lib/kubelet/pods/2a65bb94-e7dd-4240-af0c-4ce7c24bc599/volumes" Oct 31 00:32:15.086525 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:32:15.086511 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f5c1c1eb-bce5-4ef6-8246-c0638f5c8b6b" path="/var/lib/kubelet/pods/f5c1c1eb-bce5-4ef6-8246-c0638f5c8b6b/volumes" Oct 31 00:32:15.573516 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:32:15.573475 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-5c3374/premium-w1-5c3374-kserve-6c56d4dcd-llwcl" podUID="dc813f02-e077-4251-8c1e-fea0986b0f36" containerName="main" containerID="cri-o://66afb0750f24146730758a0ff9eb3d95a460a6962908d6133b63e905a86b82eb" gracePeriod=45 Oct 31 00:32:15.831932 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:32:15.831876 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-5c3374/premium-w1-5c3374-kserve-6c56d4dcd-llwcl" Oct 31 00:32:15.989278 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:32:15.989237 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc813f02-e077-4251-8c1e-fea0986b0f36-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc813f02-e077-4251-8c1e-fea0986b0f36-tmp-dir\") pod \"dc813f02-e077-4251-8c1e-fea0986b0f36\" (UID: \"dc813f02-e077-4251-8c1e-fea0986b0f36\") " Oct 31 00:32:15.989473 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:32:15.989315 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc813f02-e077-4251-8c1e-fea0986b0f36-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc813f02-e077-4251-8c1e-fea0986b0f36-model-cache\") pod \"dc813f02-e077-4251-8c1e-fea0986b0f36\" (UID: \"dc813f02-e077-4251-8c1e-fea0986b0f36\") " Oct 31 00:32:15.989473 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:32:15.989365 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dc813f02-e077-4251-8c1e-fea0986b0f36-tls-certs\" (UniqueName: \"kubernetes.io/secret/dc813f02-e077-4251-8c1e-fea0986b0f36-tls-certs\") pod \"dc813f02-e077-4251-8c1e-fea0986b0f36\" (UID: \"dc813f02-e077-4251-8c1e-fea0986b0f36\") " Oct 31 00:32:15.989473 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:32:15.989396 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc813f02-e077-4251-8c1e-fea0986b0f36-kube-api-access-hkhrl\" (UniqueName: \"kubernetes.io/projected/dc813f02-e077-4251-8c1e-fea0986b0f36-kube-api-access-hkhrl\") pod \"dc813f02-e077-4251-8c1e-fea0986b0f36\" (UID: \"dc813f02-e077-4251-8c1e-fea0986b0f36\") " Oct 31 00:32:15.989599 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:32:15.989507 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc813f02-e077-4251-8c1e-fea0986b0f36-dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc813f02-e077-4251-8c1e-fea0986b0f36-dshm\") pod \"dc813f02-e077-4251-8c1e-fea0986b0f36\" (UID: \"dc813f02-e077-4251-8c1e-fea0986b0f36\") " Oct 31 00:32:15.989599 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:32:15.989589 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc813f02-e077-4251-8c1e-fea0986b0f36-home\" (UniqueName: \"kubernetes.io/empty-dir/dc813f02-e077-4251-8c1e-fea0986b0f36-home\") pod \"dc813f02-e077-4251-8c1e-fea0986b0f36\" (UID: \"dc813f02-e077-4251-8c1e-fea0986b0f36\") " Oct 31 00:32:15.989665 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:32:15.989592 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc813f02-e077-4251-8c1e-fea0986b0f36-tmp-dir" pod "dc813f02-e077-4251-8c1e-fea0986b0f36" (UID: "dc813f02-e077-4251-8c1e-fea0986b0f36"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:32:15.990079 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:32:15.989858 4927 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc813f02-e077-4251-8c1e-fea0986b0f36-tmp-dir\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:32:15.990226 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:32:15.990134 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc813f02-e077-4251-8c1e-fea0986b0f36-model-cache" pod "dc813f02-e077-4251-8c1e-fea0986b0f36" (UID: "dc813f02-e077-4251-8c1e-fea0986b0f36"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:32:15.990770 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:32:15.990734 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc813f02-e077-4251-8c1e-fea0986b0f36-home" pod "dc813f02-e077-4251-8c1e-fea0986b0f36" (UID: "dc813f02-e077-4251-8c1e-fea0986b0f36"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:32:15.992495 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:32:15.992469 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc813f02-e077-4251-8c1e-fea0986b0f36-dshm" pod "dc813f02-e077-4251-8c1e-fea0986b0f36" (UID: "dc813f02-e077-4251-8c1e-fea0986b0f36"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:32:15.992719 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:32:15.992697 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc813f02-e077-4251-8c1e-fea0986b0f36-tls-certs" pod "dc813f02-e077-4251-8c1e-fea0986b0f36" (UID: "dc813f02-e077-4251-8c1e-fea0986b0f36"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:32:15.992784 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:32:15.992737 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc813f02-e077-4251-8c1e-fea0986b0f36-kube-api-access-hkhrl" pod "dc813f02-e077-4251-8c1e-fea0986b0f36" (UID: "dc813f02-e077-4251-8c1e-fea0986b0f36"). InnerVolumeSpecName "kube-api-access-hkhrl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:32:16.026890 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:32:16.026857 4927 generic.go:356] "Generic (PLEG): container finished" podID="dc813f02-e077-4251-8c1e-fea0986b0f36" containerID="66afb0750f24146730758a0ff9eb3d95a460a6962908d6133b63e905a86b82eb" exitCode=0 Oct 31 00:32:16.027074 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:32:16.026918 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-5c3374/premium-w1-5c3374-kserve-6c56d4dcd-llwcl" Oct 31 00:32:16.027074 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:32:16.026947 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-5c3374/premium-w1-5c3374-kserve-6c56d4dcd-llwcl" event={"ID":"dc813f02-e077-4251-8c1e-fea0986b0f36","Type":"ContainerDied","Data":"66afb0750f24146730758a0ff9eb3d95a460a6962908d6133b63e905a86b82eb"} Oct 31 00:32:16.027074 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:32:16.026990 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-5c3374/premium-w1-5c3374-kserve-6c56d4dcd-llwcl" event={"ID":"dc813f02-e077-4251-8c1e-fea0986b0f36","Type":"ContainerDied","Data":"daf405cfe626671f3e2381c112a39a75147c9cd35d30e06e7dd450ceab0f09ce"} Oct 31 00:32:16.027074 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:32:16.027007 4927 scope.go:122] "RemoveContainer" containerID="66afb0750f24146730758a0ff9eb3d95a460a6962908d6133b63e905a86b82eb" Oct 31 00:32:16.039290 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:32:16.039268 4927 scope.go:122] "RemoveContainer" containerID="66afb0750f24146730758a0ff9eb3d95a460a6962908d6133b63e905a86b82eb" Oct 31 00:32:16.039679 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:32:16.039659 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66afb0750f24146730758a0ff9eb3d95a460a6962908d6133b63e905a86b82eb\": container with ID starting with 66afb0750f24146730758a0ff9eb3d95a460a6962908d6133b63e905a86b82eb not found: ID does not exist" containerID="66afb0750f24146730758a0ff9eb3d95a460a6962908d6133b63e905a86b82eb" Oct 31 00:32:16.039751 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:32:16.039688 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66afb0750f24146730758a0ff9eb3d95a460a6962908d6133b63e905a86b82eb"} err="failed to get container status \"66afb0750f24146730758a0ff9eb3d95a460a6962908d6133b63e905a86b82eb\": rpc error: code = NotFound desc = could not find container \"66afb0750f24146730758a0ff9eb3d95a460a6962908d6133b63e905a86b82eb\": container with ID starting with 66afb0750f24146730758a0ff9eb3d95a460a6962908d6133b63e905a86b82eb not found: ID does not exist" Oct 31 00:32:16.091478 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:32:16.091381 4927 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc813f02-e077-4251-8c1e-fea0986b0f36-dshm\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:32:16.091478 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:32:16.091419 4927 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dc813f02-e077-4251-8c1e-fea0986b0f36-home\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:32:16.091478 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:32:16.091434 4927 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc813f02-e077-4251-8c1e-fea0986b0f36-model-cache\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:32:16.091478 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:32:16.091448 4927 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc813f02-e077-4251-8c1e-fea0986b0f36-tls-certs\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:32:16.091478 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:32:16.091463 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hkhrl\" (UniqueName: \"kubernetes.io/projected/dc813f02-e077-4251-8c1e-fea0986b0f36-kube-api-access-hkhrl\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:32:16.200576 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:32:16.200543 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-5c3374/premium-w1-5c3374-kserve-6c56d4dcd-llwcl"] Oct 31 00:32:16.204843 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:32:16.204815 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-5c3374/premium-w1-5c3374-kserve-6c56d4dcd-llwcl"] Oct 31 00:32:17.085280 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:32:17.085243 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc813f02-e077-4251-8c1e-fea0986b0f36" path="/var/lib/kubelet/pods/dc813f02-e077-4251-8c1e-fea0986b0f36/volumes" Oct 31 00:32:19.677868 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:32:19.677824 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sqv69" containerName="extractor" Oct 31 00:32:26.463517 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:32:26.463484 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:32:27.921839 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:32:27.921802 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:32:28.182413 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:32:28.182318 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:32:28.388184 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:32:28.388151 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-5c3374-76h6v"] Oct 31 00:32:28.438886 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:32:28.438811 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-5c3374-76h6v"] Oct 31 00:32:28.851837 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:32:28.851760 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:32:29.084738 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:32:29.084697 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a7aec19d-427d-4215-bdda-6b5ee3ee6157" path="/var/lib/kubelet/pods/a7aec19d-427d-4215-bdda-6b5ee3ee6157/volumes" Oct 31 00:32:32.055840 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:32:32.055803 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-517d808e-6ltzk"] Oct 31 00:32:32.106743 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:32:32.106710 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-517d808e-6ltzk"] Oct 31 00:32:33.084554 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:32:33.084521 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5f5046d-b709-4aed-9cd8-0d480dc2960e" path="/var/lib/kubelet/pods/e5f5046d-b709-4aed-9cd8-0d480dc2960e/volumes" Oct 31 00:32:33.559721 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:32:33.559690 4927 scope.go:122] "RemoveContainer" containerID="2a9a1dc22d5842c3027722852dbf2f09a8c03e50fa6a17b20e14d13a4f43a983" Oct 31 00:32:35.560493 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:32:35.560458 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:32:36.033555 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:32:36.033519 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:32:36.195234 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:32:36.195199 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:32:36.374538 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:32:36.374441 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:32:37.886200 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:32:37.886165 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:32:38.098252 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:32:38.098207 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:32:38.612806 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:32:38.612767 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:32:44.640551 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:32:44.640503 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:32:49.678540 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:32:49.678501 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sqv69" containerName="extractor" Oct 31 00:32:53.050720 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:32:53.050684 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-83ae5d88-mmqdp"] Oct 31 00:32:53.101731 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:32:53.101695 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-83ae5d88-mmqdp"] Oct 31 00:32:55.085027 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:32:55.084980 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="085c6984-6c5b-4346-a05f-c29c82c4816c" path="/var/lib/kubelet/pods/085c6984-6c5b-4346-a05f-c29c82c4816c/volumes" Oct 31 00:33:06.364693 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:33:06.364656 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-gzhjj"] Oct 31 00:33:06.365205 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:33:06.365036 4927 cpu_manager.go:414] "containerMap: removing container" podUID="dc813f02-e077-4251-8c1e-fea0986b0f36" containerName="main" Oct 31 00:33:06.365205 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:33:06.365046 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc813f02-e077-4251-8c1e-fea0986b0f36" containerName="main" Oct 31 00:33:06.365205 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:33:06.365057 4927 cpu_manager.go:414] "containerMap: removing container" podUID="84baa184-a53a-4f9f-8597-fd7c95e04d0d" containerName="istio-proxy" Oct 31 00:33:06.365205 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:33:06.365063 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84baa184-a53a-4f9f-8597-fd7c95e04d0d" containerName="istio-proxy" Oct 31 00:33:06.365205 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:33:06.365093 4927 cpu_manager.go:414] "containerMap: removing container" podUID="f5c1c1eb-bce5-4ef6-8246-c0638f5c8b6b" containerName="payload-pre-processing" Oct 31 00:33:06.365205 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:33:06.365098 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5c1c1eb-bce5-4ef6-8246-c0638f5c8b6b" containerName="payload-pre-processing" Oct 31 00:33:06.365205 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:33:06.365109 4927 cpu_manager.go:414] "containerMap: removing container" podUID="d0c77474-1554-4f73-84ef-524855a5160a" containerName="revoke-keys" Oct 31 00:33:06.365205 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:33:06.365113 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0c77474-1554-4f73-84ef-524855a5160a" containerName="revoke-keys" Oct 31 00:33:06.365205 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:33:06.365122 4927 cpu_manager.go:414] "containerMap: removing container" podUID="2a65bb94-e7dd-4240-af0c-4ce7c24bc599" containerName="payload-processing" Oct 31 00:33:06.365205 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:33:06.365127 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a65bb94-e7dd-4240-af0c-4ce7c24bc599" containerName="payload-processing" Oct 31 00:33:06.365205 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:33:06.365188 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5c1c1eb-bce5-4ef6-8246-c0638f5c8b6b" containerName="payload-pre-processing" Oct 31 00:33:06.365205 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:33:06.365197 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0c77474-1554-4f73-84ef-524855a5160a" containerName="revoke-keys" Oct 31 00:33:06.365205 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:33:06.365204 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a65bb94-e7dd-4240-af0c-4ce7c24bc599" containerName="payload-processing" Oct 31 00:33:06.365205 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:33:06.365210 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc813f02-e077-4251-8c1e-fea0986b0f36" containerName="main" Oct 31 00:33:06.368454 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:33:06.368438 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-gzhjj" Oct 31 00:33:06.422771 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:33:06.422738 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-gzhjj"] Oct 31 00:33:06.454754 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:33:06.454719 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5j8v\" (UniqueName: \"kubernetes.io/projected/792ee66f-793e-40cb-90a3-f2d49af46273-kube-api-access-m5j8v\") pod \"maas-api-revoke-sub-85237e45-gzhjj\" (UID: \"792ee66f-793e-40cb-90a3-f2d49af46273\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-gzhjj" Oct 31 00:33:06.454922 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:33:06.454825 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/792ee66f-793e-40cb-90a3-f2d49af46273-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-gzhjj\" (UID: \"792ee66f-793e-40cb-90a3-f2d49af46273\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-gzhjj" Oct 31 00:33:06.454922 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:33:06.454858 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/792ee66f-793e-40cb-90a3-f2d49af46273-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-gzhjj\" (UID: \"792ee66f-793e-40cb-90a3-f2d49af46273\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-gzhjj" Oct 31 00:33:06.471008 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:33:06.470979 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:33:06.471134 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:33:06.471017 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4fdzs\"" Oct 31 00:33:06.555840 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:33:06.555804 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5j8v\" (UniqueName: \"kubernetes.io/projected/792ee66f-793e-40cb-90a3-f2d49af46273-kube-api-access-m5j8v\") pod \"maas-api-revoke-sub-85237e45-gzhjj\" (UID: \"792ee66f-793e-40cb-90a3-f2d49af46273\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-gzhjj" Oct 31 00:33:06.556056 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:33:06.555867 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/792ee66f-793e-40cb-90a3-f2d49af46273-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-gzhjj\" (UID: \"792ee66f-793e-40cb-90a3-f2d49af46273\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-gzhjj" Oct 31 00:33:06.556056 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:33:06.555894 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/792ee66f-793e-40cb-90a3-f2d49af46273-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-gzhjj\" (UID: \"792ee66f-793e-40cb-90a3-f2d49af46273\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-gzhjj" Oct 31 00:33:06.556600 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:33:06.556575 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/792ee66f-793e-40cb-90a3-f2d49af46273-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-gzhjj\" (UID: \"792ee66f-793e-40cb-90a3-f2d49af46273\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-gzhjj" Oct 31 00:33:06.558457 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:33:06.558439 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/792ee66f-793e-40cb-90a3-f2d49af46273-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-gzhjj\" (UID: \"792ee66f-793e-40cb-90a3-f2d49af46273\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-gzhjj" Oct 31 00:33:06.619507 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:33:06.619430 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5j8v\" (UniqueName: \"kubernetes.io/projected/792ee66f-793e-40cb-90a3-f2d49af46273-kube-api-access-m5j8v\") pod \"maas-api-revoke-sub-85237e45-gzhjj\" (UID: \"792ee66f-793e-40cb-90a3-f2d49af46273\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-gzhjj" Oct 31 00:33:06.681200 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:33:06.681166 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-gzhjj" Oct 31 00:33:07.058533 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:33:07.058403 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-gzhjj"] Oct 31 00:33:07.065042 ip-10-0-141-180 kubenswrapper[4927]: W1031 00:33:07.065015 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod792ee66f_793e_40cb_90a3_f2d49af46273.slice/crio-0457519b798aebdbaf8651ca29448142ce3437a270f8999db94e99d7a8c018f7 WatchSource:0}: Error finding container 0457519b798aebdbaf8651ca29448142ce3437a270f8999db94e99d7a8c018f7: Status 404 returned error can't find the container with id 0457519b798aebdbaf8651ca29448142ce3437a270f8999db94e99d7a8c018f7 Oct 31 00:33:07.354707 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:33:07.354675 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-gzhjj" event={"ID":"792ee66f-793e-40cb-90a3-f2d49af46273","Type":"ContainerStarted","Data":"e32104edc7fc9f328b1f03fc1b0660480e5d713fa2b27965fc983ed2f633be7f"} Oct 31 00:33:07.354856 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:33:07.354731 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-gzhjj" event={"ID":"792ee66f-793e-40cb-90a3-f2d49af46273","Type":"ContainerStarted","Data":"0457519b798aebdbaf8651ca29448142ce3437a270f8999db94e99d7a8c018f7"} Oct 31 00:33:07.473891 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:33:07.473834 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-gzhjj" podStartSLOduration=1.473820984 podStartE2EDuration="1.473820984s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:33:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:33:07.472513131 +0000 UTC m=+3100.508843970" watchObservedRunningTime="2026-10-31 00:33:07.473820984 +0000 UTC m=+3100.510151807" Oct 31 00:33:08.362780 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:33:08.362739 4927 generic.go:356] "Generic (PLEG): container finished" podID="792ee66f-793e-40cb-90a3-f2d49af46273" containerID="e32104edc7fc9f328b1f03fc1b0660480e5d713fa2b27965fc983ed2f633be7f" exitCode=0 Oct 31 00:33:08.362979 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:33:08.362825 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-gzhjj" event={"ID":"792ee66f-793e-40cb-90a3-f2d49af46273","Type":"ContainerDied","Data":"e32104edc7fc9f328b1f03fc1b0660480e5d713fa2b27965fc983ed2f633be7f"} Oct 31 00:33:09.500990 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:33:09.500962 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-gzhjj" Oct 31 00:33:09.588942 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:33:09.588903 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/792ee66f-793e-40cb-90a3-f2d49af46273-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/792ee66f-793e-40cb-90a3-f2d49af46273-maas-api-openshift-ca\") pod \"792ee66f-793e-40cb-90a3-f2d49af46273\" (UID: \"792ee66f-793e-40cb-90a3-f2d49af46273\") " Oct 31 00:33:09.589138 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:33:09.588996 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/792ee66f-793e-40cb-90a3-f2d49af46273-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/792ee66f-793e-40cb-90a3-f2d49af46273-maas-api-xks-ca\") pod \"792ee66f-793e-40cb-90a3-f2d49af46273\" (UID: \"792ee66f-793e-40cb-90a3-f2d49af46273\") " Oct 31 00:33:09.589138 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:33:09.589065 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/792ee66f-793e-40cb-90a3-f2d49af46273-kube-api-access-m5j8v\" (UniqueName: \"kubernetes.io/projected/792ee66f-793e-40cb-90a3-f2d49af46273-kube-api-access-m5j8v\") pod \"792ee66f-793e-40cb-90a3-f2d49af46273\" (UID: \"792ee66f-793e-40cb-90a3-f2d49af46273\") " Oct 31 00:33:09.589380 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:33:09.589312 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/792ee66f-793e-40cb-90a3-f2d49af46273-maas-api-openshift-ca" pod "792ee66f-793e-40cb-90a3-f2d49af46273" (UID: "792ee66f-793e-40cb-90a3-f2d49af46273"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:33:09.591025 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:33:09.591002 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/792ee66f-793e-40cb-90a3-f2d49af46273-maas-api-xks-ca" pod "792ee66f-793e-40cb-90a3-f2d49af46273" (UID: "792ee66f-793e-40cb-90a3-f2d49af46273"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:33:09.591224 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:33:09.591202 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/792ee66f-793e-40cb-90a3-f2d49af46273-kube-api-access-m5j8v" pod "792ee66f-793e-40cb-90a3-f2d49af46273" (UID: "792ee66f-793e-40cb-90a3-f2d49af46273"). InnerVolumeSpecName "kube-api-access-m5j8v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:33:09.690661 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:33:09.690630 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m5j8v\" (UniqueName: \"kubernetes.io/projected/792ee66f-793e-40cb-90a3-f2d49af46273-kube-api-access-m5j8v\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:33:09.690661 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:33:09.690661 4927 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/792ee66f-793e-40cb-90a3-f2d49af46273-maas-api-openshift-ca\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:33:09.690880 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:33:09.690672 4927 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/792ee66f-793e-40cb-90a3-f2d49af46273-maas-api-xks-ca\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:33:10.377368 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:33:10.377325 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-gzhjj" Oct 31 00:33:10.377547 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:33:10.377342 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-gzhjj" event={"ID":"792ee66f-793e-40cb-90a3-f2d49af46273","Type":"ContainerDied","Data":"0457519b798aebdbaf8651ca29448142ce3437a270f8999db94e99d7a8c018f7"} Oct 31 00:33:10.377547 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:33:10.377454 4927 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0457519b798aebdbaf8651ca29448142ce3437a270f8999db94e99d7a8c018f7" Oct 31 00:33:10.657908 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:33:10.657870 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:33:15.275282 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:33:15.275244 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-4sgzb"] Oct 31 00:33:15.275823 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:33:15.275709 4927 cpu_manager.go:414] "containerMap: removing container" podUID="792ee66f-793e-40cb-90a3-f2d49af46273" containerName="revoke-keys" Oct 31 00:33:15.275823 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:33:15.275728 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="792ee66f-793e-40cb-90a3-f2d49af46273" containerName="revoke-keys" Oct 31 00:33:15.275823 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:33:15.275812 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="792ee66f-793e-40cb-90a3-f2d49af46273" containerName="revoke-keys" Oct 31 00:33:15.279192 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:33:15.279174 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-4sgzb" Oct 31 00:33:15.388062 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:33:15.388028 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-4sgzb"] Oct 31 00:33:15.445969 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:33:15.445936 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/531eadda-5f07-4116-8b88-6331bebe01c2-workload-certs\") pod \"maas-default-gateway-openshift-default-8f6f87f59-4sgzb\" (UID: \"531eadda-5f07-4116-8b88-6331bebe01c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-4sgzb" Oct 31 00:33:15.446146 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:33:15.445993 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/531eadda-5f07-4116-8b88-6331bebe01c2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8f6f87f59-4sgzb\" (UID: \"531eadda-5f07-4116-8b88-6331bebe01c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-4sgzb" Oct 31 00:33:15.446202 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:33:15.446155 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/531eadda-5f07-4116-8b88-6331bebe01c2-istio-envoy\") pod \"maas-default-gateway-openshift-default-8f6f87f59-4sgzb\" (UID: \"531eadda-5f07-4116-8b88-6331bebe01c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-4sgzb" Oct 31 00:33:15.446202 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:33:15.446190 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/531eadda-5f07-4116-8b88-6331bebe01c2-istio-data\") pod \"maas-default-gateway-openshift-default-8f6f87f59-4sgzb\" (UID: \"531eadda-5f07-4116-8b88-6331bebe01c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-4sgzb" Oct 31 00:33:15.446273 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:33:15.446251 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/531eadda-5f07-4116-8b88-6331bebe01c2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8f6f87f59-4sgzb\" (UID: \"531eadda-5f07-4116-8b88-6331bebe01c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-4sgzb" Oct 31 00:33:15.446417 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:33:15.446395 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/531eadda-5f07-4116-8b88-6331bebe01c2-istio-token\") pod \"maas-default-gateway-openshift-default-8f6f87f59-4sgzb\" (UID: \"531eadda-5f07-4116-8b88-6331bebe01c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-4sgzb" Oct 31 00:33:15.446513 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:33:15.446438 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8bwx6\" (UniqueName: \"kubernetes.io/projected/531eadda-5f07-4116-8b88-6331bebe01c2-kube-api-access-8bwx6\") pod \"maas-default-gateway-openshift-default-8f6f87f59-4sgzb\" (UID: \"531eadda-5f07-4116-8b88-6331bebe01c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-4sgzb" Oct 31 00:33:15.446513 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:33:15.446495 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/531eadda-5f07-4116-8b88-6331bebe01c2-workload-socket\") pod \"maas-default-gateway-openshift-default-8f6f87f59-4sgzb\" (UID: \"531eadda-5f07-4116-8b88-6331bebe01c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-4sgzb" Oct 31 00:33:15.446586 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:33:15.446530 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/531eadda-5f07-4116-8b88-6331bebe01c2-credential-socket\") pod \"maas-default-gateway-openshift-default-8f6f87f59-4sgzb\" (UID: \"531eadda-5f07-4116-8b88-6331bebe01c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-4sgzb" Oct 31 00:33:15.547469 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:33:15.547384 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/531eadda-5f07-4116-8b88-6331bebe01c2-istio-token\") pod \"maas-default-gateway-openshift-default-8f6f87f59-4sgzb\" (UID: \"531eadda-5f07-4116-8b88-6331bebe01c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-4sgzb" Oct 31 00:33:15.547469 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:33:15.547425 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8bwx6\" (UniqueName: \"kubernetes.io/projected/531eadda-5f07-4116-8b88-6331bebe01c2-kube-api-access-8bwx6\") pod \"maas-default-gateway-openshift-default-8f6f87f59-4sgzb\" (UID: \"531eadda-5f07-4116-8b88-6331bebe01c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-4sgzb" Oct 31 00:33:15.547469 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:33:15.547446 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/531eadda-5f07-4116-8b88-6331bebe01c2-workload-socket\") pod \"maas-default-gateway-openshift-default-8f6f87f59-4sgzb\" (UID: \"531eadda-5f07-4116-8b88-6331bebe01c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-4sgzb" Oct 31 00:33:15.547469 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:33:15.547470 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/531eadda-5f07-4116-8b88-6331bebe01c2-credential-socket\") pod \"maas-default-gateway-openshift-default-8f6f87f59-4sgzb\" (UID: \"531eadda-5f07-4116-8b88-6331bebe01c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-4sgzb" Oct 31 00:33:15.547830 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:33:15.547597 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/531eadda-5f07-4116-8b88-6331bebe01c2-workload-certs\") pod \"maas-default-gateway-openshift-default-8f6f87f59-4sgzb\" (UID: \"531eadda-5f07-4116-8b88-6331bebe01c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-4sgzb" Oct 31 00:33:15.547830 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:33:15.547656 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/531eadda-5f07-4116-8b88-6331bebe01c2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8f6f87f59-4sgzb\" (UID: \"531eadda-5f07-4116-8b88-6331bebe01c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-4sgzb" Oct 31 00:33:15.547830 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:33:15.547730 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/531eadda-5f07-4116-8b88-6331bebe01c2-istio-envoy\") pod \"maas-default-gateway-openshift-default-8f6f87f59-4sgzb\" (UID: \"531eadda-5f07-4116-8b88-6331bebe01c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-4sgzb" Oct 31 00:33:15.547830 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:33:15.547758 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/531eadda-5f07-4116-8b88-6331bebe01c2-istio-data\") pod \"maas-default-gateway-openshift-default-8f6f87f59-4sgzb\" (UID: \"531eadda-5f07-4116-8b88-6331bebe01c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-4sgzb" Oct 31 00:33:15.547830 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:33:15.547795 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/531eadda-5f07-4116-8b88-6331bebe01c2-credential-socket\") pod \"maas-default-gateway-openshift-default-8f6f87f59-4sgzb\" (UID: \"531eadda-5f07-4116-8b88-6331bebe01c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-4sgzb" Oct 31 00:33:15.548078 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:33:15.547799 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/531eadda-5f07-4116-8b88-6331bebe01c2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8f6f87f59-4sgzb\" (UID: \"531eadda-5f07-4116-8b88-6331bebe01c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-4sgzb" Oct 31 00:33:15.548078 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:33:15.547905 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/531eadda-5f07-4116-8b88-6331bebe01c2-workload-socket\") pod \"maas-default-gateway-openshift-default-8f6f87f59-4sgzb\" (UID: \"531eadda-5f07-4116-8b88-6331bebe01c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-4sgzb" Oct 31 00:33:15.548078 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:33:15.548005 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/531eadda-5f07-4116-8b88-6331bebe01c2-workload-certs\") pod \"maas-default-gateway-openshift-default-8f6f87f59-4sgzb\" (UID: \"531eadda-5f07-4116-8b88-6331bebe01c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-4sgzb" Oct 31 00:33:15.548229 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:33:15.548083 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/531eadda-5f07-4116-8b88-6331bebe01c2-istio-data\") pod \"maas-default-gateway-openshift-default-8f6f87f59-4sgzb\" (UID: \"531eadda-5f07-4116-8b88-6331bebe01c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-4sgzb" Oct 31 00:33:15.548425 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:33:15.548407 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/531eadda-5f07-4116-8b88-6331bebe01c2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8f6f87f59-4sgzb\" (UID: \"531eadda-5f07-4116-8b88-6331bebe01c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-4sgzb" Oct 31 00:33:15.550092 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:33:15.550068 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/531eadda-5f07-4116-8b88-6331bebe01c2-istio-envoy\") pod \"maas-default-gateway-openshift-default-8f6f87f59-4sgzb\" (UID: \"531eadda-5f07-4116-8b88-6331bebe01c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-4sgzb" Oct 31 00:33:15.550186 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:33:15.550106 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/531eadda-5f07-4116-8b88-6331bebe01c2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8f6f87f59-4sgzb\" (UID: \"531eadda-5f07-4116-8b88-6331bebe01c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-4sgzb" Oct 31 00:33:15.609811 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:33:15.609779 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/531eadda-5f07-4116-8b88-6331bebe01c2-istio-token\") pod \"maas-default-gateway-openshift-default-8f6f87f59-4sgzb\" (UID: \"531eadda-5f07-4116-8b88-6331bebe01c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-4sgzb" Oct 31 00:33:15.609946 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:33:15.609864 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8bwx6\" (UniqueName: \"kubernetes.io/projected/531eadda-5f07-4116-8b88-6331bebe01c2-kube-api-access-8bwx6\") pod \"maas-default-gateway-openshift-default-8f6f87f59-4sgzb\" (UID: \"531eadda-5f07-4116-8b88-6331bebe01c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-4sgzb" Oct 31 00:33:15.891755 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:33:15.891667 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-4sgzb" Oct 31 00:33:16.325500 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:33:16.325474 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-4sgzb"] Oct 31 00:33:16.327003 ip-10-0-141-180 kubenswrapper[4927]: W1031 00:33:16.326978 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod531eadda_5f07_4116_8b88_6331bebe01c2.slice/crio-33fa4a4a443ab7d5bcc24508786a36507a25618019a3302c9ad64d902bee4d93 WatchSource:0}: Error finding container 33fa4a4a443ab7d5bcc24508786a36507a25618019a3302c9ad64d902bee4d93: Status 404 returned error can't find the container with id 33fa4a4a443ab7d5bcc24508786a36507a25618019a3302c9ad64d902bee4d93 Oct 31 00:33:16.329583 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:33:16.329544 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:33:16.329675 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:33:16.329618 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:33:16.329675 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:33:16.329655 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:33:16.417784 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:33:16.417743 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-4sgzb" event={"ID":"531eadda-5f07-4116-8b88-6331bebe01c2","Type":"ContainerStarted","Data":"2f93abb10083b212d2d4b11c2be481bbe313c9ca6a5f7a55b2a88685266b69bb"} Oct 31 00:33:16.417973 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:33:16.417848 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-4sgzb" event={"ID":"531eadda-5f07-4116-8b88-6331bebe01c2","Type":"ContainerStarted","Data":"33fa4a4a443ab7d5bcc24508786a36507a25618019a3302c9ad64d902bee4d93"} Oct 31 00:33:16.547203 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:33:16.547136 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-4sgzb" podStartSLOduration=1.547117487 podStartE2EDuration="1.547117487s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:33:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:33:16.54244042 +0000 UTC m=+3109.578771265" watchObservedRunningTime="2026-10-31 00:33:16.547117487 +0000 UTC m=+3109.583448310" Oct 31 00:33:16.892089 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:33:16.892055 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-4sgzb" Oct 31 00:33:17.893877 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:33:17.893836 4927 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-4sgzb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.16:15021/healthz/ready\": dial tcp 10.132.1.16:15021: connect: connection refused" start-of-body= Oct 31 00:33:17.894272 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:33:17.893904 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-4sgzb" podUID="531eadda-5f07-4116-8b88-6331bebe01c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.16:15021/healthz/ready\": dial tcp 10.132.1.16:15021: connect: connection refused" Oct 31 00:33:18.892650 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:33:18.892609 4927 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-4sgzb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.16:15021/healthz/ready\": dial tcp 10.132.1.16:15021: connect: connection refused" start-of-body= Oct 31 00:33:18.892839 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:33:18.892664 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-4sgzb" podUID="531eadda-5f07-4116-8b88-6331bebe01c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.16:15021/healthz/ready\": dial tcp 10.132.1.16:15021: connect: connection refused" Oct 31 00:33:18.954816 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:33:18.954781 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:33:19.677789 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:33:19.677750 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sqv69" containerName="extractor" Oct 31 00:33:19.892901 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:33:19.892818 4927 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-4sgzb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.16:15021/healthz/ready\": dial tcp 10.132.1.16:15021: connect: connection refused" start-of-body= Oct 31 00:33:19.892901 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:33:19.892872 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-4sgzb" podUID="531eadda-5f07-4116-8b88-6331bebe01c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.16:15021/healthz/ready\": dial tcp 10.132.1.16:15021: connect: connection refused" Oct 31 00:33:20.893016 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:33:20.892982 4927 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-4sgzb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.16:15021/healthz/ready\": dial tcp 10.132.1.16:15021: connect: connection refused" start-of-body= Oct 31 00:33:20.893524 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:33:20.893493 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-4sgzb" podUID="531eadda-5f07-4116-8b88-6331bebe01c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.16:15021/healthz/ready\": dial tcp 10.132.1.16:15021: connect: connection refused" Oct 31 00:33:21.460188 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:33:21.460152 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:33:21.892965 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:33:21.892880 4927 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-4sgzb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.16:15021/healthz/ready\": dial tcp 10.132.1.16:15021: connect: connection refused" start-of-body= Oct 31 00:33:21.892965 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:33:21.892941 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-4sgzb" podUID="531eadda-5f07-4116-8b88-6331bebe01c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.16:15021/healthz/ready\": dial tcp 10.132.1.16:15021: connect: connection refused" Oct 31 00:33:22.051590 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:33:22.051549 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-17cd123d-jj5sg"] Oct 31 00:33:22.060002 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:33:22.059971 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-17cd123d-jj5sg"] Oct 31 00:33:22.892875 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:33:22.892826 4927 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-4sgzb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.16:15021/healthz/ready\": dial tcp 10.132.1.16:15021: connect: connection refused" start-of-body= Oct 31 00:33:22.893058 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:33:22.892903 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-4sgzb" podUID="531eadda-5f07-4116-8b88-6331bebe01c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.16:15021/healthz/ready\": dial tcp 10.132.1.16:15021: connect: connection refused" Oct 31 00:33:23.085787 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:33:23.085745 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd537237-85e9-4a54-8ace-8ca915d2057d" path="/var/lib/kubelet/pods/dd537237-85e9-4a54-8ace-8ca915d2057d/volumes" Oct 31 00:33:24.473051 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:33:24.473016 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-4sgzb" Oct 31 00:33:24.474030 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:33:24.474001 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-4sgzb" Oct 31 00:33:24.474926 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:33:24.474902 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-4sgzb" Oct 31 00:33:33.608644 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:33:33.608611 4927 scope.go:122] "RemoveContainer" containerID="5d032e6fb419be7d3db411a038e3db084f8bdfc111084bc9b03c67501118cb17" Oct 31 00:33:33.617981 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:33:33.617959 4927 scope.go:122] "RemoveContainer" containerID="902c792a2f697e696ce631afd9a8da776f53ebff8a73e2fb62b7ae1bfe90148a" Oct 31 00:33:35.923925 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:33:35.923890 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:33:36.284646 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:33:36.284554 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:33:38.077470 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:33:38.077431 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:33:49.677289 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:33:49.677251 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sqv69" containerName="extractor" Oct 31 00:34:09.987495 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:34:09.987453 4927 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-lwhxc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:34:09.989928 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:34:09.987524 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-lwhxc" podUID="e8c6381b-c2ea-4a6c-8929-a7dd5fb29514" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:34:16.017402 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:34:16.017366 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-dgbqb"] Oct 31 00:34:16.021200 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:34:16.021183 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-dgbqb" Oct 31 00:34:16.075832 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:34:16.075797 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-dgbqb"] Oct 31 00:34:16.119298 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:34:16.119267 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1fe65e4-7f84-42e7-9721-401adb8493b2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-dgbqb\" (UID: \"a1fe65e4-7f84-42e7-9721-401adb8493b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-dgbqb" Oct 31 00:34:16.119481 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:34:16.119313 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pz7dr\" (UniqueName: \"kubernetes.io/projected/a1fe65e4-7f84-42e7-9721-401adb8493b2-kube-api-access-pz7dr\") pod \"maas-api-revoke-sub-c1e271a9-dgbqb\" (UID: \"a1fe65e4-7f84-42e7-9721-401adb8493b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-dgbqb" Oct 31 00:34:16.119481 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:34:16.119441 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1fe65e4-7f84-42e7-9721-401adb8493b2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-dgbqb\" (UID: \"a1fe65e4-7f84-42e7-9721-401adb8493b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-dgbqb" Oct 31 00:34:16.123788 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:34:16.123765 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:34:16.123929 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:34:16.123816 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4fdzs\"" Oct 31 00:34:16.220659 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:34:16.220623 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1fe65e4-7f84-42e7-9721-401adb8493b2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-dgbqb\" (UID: \"a1fe65e4-7f84-42e7-9721-401adb8493b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-dgbqb" Oct 31 00:34:16.220856 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:34:16.220704 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pz7dr\" (UniqueName: \"kubernetes.io/projected/a1fe65e4-7f84-42e7-9721-401adb8493b2-kube-api-access-pz7dr\") pod \"maas-api-revoke-sub-c1e271a9-dgbqb\" (UID: \"a1fe65e4-7f84-42e7-9721-401adb8493b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-dgbqb" Oct 31 00:34:16.220856 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:34:16.220791 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1fe65e4-7f84-42e7-9721-401adb8493b2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-dgbqb\" (UID: \"a1fe65e4-7f84-42e7-9721-401adb8493b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-dgbqb" Oct 31 00:34:16.221320 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:34:16.221300 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1fe65e4-7f84-42e7-9721-401adb8493b2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-dgbqb\" (UID: \"a1fe65e4-7f84-42e7-9721-401adb8493b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-dgbqb" Oct 31 00:34:16.223204 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:34:16.223177 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1fe65e4-7f84-42e7-9721-401adb8493b2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-dgbqb\" (UID: \"a1fe65e4-7f84-42e7-9721-401adb8493b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-dgbqb" Oct 31 00:34:16.283433 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:34:16.283327 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pz7dr\" (UniqueName: \"kubernetes.io/projected/a1fe65e4-7f84-42e7-9721-401adb8493b2-kube-api-access-pz7dr\") pod \"maas-api-revoke-sub-c1e271a9-dgbqb\" (UID: \"a1fe65e4-7f84-42e7-9721-401adb8493b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-dgbqb" Oct 31 00:34:16.333592 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:34:16.333560 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-dgbqb" Oct 31 00:34:16.710589 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:34:16.710550 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-dgbqb"] Oct 31 00:34:16.712536 ip-10-0-141-180 kubenswrapper[4927]: W1031 00:34:16.712506 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1fe65e4_7f84_42e7_9721_401adb8493b2.slice/crio-00109a32c279c1e1e7a63e54b5ae3334c5021abd38e0d4cc3394a2596453be97 WatchSource:0}: Error finding container 00109a32c279c1e1e7a63e54b5ae3334c5021abd38e0d4cc3394a2596453be97: Status 404 returned error can't find the container with id 00109a32c279c1e1e7a63e54b5ae3334c5021abd38e0d4cc3394a2596453be97 Oct 31 00:34:16.811348 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:34:16.811303 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-dgbqb" event={"ID":"a1fe65e4-7f84-42e7-9721-401adb8493b2","Type":"ContainerStarted","Data":"00109a32c279c1e1e7a63e54b5ae3334c5021abd38e0d4cc3394a2596453be97"} Oct 31 00:34:17.819174 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:34:17.819137 4927 generic.go:356] "Generic (PLEG): container finished" podID="a1fe65e4-7f84-42e7-9721-401adb8493b2" containerID="1392615bec61d83fcd3e76956e0c847e641612ddfa5a7ebb4e2217f2825e284d" exitCode=0 Oct 31 00:34:17.819602 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:34:17.819224 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-dgbqb" event={"ID":"a1fe65e4-7f84-42e7-9721-401adb8493b2","Type":"ContainerDied","Data":"1392615bec61d83fcd3e76956e0c847e641612ddfa5a7ebb4e2217f2825e284d"} Oct 31 00:34:18.965666 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:34:18.965636 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-dgbqb" Oct 31 00:34:19.047754 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:34:19.047718 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a1fe65e4-7f84-42e7-9721-401adb8493b2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1fe65e4-7f84-42e7-9721-401adb8493b2-maas-api-openshift-ca\") pod \"a1fe65e4-7f84-42e7-9721-401adb8493b2\" (UID: \"a1fe65e4-7f84-42e7-9721-401adb8493b2\") " Oct 31 00:34:19.047937 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:34:19.047768 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1fe65e4-7f84-42e7-9721-401adb8493b2-kube-api-access-pz7dr\" (UniqueName: \"kubernetes.io/projected/a1fe65e4-7f84-42e7-9721-401adb8493b2-kube-api-access-pz7dr\") pod \"a1fe65e4-7f84-42e7-9721-401adb8493b2\" (UID: \"a1fe65e4-7f84-42e7-9721-401adb8493b2\") " Oct 31 00:34:19.047937 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:34:19.047894 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a1fe65e4-7f84-42e7-9721-401adb8493b2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1fe65e4-7f84-42e7-9721-401adb8493b2-maas-api-xks-ca\") pod \"a1fe65e4-7f84-42e7-9721-401adb8493b2\" (UID: \"a1fe65e4-7f84-42e7-9721-401adb8493b2\") " Oct 31 00:34:19.048185 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:34:19.048160 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1fe65e4-7f84-42e7-9721-401adb8493b2-maas-api-openshift-ca" pod "a1fe65e4-7f84-42e7-9721-401adb8493b2" (UID: "a1fe65e4-7f84-42e7-9721-401adb8493b2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:34:19.048275 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:34:19.048226 4927 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1fe65e4-7f84-42e7-9721-401adb8493b2-maas-api-openshift-ca\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:34:19.050007 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:34:19.049977 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a1fe65e4-7f84-42e7-9721-401adb8493b2-maas-api-xks-ca" pod "a1fe65e4-7f84-42e7-9721-401adb8493b2" (UID: "a1fe65e4-7f84-42e7-9721-401adb8493b2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:34:19.050007 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:34:19.049994 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1fe65e4-7f84-42e7-9721-401adb8493b2-kube-api-access-pz7dr" pod "a1fe65e4-7f84-42e7-9721-401adb8493b2" (UID: "a1fe65e4-7f84-42e7-9721-401adb8493b2"). InnerVolumeSpecName "kube-api-access-pz7dr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:34:19.149217 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:34:19.149184 4927 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1fe65e4-7f84-42e7-9721-401adb8493b2-maas-api-xks-ca\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:34:19.149217 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:34:19.149214 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pz7dr\" (UniqueName: \"kubernetes.io/projected/a1fe65e4-7f84-42e7-9721-401adb8493b2-kube-api-access-pz7dr\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:34:19.677623 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:34:19.677575 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sqv69" containerName="extractor" Oct 31 00:34:19.833227 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:34:19.833141 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-dgbqb" Oct 31 00:34:19.833227 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:34:19.833172 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-dgbqb" event={"ID":"a1fe65e4-7f84-42e7-9721-401adb8493b2","Type":"ContainerDied","Data":"00109a32c279c1e1e7a63e54b5ae3334c5021abd38e0d4cc3394a2596453be97"} Oct 31 00:34:19.833227 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:34:19.833221 4927 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="00109a32c279c1e1e7a63e54b5ae3334c5021abd38e0d4cc3394a2596453be97" Oct 31 00:34:23.077215 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:34:23.077174 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:34:30.298835 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:34:30.298788 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-56c5v"] Oct 31 00:34:30.299266 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:34:30.299212 4927 cpu_manager.go:414] "containerMap: removing container" podUID="a1fe65e4-7f84-42e7-9721-401adb8493b2" containerName="revoke-keys" Oct 31 00:34:30.299266 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:34:30.299224 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1fe65e4-7f84-42e7-9721-401adb8493b2" containerName="revoke-keys" Oct 31 00:34:30.299383 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:34:30.299351 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1fe65e4-7f84-42e7-9721-401adb8493b2" containerName="revoke-keys" Oct 31 00:34:30.302128 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:34:30.302108 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-56c5v" Oct 31 00:34:30.429004 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:34:30.428966 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-56c5v"] Oct 31 00:34:30.457006 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:34:30.456958 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2759aa4-d678-4552-94ba-cdab90a866f7-workload-certs\") pod \"maas-default-gateway-openshift-default-8f6f87f59-56c5v\" (UID: \"e2759aa4-d678-4552-94ba-cdab90a866f7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-56c5v" Oct 31 00:34:30.457199 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:34:30.457010 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2759aa4-d678-4552-94ba-cdab90a866f7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8f6f87f59-56c5v\" (UID: \"e2759aa4-d678-4552-94ba-cdab90a866f7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-56c5v" Oct 31 00:34:30.457251 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:34:30.457211 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2759aa4-d678-4552-94ba-cdab90a866f7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8f6f87f59-56c5v\" (UID: \"e2759aa4-d678-4552-94ba-cdab90a866f7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-56c5v" Oct 31 00:34:30.457311 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:34:30.457254 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2759aa4-d678-4552-94ba-cdab90a866f7-workload-socket\") pod \"maas-default-gateway-openshift-default-8f6f87f59-56c5v\" (UID: \"e2759aa4-d678-4552-94ba-cdab90a866f7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-56c5v" Oct 31 00:34:30.457418 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:34:30.457385 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2759aa4-d678-4552-94ba-cdab90a866f7-istio-envoy\") pod \"maas-default-gateway-openshift-default-8f6f87f59-56c5v\" (UID: \"e2759aa4-d678-4552-94ba-cdab90a866f7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-56c5v" Oct 31 00:34:30.457487 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:34:30.457455 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2759aa4-d678-4552-94ba-cdab90a866f7-istio-token\") pod \"maas-default-gateway-openshift-default-8f6f87f59-56c5v\" (UID: \"e2759aa4-d678-4552-94ba-cdab90a866f7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-56c5v" Oct 31 00:34:30.457559 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:34:30.457537 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2759aa4-d678-4552-94ba-cdab90a866f7-credential-socket\") pod \"maas-default-gateway-openshift-default-8f6f87f59-56c5v\" (UID: \"e2759aa4-d678-4552-94ba-cdab90a866f7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-56c5v" Oct 31 00:34:30.457598 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:34:30.457579 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2759aa4-d678-4552-94ba-cdab90a866f7-istio-data\") pod \"maas-default-gateway-openshift-default-8f6f87f59-56c5v\" (UID: \"e2759aa4-d678-4552-94ba-cdab90a866f7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-56c5v" Oct 31 00:34:30.457649 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:34:30.457626 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxjjd\" (UniqueName: \"kubernetes.io/projected/e2759aa4-d678-4552-94ba-cdab90a866f7-kube-api-access-qxjjd\") pod \"maas-default-gateway-openshift-default-8f6f87f59-56c5v\" (UID: \"e2759aa4-d678-4552-94ba-cdab90a866f7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-56c5v" Oct 31 00:34:30.558471 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:34:30.558375 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2759aa4-d678-4552-94ba-cdab90a866f7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8f6f87f59-56c5v\" (UID: \"e2759aa4-d678-4552-94ba-cdab90a866f7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-56c5v" Oct 31 00:34:30.558471 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:34:30.558431 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2759aa4-d678-4552-94ba-cdab90a866f7-workload-socket\") pod \"maas-default-gateway-openshift-default-8f6f87f59-56c5v\" (UID: \"e2759aa4-d678-4552-94ba-cdab90a866f7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-56c5v" Oct 31 00:34:30.558471 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:34:30.558459 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2759aa4-d678-4552-94ba-cdab90a866f7-istio-envoy\") pod \"maas-default-gateway-openshift-default-8f6f87f59-56c5v\" (UID: \"e2759aa4-d678-4552-94ba-cdab90a866f7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-56c5v" Oct 31 00:34:30.558786 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:34:30.558507 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2759aa4-d678-4552-94ba-cdab90a866f7-istio-token\") pod \"maas-default-gateway-openshift-default-8f6f87f59-56c5v\" (UID: \"e2759aa4-d678-4552-94ba-cdab90a866f7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-56c5v" Oct 31 00:34:30.558786 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:34:30.558562 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2759aa4-d678-4552-94ba-cdab90a866f7-credential-socket\") pod \"maas-default-gateway-openshift-default-8f6f87f59-56c5v\" (UID: \"e2759aa4-d678-4552-94ba-cdab90a866f7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-56c5v" Oct 31 00:34:30.558786 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:34:30.558589 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2759aa4-d678-4552-94ba-cdab90a866f7-istio-data\") pod \"maas-default-gateway-openshift-default-8f6f87f59-56c5v\" (UID: \"e2759aa4-d678-4552-94ba-cdab90a866f7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-56c5v" Oct 31 00:34:30.558786 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:34:30.558614 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxjjd\" (UniqueName: \"kubernetes.io/projected/e2759aa4-d678-4552-94ba-cdab90a866f7-kube-api-access-qxjjd\") pod \"maas-default-gateway-openshift-default-8f6f87f59-56c5v\" (UID: \"e2759aa4-d678-4552-94ba-cdab90a866f7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-56c5v" Oct 31 00:34:30.558786 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:34:30.558646 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2759aa4-d678-4552-94ba-cdab90a866f7-workload-certs\") pod \"maas-default-gateway-openshift-default-8f6f87f59-56c5v\" (UID: \"e2759aa4-d678-4552-94ba-cdab90a866f7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-56c5v" Oct 31 00:34:30.558786 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:34:30.558672 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2759aa4-d678-4552-94ba-cdab90a866f7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8f6f87f59-56c5v\" (UID: \"e2759aa4-d678-4552-94ba-cdab90a866f7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-56c5v" Oct 31 00:34:30.559443 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:34:30.559411 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2759aa4-d678-4552-94ba-cdab90a866f7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8f6f87f59-56c5v\" (UID: \"e2759aa4-d678-4552-94ba-cdab90a866f7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-56c5v" Oct 31 00:34:30.559627 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:34:30.559540 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2759aa4-d678-4552-94ba-cdab90a866f7-credential-socket\") pod \"maas-default-gateway-openshift-default-8f6f87f59-56c5v\" (UID: \"e2759aa4-d678-4552-94ba-cdab90a866f7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-56c5v" Oct 31 00:34:30.559719 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:34:30.559634 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2759aa4-d678-4552-94ba-cdab90a866f7-istio-data\") pod \"maas-default-gateway-openshift-default-8f6f87f59-56c5v\" (UID: \"e2759aa4-d678-4552-94ba-cdab90a866f7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-56c5v" Oct 31 00:34:30.559917 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:34:30.559888 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2759aa4-d678-4552-94ba-cdab90a866f7-workload-socket\") pod \"maas-default-gateway-openshift-default-8f6f87f59-56c5v\" (UID: \"e2759aa4-d678-4552-94ba-cdab90a866f7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-56c5v" Oct 31 00:34:30.560125 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:34:30.560099 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2759aa4-d678-4552-94ba-cdab90a866f7-workload-certs\") pod \"maas-default-gateway-openshift-default-8f6f87f59-56c5v\" (UID: \"e2759aa4-d678-4552-94ba-cdab90a866f7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-56c5v" Oct 31 00:34:30.562132 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:34:30.562106 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2759aa4-d678-4552-94ba-cdab90a866f7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8f6f87f59-56c5v\" (UID: \"e2759aa4-d678-4552-94ba-cdab90a866f7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-56c5v" Oct 31 00:34:30.562254 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:34:30.562114 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2759aa4-d678-4552-94ba-cdab90a866f7-istio-envoy\") pod \"maas-default-gateway-openshift-default-8f6f87f59-56c5v\" (UID: \"e2759aa4-d678-4552-94ba-cdab90a866f7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-56c5v" Oct 31 00:34:30.625637 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:34:30.625605 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxjjd\" (UniqueName: \"kubernetes.io/projected/e2759aa4-d678-4552-94ba-cdab90a866f7-kube-api-access-qxjjd\") pod \"maas-default-gateway-openshift-default-8f6f87f59-56c5v\" (UID: \"e2759aa4-d678-4552-94ba-cdab90a866f7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-56c5v" Oct 31 00:34:30.627381 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:34:30.627356 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2759aa4-d678-4552-94ba-cdab90a866f7-istio-token\") pod \"maas-default-gateway-openshift-default-8f6f87f59-56c5v\" (UID: \"e2759aa4-d678-4552-94ba-cdab90a866f7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-56c5v" Oct 31 00:34:30.917491 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:34:30.917457 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-56c5v" Oct 31 00:34:31.363192 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:34:31.363166 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-56c5v"] Oct 31 00:34:31.365032 ip-10-0-141-180 kubenswrapper[4927]: W1031 00:34:31.364999 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2759aa4_d678_4552_94ba_cdab90a866f7.slice/crio-2b766d8aca0eaf56912eaa56ef1447d4e084bd5c69015d48cf51036bed218b75 WatchSource:0}: Error finding container 2b766d8aca0eaf56912eaa56ef1447d4e084bd5c69015d48cf51036bed218b75: Status 404 returned error can't find the container with id 2b766d8aca0eaf56912eaa56ef1447d4e084bd5c69015d48cf51036bed218b75 Oct 31 00:34:31.367225 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:34:31.367182 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:34:31.367327 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:34:31.367259 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:34:31.367327 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:34:31.367291 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:34:31.911366 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:34:31.907398 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-56c5v" event={"ID":"e2759aa4-d678-4552-94ba-cdab90a866f7","Type":"ContainerStarted","Data":"08f00f2daf1ca0abe137c5a2fa622b2de94c504f27dd44fa9582172b16cbf232"} Oct 31 00:34:31.911366 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:34:31.907468 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-56c5v" event={"ID":"e2759aa4-d678-4552-94ba-cdab90a866f7","Type":"ContainerStarted","Data":"2b766d8aca0eaf56912eaa56ef1447d4e084bd5c69015d48cf51036bed218b75"} Oct 31 00:34:31.919515 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:34:31.919430 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-56c5v" Oct 31 00:34:32.085740 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:34:32.085691 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-56c5v" podStartSLOduration=2.085677995 podStartE2EDuration="2.085677995s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:34:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:34:32.083764414 +0000 UTC m=+3185.120095257" watchObservedRunningTime="2026-10-31 00:34:32.085677995 +0000 UTC m=+3185.122008816" Oct 31 00:34:32.932501 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:34:32.932461 4927 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-56c5v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.18:15021/healthz/ready\": dial tcp 10.132.1.18:15021: connect: connection refused" start-of-body= Oct 31 00:34:32.932926 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:34:32.932523 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-56c5v" podUID="e2759aa4-d678-4552-94ba-cdab90a866f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.18:15021/healthz/ready\": dial tcp 10.132.1.18:15021: connect: connection refused" Oct 31 00:34:33.644886 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:34:33.644855 4927 scope.go:122] "RemoveContainer" containerID="002ae68aa4fde212d44dc94e9ec1a9a24e875b14a3be99932a6f7f21f795a97d" Oct 31 00:34:33.656520 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:34:33.656487 4927 scope.go:122] "RemoveContainer" containerID="4ea0caf9f5c31d337f75b3402c35dabcc1642bbd258b4674a56fd6417b3fce3a" Oct 31 00:34:33.671505 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:34:33.671478 4927 scope.go:122] "RemoveContainer" containerID="5ff3010631c35b1b8eeb97fa47d4a597d3c9abf24e922a7d18122ed165c36f1b" Oct 31 00:34:33.679644 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:34:33.679622 4927 scope.go:122] "RemoveContainer" containerID="3cba08619886d1ebe196206e8895cbc422324fc8e3d2626f13c9a70ae3827283" Oct 31 00:34:33.687520 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:34:33.687500 4927 scope.go:122] "RemoveContainer" containerID="972ad31fdb50e0c19c46f8b0620a553cd610bb681037e0ae9322cb27883e7793" Oct 31 00:34:33.930363 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:34:33.924804 4927 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-56c5v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.18:15021/healthz/ready\": dial tcp 10.132.1.18:15021: connect: connection refused" start-of-body= Oct 31 00:34:33.930363 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:34:33.924869 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-56c5v" podUID="e2759aa4-d678-4552-94ba-cdab90a866f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.18:15021/healthz/ready\": dial tcp 10.132.1.18:15021: connect: connection refused" Oct 31 00:34:34.921790 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:34:34.921749 4927 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-56c5v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.18:15021/healthz/ready\": dial tcp 10.132.1.18:15021: connect: connection refused" start-of-body= Oct 31 00:34:34.922479 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:34:34.922438 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-56c5v" podUID="e2759aa4-d678-4552-94ba-cdab90a866f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.18:15021/healthz/ready\": dial tcp 10.132.1.18:15021: connect: connection refused" Oct 31 00:34:35.920766 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:34:35.920726 4927 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-56c5v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.18:15021/healthz/ready\": dial tcp 10.132.1.18:15021: connect: connection refused" start-of-body= Oct 31 00:34:35.921061 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:34:35.921032 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-56c5v" podUID="e2759aa4-d678-4552-94ba-cdab90a866f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.18:15021/healthz/ready\": dial tcp 10.132.1.18:15021: connect: connection refused" Oct 31 00:34:36.340448 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:34:36.338992 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:34:36.926417 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:34:36.923812 4927 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-56c5v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.18:15021/healthz/ready\": dial tcp 10.132.1.18:15021: connect: connection refused" start-of-body= Oct 31 00:34:36.926417 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:34:36.923877 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-56c5v" podUID="e2759aa4-d678-4552-94ba-cdab90a866f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.18:15021/healthz/ready\": dial tcp 10.132.1.18:15021: connect: connection refused" Oct 31 00:34:37.918752 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:34:37.918704 4927 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-56c5v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.18:15021/healthz/ready\": dial tcp 10.132.1.18:15021: connect: connection refused" start-of-body= Oct 31 00:34:37.919170 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:34:37.918778 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-56c5v" podUID="e2759aa4-d678-4552-94ba-cdab90a866f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.18:15021/healthz/ready\": dial tcp 10.132.1.18:15021: connect: connection refused" Oct 31 00:34:39.919476 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:34:39.919435 4927 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-56c5v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.18:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:34:39.919972 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:34:39.919503 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-56c5v" podUID="e2759aa4-d678-4552-94ba-cdab90a866f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.18:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:34:40.961558 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:34:40.961518 4927 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-56c5v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.18:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:34:40.962120 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:34:40.961579 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-56c5v" podUID="e2759aa4-d678-4552-94ba-cdab90a866f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.18:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:34:41.658235 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:34:41.658198 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-56c5v" Oct 31 00:34:41.658567 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:34:41.658543 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-56c5v" Oct 31 00:34:41.659213 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:34:41.659193 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-56c5v" Oct 31 00:34:47.483284 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:34:47.483248 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-mqxxc"] Oct 31 00:34:47.489097 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:34:47.489071 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-mqxxc" Oct 31 00:34:47.544668 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:34:47.544634 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-mqxxc"] Oct 31 00:34:47.573204 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:34:47.573173 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c2731fb-f9c4-4962-8637-2aa44d6acf15-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-mqxxc\" (UID: \"0c2731fb-f9c4-4962-8637-2aa44d6acf15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-mqxxc" Oct 31 00:34:47.573383 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:34:47.573208 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c2731fb-f9c4-4962-8637-2aa44d6acf15-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-mqxxc\" (UID: \"0c2731fb-f9c4-4962-8637-2aa44d6acf15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-mqxxc" Oct 31 00:34:47.573454 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:34:47.573424 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwsrt\" (UniqueName: \"kubernetes.io/projected/0c2731fb-f9c4-4962-8637-2aa44d6acf15-kube-api-access-hwsrt\") pod \"maas-api-revoke-sub-2e1f5b5f-mqxxc\" (UID: \"0c2731fb-f9c4-4962-8637-2aa44d6acf15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-mqxxc" Oct 31 00:34:47.591194 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:34:47.591163 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4fdzs\"" Oct 31 00:34:47.591325 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:34:47.591199 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:34:47.674791 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:34:47.674759 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwsrt\" (UniqueName: \"kubernetes.io/projected/0c2731fb-f9c4-4962-8637-2aa44d6acf15-kube-api-access-hwsrt\") pod \"maas-api-revoke-sub-2e1f5b5f-mqxxc\" (UID: \"0c2731fb-f9c4-4962-8637-2aa44d6acf15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-mqxxc" Oct 31 00:34:47.674988 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:34:47.674825 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c2731fb-f9c4-4962-8637-2aa44d6acf15-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-mqxxc\" (UID: \"0c2731fb-f9c4-4962-8637-2aa44d6acf15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-mqxxc" Oct 31 00:34:47.674988 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:34:47.674842 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c2731fb-f9c4-4962-8637-2aa44d6acf15-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-mqxxc\" (UID: \"0c2731fb-f9c4-4962-8637-2aa44d6acf15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-mqxxc" Oct 31 00:34:47.675488 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:34:47.675468 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c2731fb-f9c4-4962-8637-2aa44d6acf15-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-mqxxc\" (UID: \"0c2731fb-f9c4-4962-8637-2aa44d6acf15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-mqxxc" Oct 31 00:34:47.677288 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:34:47.677266 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c2731fb-f9c4-4962-8637-2aa44d6acf15-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-mqxxc\" (UID: \"0c2731fb-f9c4-4962-8637-2aa44d6acf15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-mqxxc" Oct 31 00:34:47.738001 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:34:47.737930 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwsrt\" (UniqueName: \"kubernetes.io/projected/0c2731fb-f9c4-4962-8637-2aa44d6acf15-kube-api-access-hwsrt\") pod \"maas-api-revoke-sub-2e1f5b5f-mqxxc\" (UID: \"0c2731fb-f9c4-4962-8637-2aa44d6acf15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-mqxxc" Oct 31 00:34:47.803692 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:34:47.803658 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-mqxxc" Oct 31 00:34:48.182484 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:34:48.182453 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-mqxxc"] Oct 31 00:34:48.184112 ip-10-0-141-180 kubenswrapper[4927]: W1031 00:34:48.184079 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c2731fb_f9c4_4962_8637_2aa44d6acf15.slice/crio-4f30b28d12c6c949b901d8cbda9960680d0c1fdfb0266beab30d072ed91d4e9b WatchSource:0}: Error finding container 4f30b28d12c6c949b901d8cbda9960680d0c1fdfb0266beab30d072ed91d4e9b: Status 404 returned error can't find the container with id 4f30b28d12c6c949b901d8cbda9960680d0c1fdfb0266beab30d072ed91d4e9b Oct 31 00:34:49.060233 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:34:49.060201 4927 generic.go:356] "Generic (PLEG): container finished" podID="0c2731fb-f9c4-4962-8637-2aa44d6acf15" containerID="caee348f9db214de9716c2e9bbeddd5e835949f2f03b66424c5bafaa8e189563" exitCode=0 Oct 31 00:34:49.060610 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:34:49.060287 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-mqxxc" event={"ID":"0c2731fb-f9c4-4962-8637-2aa44d6acf15","Type":"ContainerDied","Data":"caee348f9db214de9716c2e9bbeddd5e835949f2f03b66424c5bafaa8e189563"} Oct 31 00:34:49.060610 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:34:49.060360 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-mqxxc" event={"ID":"0c2731fb-f9c4-4962-8637-2aa44d6acf15","Type":"ContainerStarted","Data":"4f30b28d12c6c949b901d8cbda9960680d0c1fdfb0266beab30d072ed91d4e9b"} Oct 31 00:34:49.678369 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:34:49.678313 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sqv69" containerName="extractor" Oct 31 00:34:50.201512 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:34:50.201486 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-mqxxc" Oct 31 00:34:50.299536 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:34:50.299501 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0c2731fb-f9c4-4962-8637-2aa44d6acf15-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c2731fb-f9c4-4962-8637-2aa44d6acf15-maas-api-xks-ca\") pod \"0c2731fb-f9c4-4962-8637-2aa44d6acf15\" (UID: \"0c2731fb-f9c4-4962-8637-2aa44d6acf15\") " Oct 31 00:34:50.299712 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:34:50.299551 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c2731fb-f9c4-4962-8637-2aa44d6acf15-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c2731fb-f9c4-4962-8637-2aa44d6acf15-maas-api-openshift-ca\") pod \"0c2731fb-f9c4-4962-8637-2aa44d6acf15\" (UID: \"0c2731fb-f9c4-4962-8637-2aa44d6acf15\") " Oct 31 00:34:50.299712 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:34:50.299609 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c2731fb-f9c4-4962-8637-2aa44d6acf15-kube-api-access-hwsrt\" (UniqueName: \"kubernetes.io/projected/0c2731fb-f9c4-4962-8637-2aa44d6acf15-kube-api-access-hwsrt\") pod \"0c2731fb-f9c4-4962-8637-2aa44d6acf15\" (UID: \"0c2731fb-f9c4-4962-8637-2aa44d6acf15\") " Oct 31 00:34:50.299994 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:34:50.299963 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c2731fb-f9c4-4962-8637-2aa44d6acf15-maas-api-openshift-ca" pod "0c2731fb-f9c4-4962-8637-2aa44d6acf15" (UID: "0c2731fb-f9c4-4962-8637-2aa44d6acf15"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:34:50.301584 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:34:50.301560 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c2731fb-f9c4-4962-8637-2aa44d6acf15-maas-api-xks-ca" pod "0c2731fb-f9c4-4962-8637-2aa44d6acf15" (UID: "0c2731fb-f9c4-4962-8637-2aa44d6acf15"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:34:50.301683 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:34:50.301661 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c2731fb-f9c4-4962-8637-2aa44d6acf15-kube-api-access-hwsrt" pod "0c2731fb-f9c4-4962-8637-2aa44d6acf15" (UID: "0c2731fb-f9c4-4962-8637-2aa44d6acf15"). InnerVolumeSpecName "kube-api-access-hwsrt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:34:50.400753 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:34:50.400673 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hwsrt\" (UniqueName: \"kubernetes.io/projected/0c2731fb-f9c4-4962-8637-2aa44d6acf15-kube-api-access-hwsrt\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:34:50.400753 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:34:50.400707 4927 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c2731fb-f9c4-4962-8637-2aa44d6acf15-maas-api-xks-ca\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:34:50.400753 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:34:50.400719 4927 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c2731fb-f9c4-4962-8637-2aa44d6acf15-maas-api-openshift-ca\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:34:51.074040 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:34:51.074006 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-mqxxc" Oct 31 00:34:51.074225 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:34:51.074040 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-mqxxc" event={"ID":"0c2731fb-f9c4-4962-8637-2aa44d6acf15","Type":"ContainerDied","Data":"4f30b28d12c6c949b901d8cbda9960680d0c1fdfb0266beab30d072ed91d4e9b"} Oct 31 00:34:51.074225 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:34:51.074123 4927 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4f30b28d12c6c949b901d8cbda9960680d0c1fdfb0266beab30d072ed91d4e9b" Oct 31 00:35:00.072986 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:35:00.072950 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-k2s8w"] Oct 31 00:35:00.073524 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:35:00.073436 4927 cpu_manager.go:414] "containerMap: removing container" podUID="0c2731fb-f9c4-4962-8637-2aa44d6acf15" containerName="revoke-keys" Oct 31 00:35:00.073524 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:35:00.073449 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c2731fb-f9c4-4962-8637-2aa44d6acf15" containerName="revoke-keys" Oct 31 00:35:00.073524 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:35:00.073515 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c2731fb-f9c4-4962-8637-2aa44d6acf15" containerName="revoke-keys" Oct 31 00:35:00.076809 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:35:00.076790 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-k2s8w" Oct 31 00:35:00.133535 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:35:00.133500 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-k2s8w"] Oct 31 00:35:00.180663 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:35:00.180624 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:35:00.180841 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:35:00.180748 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4fdzs\"" Oct 31 00:35:00.189055 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:35:00.189027 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f157e2ce-caed-406b-a000-7bd759cb94df-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-k2s8w\" (UID: \"f157e2ce-caed-406b-a000-7bd759cb94df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-k2s8w" Oct 31 00:35:00.189154 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:35:00.189136 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-72vvw\" (UniqueName: \"kubernetes.io/projected/f157e2ce-caed-406b-a000-7bd759cb94df-kube-api-access-72vvw\") pod \"maas-api-revoke-sub-ec843018-k2s8w\" (UID: \"f157e2ce-caed-406b-a000-7bd759cb94df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-k2s8w" Oct 31 00:35:00.189202 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:35:00.189183 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f157e2ce-caed-406b-a000-7bd759cb94df-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-k2s8w\" (UID: \"f157e2ce-caed-406b-a000-7bd759cb94df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-k2s8w" Oct 31 00:35:00.290033 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:35:00.289965 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72vvw\" (UniqueName: \"kubernetes.io/projected/f157e2ce-caed-406b-a000-7bd759cb94df-kube-api-access-72vvw\") pod \"maas-api-revoke-sub-ec843018-k2s8w\" (UID: \"f157e2ce-caed-406b-a000-7bd759cb94df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-k2s8w" Oct 31 00:35:00.290238 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:35:00.290053 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f157e2ce-caed-406b-a000-7bd759cb94df-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-k2s8w\" (UID: \"f157e2ce-caed-406b-a000-7bd759cb94df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-k2s8w" Oct 31 00:35:00.290238 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:35:00.290155 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f157e2ce-caed-406b-a000-7bd759cb94df-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-k2s8w\" (UID: \"f157e2ce-caed-406b-a000-7bd759cb94df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-k2s8w" Oct 31 00:35:00.290741 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:35:00.290715 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f157e2ce-caed-406b-a000-7bd759cb94df-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-k2s8w\" (UID: \"f157e2ce-caed-406b-a000-7bd759cb94df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-k2s8w" Oct 31 00:35:00.292574 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:35:00.292547 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f157e2ce-caed-406b-a000-7bd759cb94df-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-k2s8w\" (UID: \"f157e2ce-caed-406b-a000-7bd759cb94df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-k2s8w" Oct 31 00:35:00.352731 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:35:00.352655 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-72vvw\" (UniqueName: \"kubernetes.io/projected/f157e2ce-caed-406b-a000-7bd759cb94df-kube-api-access-72vvw\") pod \"maas-api-revoke-sub-ec843018-k2s8w\" (UID: \"f157e2ce-caed-406b-a000-7bd759cb94df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-k2s8w" Oct 31 00:35:00.390715 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:35:00.390678 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-k2s8w" Oct 31 00:35:00.775009 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:35:00.774860 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-k2s8w"] Oct 31 00:35:01.146909 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:35:01.146867 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-k2s8w" event={"ID":"f157e2ce-caed-406b-a000-7bd759cb94df","Type":"ContainerStarted","Data":"537913eaa90ad68dd091a12007c37050a9bc9d9b1e9c650d6d3f910ed9758426"} Oct 31 00:35:01.147325 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:35:01.146953 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-k2s8w" event={"ID":"f157e2ce-caed-406b-a000-7bd759cb94df","Type":"ContainerStarted","Data":"c6401a617340ce6c6b9e04ed4ed348422ac10435a7333136f689ee6cc0b6c614"} Oct 31 00:35:01.271854 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:35:01.271788 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-k2s8w" podStartSLOduration=1.271773501 podStartE2EDuration="1.271773501s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:35:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:35:01.269312007 +0000 UTC m=+3214.305642832" watchObservedRunningTime="2026-10-31 00:35:01.271773501 +0000 UTC m=+3214.308104352" Oct 31 00:35:02.156640 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:35:02.156539 4927 generic.go:356] "Generic (PLEG): container finished" podID="f157e2ce-caed-406b-a000-7bd759cb94df" containerID="537913eaa90ad68dd091a12007c37050a9bc9d9b1e9c650d6d3f910ed9758426" exitCode=0 Oct 31 00:35:02.156640 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:35:02.156587 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-k2s8w" event={"ID":"f157e2ce-caed-406b-a000-7bd759cb94df","Type":"ContainerDied","Data":"537913eaa90ad68dd091a12007c37050a9bc9d9b1e9c650d6d3f910ed9758426"} Oct 31 00:35:03.308083 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:35:03.308050 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-k2s8w" Oct 31 00:35:03.420091 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:35:03.419998 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f157e2ce-caed-406b-a000-7bd759cb94df-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f157e2ce-caed-406b-a000-7bd759cb94df-maas-api-openshift-ca\") pod \"f157e2ce-caed-406b-a000-7bd759cb94df\" (UID: \"f157e2ce-caed-406b-a000-7bd759cb94df\") " Oct 31 00:35:03.420263 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:35:03.420130 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f157e2ce-caed-406b-a000-7bd759cb94df-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f157e2ce-caed-406b-a000-7bd759cb94df-maas-api-xks-ca\") pod \"f157e2ce-caed-406b-a000-7bd759cb94df\" (UID: \"f157e2ce-caed-406b-a000-7bd759cb94df\") " Oct 31 00:35:03.420263 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:35:03.420234 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f157e2ce-caed-406b-a000-7bd759cb94df-kube-api-access-72vvw\" (UniqueName: \"kubernetes.io/projected/f157e2ce-caed-406b-a000-7bd759cb94df-kube-api-access-72vvw\") pod \"f157e2ce-caed-406b-a000-7bd759cb94df\" (UID: \"f157e2ce-caed-406b-a000-7bd759cb94df\") " Oct 31 00:35:03.420478 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:35:03.420450 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f157e2ce-caed-406b-a000-7bd759cb94df-maas-api-openshift-ca" pod "f157e2ce-caed-406b-a000-7bd759cb94df" (UID: "f157e2ce-caed-406b-a000-7bd759cb94df"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:35:03.420628 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:35:03.420554 4927 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f157e2ce-caed-406b-a000-7bd759cb94df-maas-api-openshift-ca\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:35:03.422419 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:35:03.422389 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f157e2ce-caed-406b-a000-7bd759cb94df-kube-api-access-72vvw" pod "f157e2ce-caed-406b-a000-7bd759cb94df" (UID: "f157e2ce-caed-406b-a000-7bd759cb94df"). InnerVolumeSpecName "kube-api-access-72vvw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:35:03.422517 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:35:03.422396 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f157e2ce-caed-406b-a000-7bd759cb94df-maas-api-xks-ca" pod "f157e2ce-caed-406b-a000-7bd759cb94df" (UID: "f157e2ce-caed-406b-a000-7bd759cb94df"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:35:03.521358 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:35:03.521285 4927 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f157e2ce-caed-406b-a000-7bd759cb94df-maas-api-xks-ca\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:35:03.521544 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:35:03.521366 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-72vvw\" (UniqueName: \"kubernetes.io/projected/f157e2ce-caed-406b-a000-7bd759cb94df-kube-api-access-72vvw\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:35:04.169519 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:35:04.169482 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-k2s8w" Oct 31 00:35:04.169519 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:35:04.169502 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-k2s8w" event={"ID":"f157e2ce-caed-406b-a000-7bd759cb94df","Type":"ContainerDied","Data":"c6401a617340ce6c6b9e04ed4ed348422ac10435a7333136f689ee6cc0b6c614"} Oct 31 00:35:04.169757 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:35:04.169562 4927 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c6401a617340ce6c6b9e04ed4ed348422ac10435a7333136f689ee6cc0b6c614" Oct 31 00:35:19.703726 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:35:19.703690 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sqv69" containerName="extractor" Oct 31 00:35:33.748098 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:35:33.748059 4927 scope.go:122] "RemoveContainer" containerID="5a5db4e56ed1644f5ed54fb05f56577e974d70330bf9988c5c7d0cc94b2a8f4b" Oct 31 00:35:35.174314 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:35:35.174276 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-4rx5g"] Oct 31 00:35:35.174749 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:35:35.174733 4927 cpu_manager.go:414] "containerMap: removing container" podUID="f157e2ce-caed-406b-a000-7bd759cb94df" containerName="revoke-keys" Oct 31 00:35:35.174796 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:35:35.174750 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f157e2ce-caed-406b-a000-7bd759cb94df" containerName="revoke-keys" Oct 31 00:35:35.174833 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:35:35.174820 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="f157e2ce-caed-406b-a000-7bd759cb94df" containerName="revoke-keys" Oct 31 00:35:35.179618 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:35:35.179601 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-4rx5g" Oct 31 00:35:35.233176 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:35:35.233138 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-4rx5g"] Oct 31 00:35:35.236591 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:35:35.236565 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce97800b-46eb-4f70-940a-c5d6e46457dc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-4rx5g\" (UID: \"ce97800b-46eb-4f70-940a-c5d6e46457dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-4rx5g" Oct 31 00:35:35.236709 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:35:35.236604 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce97800b-46eb-4f70-940a-c5d6e46457dc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-4rx5g\" (UID: \"ce97800b-46eb-4f70-940a-c5d6e46457dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-4rx5g" Oct 31 00:35:35.236752 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:35:35.236698 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t6x22\" (UniqueName: \"kubernetes.io/projected/ce97800b-46eb-4f70-940a-c5d6e46457dc-kube-api-access-t6x22\") pod \"maas-api-revoke-sub-046e1343-4rx5g\" (UID: \"ce97800b-46eb-4f70-940a-c5d6e46457dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-4rx5g" Oct 31 00:35:35.281013 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:35:35.280979 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:35:35.281164 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:35:35.281048 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4fdzs\"" Oct 31 00:35:35.337962 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:35:35.337933 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce97800b-46eb-4f70-940a-c5d6e46457dc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-4rx5g\" (UID: \"ce97800b-46eb-4f70-940a-c5d6e46457dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-4rx5g" Oct 31 00:35:35.338139 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:35:35.337987 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6x22\" (UniqueName: \"kubernetes.io/projected/ce97800b-46eb-4f70-940a-c5d6e46457dc-kube-api-access-t6x22\") pod \"maas-api-revoke-sub-046e1343-4rx5g\" (UID: \"ce97800b-46eb-4f70-940a-c5d6e46457dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-4rx5g" Oct 31 00:35:35.338139 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:35:35.338058 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce97800b-46eb-4f70-940a-c5d6e46457dc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-4rx5g\" (UID: \"ce97800b-46eb-4f70-940a-c5d6e46457dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-4rx5g" Oct 31 00:35:35.338750 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:35:35.338725 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce97800b-46eb-4f70-940a-c5d6e46457dc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-4rx5g\" (UID: \"ce97800b-46eb-4f70-940a-c5d6e46457dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-4rx5g" Oct 31 00:35:35.340451 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:35:35.340428 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce97800b-46eb-4f70-940a-c5d6e46457dc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-4rx5g\" (UID: \"ce97800b-46eb-4f70-940a-c5d6e46457dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-4rx5g" Oct 31 00:35:35.401056 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:35:35.401022 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6x22\" (UniqueName: \"kubernetes.io/projected/ce97800b-46eb-4f70-940a-c5d6e46457dc-kube-api-access-t6x22\") pod \"maas-api-revoke-sub-046e1343-4rx5g\" (UID: \"ce97800b-46eb-4f70-940a-c5d6e46457dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-4rx5g" Oct 31 00:35:35.492714 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:35:35.492633 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-4rx5g" Oct 31 00:35:35.871613 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:35:35.871588 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-4rx5g"] Oct 31 00:35:35.873721 ip-10-0-141-180 kubenswrapper[4927]: W1031 00:35:35.873689 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce97800b_46eb_4f70_940a_c5d6e46457dc.slice/crio-78e2582296600d1d6a3e7335f537bb9e3c2a487eeef9fb6ca856868e9d7739fe WatchSource:0}: Error finding container 78e2582296600d1d6a3e7335f537bb9e3c2a487eeef9fb6ca856868e9d7739fe: Status 404 returned error can't find the container with id 78e2582296600d1d6a3e7335f537bb9e3c2a487eeef9fb6ca856868e9d7739fe Oct 31 00:35:36.385958 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:35:36.385921 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-4rx5g" event={"ID":"ce97800b-46eb-4f70-940a-c5d6e46457dc","Type":"ContainerStarted","Data":"466122ee70bc030cc0e6947e1986710968a71186a7a6b25ec4f25f193b09e3d7"} Oct 31 00:35:36.386446 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:35:36.385980 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-4rx5g" event={"ID":"ce97800b-46eb-4f70-940a-c5d6e46457dc","Type":"ContainerStarted","Data":"78e2582296600d1d6a3e7335f537bb9e3c2a487eeef9fb6ca856868e9d7739fe"} Oct 31 00:35:36.506548 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:35:36.506491 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-4rx5g" podStartSLOduration=1.5064775 podStartE2EDuration="1.5064775s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:35:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:35:36.503951917 +0000 UTC m=+3249.540282740" watchObservedRunningTime="2026-10-31 00:35:36.5064775 +0000 UTC m=+3249.542808345" Oct 31 00:35:37.393833 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:35:37.393789 4927 generic.go:356] "Generic (PLEG): container finished" podID="ce97800b-46eb-4f70-940a-c5d6e46457dc" containerID="466122ee70bc030cc0e6947e1986710968a71186a7a6b25ec4f25f193b09e3d7" exitCode=0 Oct 31 00:35:37.394232 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:35:37.393834 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-4rx5g" event={"ID":"ce97800b-46eb-4f70-940a-c5d6e46457dc","Type":"ContainerDied","Data":"466122ee70bc030cc0e6947e1986710968a71186a7a6b25ec4f25f193b09e3d7"} Oct 31 00:35:38.371018 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:35:38.370979 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:35:38.554318 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:35:38.554295 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-4rx5g" Oct 31 00:35:38.671541 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:35:38.671504 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce97800b-46eb-4f70-940a-c5d6e46457dc-kube-api-access-t6x22\" (UniqueName: \"kubernetes.io/projected/ce97800b-46eb-4f70-940a-c5d6e46457dc-kube-api-access-t6x22\") pod \"ce97800b-46eb-4f70-940a-c5d6e46457dc\" (UID: \"ce97800b-46eb-4f70-940a-c5d6e46457dc\") " Oct 31 00:35:38.671750 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:35:38.671596 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ce97800b-46eb-4f70-940a-c5d6e46457dc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce97800b-46eb-4f70-940a-c5d6e46457dc-maas-api-xks-ca\") pod \"ce97800b-46eb-4f70-940a-c5d6e46457dc\" (UID: \"ce97800b-46eb-4f70-940a-c5d6e46457dc\") " Oct 31 00:35:38.671750 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:35:38.671674 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce97800b-46eb-4f70-940a-c5d6e46457dc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce97800b-46eb-4f70-940a-c5d6e46457dc-maas-api-openshift-ca\") pod \"ce97800b-46eb-4f70-940a-c5d6e46457dc\" (UID: \"ce97800b-46eb-4f70-940a-c5d6e46457dc\") " Oct 31 00:35:38.672859 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:35:38.672379 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce97800b-46eb-4f70-940a-c5d6e46457dc-maas-api-openshift-ca" pod "ce97800b-46eb-4f70-940a-c5d6e46457dc" (UID: "ce97800b-46eb-4f70-940a-c5d6e46457dc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:35:38.674656 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:35:38.674539 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce97800b-46eb-4f70-940a-c5d6e46457dc-kube-api-access-t6x22" pod "ce97800b-46eb-4f70-940a-c5d6e46457dc" (UID: "ce97800b-46eb-4f70-940a-c5d6e46457dc"). InnerVolumeSpecName "kube-api-access-t6x22". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:35:38.676598 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:35:38.676571 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce97800b-46eb-4f70-940a-c5d6e46457dc-maas-api-xks-ca" pod "ce97800b-46eb-4f70-940a-c5d6e46457dc" (UID: "ce97800b-46eb-4f70-940a-c5d6e46457dc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:35:38.772467 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:35:38.772431 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t6x22\" (UniqueName: \"kubernetes.io/projected/ce97800b-46eb-4f70-940a-c5d6e46457dc-kube-api-access-t6x22\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:35:38.772467 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:35:38.772465 4927 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce97800b-46eb-4f70-940a-c5d6e46457dc-maas-api-xks-ca\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:35:38.772467 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:35:38.772476 4927 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce97800b-46eb-4f70-940a-c5d6e46457dc-maas-api-openshift-ca\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:35:39.408178 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:35:39.408148 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-4rx5g" Oct 31 00:35:39.408356 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:35:39.408177 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-4rx5g" event={"ID":"ce97800b-46eb-4f70-940a-c5d6e46457dc","Type":"ContainerDied","Data":"78e2582296600d1d6a3e7335f537bb9e3c2a487eeef9fb6ca856868e9d7739fe"} Oct 31 00:35:39.408356 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:35:39.408223 4927 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="78e2582296600d1d6a3e7335f537bb9e3c2a487eeef9fb6ca856868e9d7739fe" Oct 31 00:35:45.952790 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:35:45.952753 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-6nz9m"] Oct 31 00:35:45.953193 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:35:45.953175 4927 cpu_manager.go:414] "containerMap: removing container" podUID="ce97800b-46eb-4f70-940a-c5d6e46457dc" containerName="revoke-keys" Oct 31 00:35:45.953193 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:35:45.953189 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce97800b-46eb-4f70-940a-c5d6e46457dc" containerName="revoke-keys" Oct 31 00:35:45.953295 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:35:45.953285 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce97800b-46eb-4f70-940a-c5d6e46457dc" containerName="revoke-keys" Oct 31 00:35:45.956767 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:35:45.956750 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-6nz9m" Oct 31 00:35:46.012956 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:35:46.012920 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-6nz9m"] Oct 31 00:35:46.042103 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:35:46.042072 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7015ec94-e3c9-45c9-891b-7569c390bec7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-6nz9m\" (UID: \"7015ec94-e3c9-45c9-891b-7569c390bec7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-6nz9m" Oct 31 00:35:46.042265 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:35:46.042115 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7015ec94-e3c9-45c9-891b-7569c390bec7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-6nz9m\" (UID: \"7015ec94-e3c9-45c9-891b-7569c390bec7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-6nz9m" Oct 31 00:35:46.042265 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:35:46.042151 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jhdd\" (UniqueName: \"kubernetes.io/projected/7015ec94-e3c9-45c9-891b-7569c390bec7-kube-api-access-8jhdd\") pod \"maas-api-revoke-sub-d3fba056-6nz9m\" (UID: \"7015ec94-e3c9-45c9-891b-7569c390bec7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-6nz9m" Oct 31 00:35:46.060087 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:35:46.060062 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:35:46.060203 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:35:46.060150 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4fdzs\"" Oct 31 00:35:46.143689 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:35:46.143661 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7015ec94-e3c9-45c9-891b-7569c390bec7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-6nz9m\" (UID: \"7015ec94-e3c9-45c9-891b-7569c390bec7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-6nz9m" Oct 31 00:35:46.143829 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:35:46.143701 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7015ec94-e3c9-45c9-891b-7569c390bec7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-6nz9m\" (UID: \"7015ec94-e3c9-45c9-891b-7569c390bec7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-6nz9m" Oct 31 00:35:46.143829 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:35:46.143734 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jhdd\" (UniqueName: \"kubernetes.io/projected/7015ec94-e3c9-45c9-891b-7569c390bec7-kube-api-access-8jhdd\") pod \"maas-api-revoke-sub-d3fba056-6nz9m\" (UID: \"7015ec94-e3c9-45c9-891b-7569c390bec7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-6nz9m" Oct 31 00:35:46.144490 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:35:46.144462 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7015ec94-e3c9-45c9-891b-7569c390bec7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-6nz9m\" (UID: \"7015ec94-e3c9-45c9-891b-7569c390bec7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-6nz9m" Oct 31 00:35:46.146087 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:35:46.146069 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7015ec94-e3c9-45c9-891b-7569c390bec7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-6nz9m\" (UID: \"7015ec94-e3c9-45c9-891b-7569c390bec7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-6nz9m" Oct 31 00:35:46.208052 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:35:46.207984 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jhdd\" (UniqueName: \"kubernetes.io/projected/7015ec94-e3c9-45c9-891b-7569c390bec7-kube-api-access-8jhdd\") pod \"maas-api-revoke-sub-d3fba056-6nz9m\" (UID: \"7015ec94-e3c9-45c9-891b-7569c390bec7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-6nz9m" Oct 31 00:35:46.269915 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:35:46.269887 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-6nz9m" Oct 31 00:35:46.654327 ip-10-0-141-180 kubenswrapper[4927]: W1031 00:35:46.654295 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7015ec94_e3c9_45c9_891b_7569c390bec7.slice/crio-934c54162d13ddae421f014f8473aa7b93241540a9e3e2eb429c902461aaa46e WatchSource:0}: Error finding container 934c54162d13ddae421f014f8473aa7b93241540a9e3e2eb429c902461aaa46e: Status 404 returned error can't find the container with id 934c54162d13ddae421f014f8473aa7b93241540a9e3e2eb429c902461aaa46e Oct 31 00:35:46.654458 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:35:46.654432 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-6nz9m"] Oct 31 00:35:47.461683 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:35:47.461639 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-6nz9m" event={"ID":"7015ec94-e3c9-45c9-891b-7569c390bec7","Type":"ContainerStarted","Data":"6893c10707da6b8a4066ef0bf66e7f4891096d87adcb123b53ec280ed4002809"} Oct 31 00:35:47.462089 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:35:47.461707 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-6nz9m" event={"ID":"7015ec94-e3c9-45c9-891b-7569c390bec7","Type":"ContainerStarted","Data":"934c54162d13ddae421f014f8473aa7b93241540a9e3e2eb429c902461aaa46e"} Oct 31 00:35:47.581413 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:35:47.581355 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-6nz9m" podStartSLOduration=2.5813137729999998 podStartE2EDuration="2.581313773s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:35:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:35:47.580573026 +0000 UTC m=+3260.616903850" watchObservedRunningTime="2026-10-31 00:35:47.581313773 +0000 UTC m=+3260.617644599" Oct 31 00:35:48.469381 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:35:48.469349 4927 generic.go:356] "Generic (PLEG): container finished" podID="7015ec94-e3c9-45c9-891b-7569c390bec7" containerID="6893c10707da6b8a4066ef0bf66e7f4891096d87adcb123b53ec280ed4002809" exitCode=0 Oct 31 00:35:48.469860 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:35:48.469429 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-6nz9m" event={"ID":"7015ec94-e3c9-45c9-891b-7569c390bec7","Type":"ContainerDied","Data":"6893c10707da6b8a4066ef0bf66e7f4891096d87adcb123b53ec280ed4002809"} Oct 31 00:35:49.751356 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:35:49.745201 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sqv69" containerName="extractor" Oct 31 00:35:49.989085 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:35:49.989046 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-6nz9m" Oct 31 00:35:50.107195 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:35:50.107162 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7015ec94-e3c9-45c9-891b-7569c390bec7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7015ec94-e3c9-45c9-891b-7569c390bec7-maas-api-xks-ca\") pod \"7015ec94-e3c9-45c9-891b-7569c390bec7\" (UID: \"7015ec94-e3c9-45c9-891b-7569c390bec7\") " Oct 31 00:35:50.107422 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:35:50.107246 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7015ec94-e3c9-45c9-891b-7569c390bec7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7015ec94-e3c9-45c9-891b-7569c390bec7-maas-api-openshift-ca\") pod \"7015ec94-e3c9-45c9-891b-7569c390bec7\" (UID: \"7015ec94-e3c9-45c9-891b-7569c390bec7\") " Oct 31 00:35:50.107422 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:35:50.107363 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7015ec94-e3c9-45c9-891b-7569c390bec7-kube-api-access-8jhdd\" (UniqueName: \"kubernetes.io/projected/7015ec94-e3c9-45c9-891b-7569c390bec7-kube-api-access-8jhdd\") pod \"7015ec94-e3c9-45c9-891b-7569c390bec7\" (UID: \"7015ec94-e3c9-45c9-891b-7569c390bec7\") " Oct 31 00:35:50.107673 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:35:50.107638 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7015ec94-e3c9-45c9-891b-7569c390bec7-maas-api-openshift-ca" pod "7015ec94-e3c9-45c9-891b-7569c390bec7" (UID: "7015ec94-e3c9-45c9-891b-7569c390bec7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:35:50.136011 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:35:50.135957 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7015ec94-e3c9-45c9-891b-7569c390bec7-maas-api-xks-ca" pod "7015ec94-e3c9-45c9-891b-7569c390bec7" (UID: "7015ec94-e3c9-45c9-891b-7569c390bec7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:35:50.194178 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:35:50.194099 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7015ec94-e3c9-45c9-891b-7569c390bec7-kube-api-access-8jhdd" pod "7015ec94-e3c9-45c9-891b-7569c390bec7" (UID: "7015ec94-e3c9-45c9-891b-7569c390bec7"). InnerVolumeSpecName "kube-api-access-8jhdd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:35:50.211011 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:35:50.210975 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8jhdd\" (UniqueName: \"kubernetes.io/projected/7015ec94-e3c9-45c9-891b-7569c390bec7-kube-api-access-8jhdd\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:35:50.223439 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:35:50.223399 4927 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7015ec94-e3c9-45c9-891b-7569c390bec7-maas-api-xks-ca\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:35:50.223641 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:35:50.223623 4927 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7015ec94-e3c9-45c9-891b-7569c390bec7-maas-api-openshift-ca\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:35:50.604408 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:35:50.604305 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-6nz9m" Oct 31 00:35:50.604408 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:35:50.604357 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-6nz9m" event={"ID":"7015ec94-e3c9-45c9-891b-7569c390bec7","Type":"ContainerDied","Data":"934c54162d13ddae421f014f8473aa7b93241540a9e3e2eb429c902461aaa46e"} Oct 31 00:35:50.604602 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:35:50.604416 4927 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="934c54162d13ddae421f014f8473aa7b93241540a9e3e2eb429c902461aaa46e" Oct 31 00:35:55.053097 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:35:55.053055 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c93075a7-6dfxg"] Oct 31 00:35:55.106510 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:35:55.106466 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c93075a7-6dfxg"] Oct 31 00:35:56.054078 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:35:56.054040 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b940a211-vfr92"] Oct 31 00:35:56.105313 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:35:56.105280 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b940a211-vfr92"] Oct 31 00:35:57.085307 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:35:57.085261 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="443e190a-3873-4532-92da-5324fb3ddf18" path="/var/lib/kubelet/pods/443e190a-3873-4532-92da-5324fb3ddf18/volumes" Oct 31 00:35:57.085905 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:35:57.085883 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd7ab0e9-13cb-45b7-8a57-0f82d4895093" path="/var/lib/kubelet/pods/bd7ab0e9-13cb-45b7-8a57-0f82d4895093/volumes" Oct 31 00:36:19.677139 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:36:19.677099 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sqv69" containerName="extractor" Oct 31 00:36:23.798365 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:36:23.798311 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-2nngx"] Oct 31 00:36:23.799002 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:36:23.798980 4927 cpu_manager.go:414] "containerMap: removing container" podUID="7015ec94-e3c9-45c9-891b-7569c390bec7" containerName="revoke-keys" Oct 31 00:36:23.799100 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:36:23.799004 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7015ec94-e3c9-45c9-891b-7569c390bec7" containerName="revoke-keys" Oct 31 00:36:23.799160 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:36:23.799140 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="7015ec94-e3c9-45c9-891b-7569c390bec7" containerName="revoke-keys" Oct 31 00:36:23.802728 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:36:23.802704 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-2nngx" Oct 31 00:36:23.857696 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:36:23.857661 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-2nngx"] Oct 31 00:36:23.900110 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:36:23.900074 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/12635fcb-b16d-47d3-b03d-c173ffd4e612-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-2nngx\" (UID: \"12635fcb-b16d-47d3-b03d-c173ffd4e612\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-2nngx" Oct 31 00:36:23.900268 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:36:23.900117 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/12635fcb-b16d-47d3-b03d-c173ffd4e612-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-2nngx\" (UID: \"12635fcb-b16d-47d3-b03d-c173ffd4e612\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-2nngx" Oct 31 00:36:23.900312 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:36:23.900262 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xflbx\" (UniqueName: \"kubernetes.io/projected/12635fcb-b16d-47d3-b03d-c173ffd4e612-kube-api-access-xflbx\") pod \"maas-api-revoke-sub-ae594979-2nngx\" (UID: \"12635fcb-b16d-47d3-b03d-c173ffd4e612\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-2nngx" Oct 31 00:36:23.905868 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:36:23.905842 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:36:23.905991 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:36:23.905902 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4fdzs\"" Oct 31 00:36:24.000905 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:36:24.000878 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xflbx\" (UniqueName: \"kubernetes.io/projected/12635fcb-b16d-47d3-b03d-c173ffd4e612-kube-api-access-xflbx\") pod \"maas-api-revoke-sub-ae594979-2nngx\" (UID: \"12635fcb-b16d-47d3-b03d-c173ffd4e612\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-2nngx" Oct 31 00:36:24.001092 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:36:24.000923 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/12635fcb-b16d-47d3-b03d-c173ffd4e612-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-2nngx\" (UID: \"12635fcb-b16d-47d3-b03d-c173ffd4e612\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-2nngx" Oct 31 00:36:24.001092 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:36:24.000948 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/12635fcb-b16d-47d3-b03d-c173ffd4e612-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-2nngx\" (UID: \"12635fcb-b16d-47d3-b03d-c173ffd4e612\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-2nngx" Oct 31 00:36:24.001585 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:36:24.001564 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/12635fcb-b16d-47d3-b03d-c173ffd4e612-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-2nngx\" (UID: \"12635fcb-b16d-47d3-b03d-c173ffd4e612\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-2nngx" Oct 31 00:36:24.003317 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:36:24.003295 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/12635fcb-b16d-47d3-b03d-c173ffd4e612-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-2nngx\" (UID: \"12635fcb-b16d-47d3-b03d-c173ffd4e612\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-2nngx" Oct 31 00:36:24.063101 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:36:24.063037 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xflbx\" (UniqueName: \"kubernetes.io/projected/12635fcb-b16d-47d3-b03d-c173ffd4e612-kube-api-access-xflbx\") pod \"maas-api-revoke-sub-ae594979-2nngx\" (UID: \"12635fcb-b16d-47d3-b03d-c173ffd4e612\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-2nngx" Oct 31 00:36:24.115705 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:36:24.115675 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-2nngx" Oct 31 00:36:24.493120 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:36:24.493086 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-2nngx"] Oct 31 00:36:24.495180 ip-10-0-141-180 kubenswrapper[4927]: W1031 00:36:24.495149 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12635fcb_b16d_47d3_b03d_c173ffd4e612.slice/crio-a44ef9838622c5d55de8259c1f887daed72ecc56bee3ffd4a24b7cd34e845347 WatchSource:0}: Error finding container a44ef9838622c5d55de8259c1f887daed72ecc56bee3ffd4a24b7cd34e845347: Status 404 returned error can't find the container with id a44ef9838622c5d55de8259c1f887daed72ecc56bee3ffd4a24b7cd34e845347 Oct 31 00:36:24.852590 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:36:24.852494 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-2nngx" event={"ID":"12635fcb-b16d-47d3-b03d-c173ffd4e612","Type":"ContainerStarted","Data":"0dd237469a44b2255386c7e6d036c069ca460d1caa6b2c02b745c46e465460bb"} Oct 31 00:36:24.852590 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:36:24.852559 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-2nngx" event={"ID":"12635fcb-b16d-47d3-b03d-c173ffd4e612","Type":"ContainerStarted","Data":"a44ef9838622c5d55de8259c1f887daed72ecc56bee3ffd4a24b7cd34e845347"} Oct 31 00:36:24.974116 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:36:24.974052 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-2nngx" podStartSLOduration=1.974034666 podStartE2EDuration="1.974034666s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:36:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:36:24.971645333 +0000 UTC m=+3298.007976155" watchObservedRunningTime="2026-10-31 00:36:24.974034666 +0000 UTC m=+3298.010365490" Oct 31 00:36:25.860110 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:36:25.860068 4927 generic.go:356] "Generic (PLEG): container finished" podID="12635fcb-b16d-47d3-b03d-c173ffd4e612" containerID="0dd237469a44b2255386c7e6d036c069ca460d1caa6b2c02b745c46e465460bb" exitCode=0 Oct 31 00:36:25.860605 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:36:25.860160 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-2nngx" event={"ID":"12635fcb-b16d-47d3-b03d-c173ffd4e612","Type":"ContainerDied","Data":"0dd237469a44b2255386c7e6d036c069ca460d1caa6b2c02b745c46e465460bb"} Oct 31 00:36:27.013822 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:36:27.013794 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-2nngx" Oct 31 00:36:27.032194 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:36:27.032166 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/12635fcb-b16d-47d3-b03d-c173ffd4e612-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/12635fcb-b16d-47d3-b03d-c173ffd4e612-maas-api-xks-ca\") pod \"12635fcb-b16d-47d3-b03d-c173ffd4e612\" (UID: \"12635fcb-b16d-47d3-b03d-c173ffd4e612\") " Oct 31 00:36:27.032319 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:36:27.032206 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/12635fcb-b16d-47d3-b03d-c173ffd4e612-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/12635fcb-b16d-47d3-b03d-c173ffd4e612-maas-api-openshift-ca\") pod \"12635fcb-b16d-47d3-b03d-c173ffd4e612\" (UID: \"12635fcb-b16d-47d3-b03d-c173ffd4e612\") " Oct 31 00:36:27.032319 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:36:27.032283 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12635fcb-b16d-47d3-b03d-c173ffd4e612-kube-api-access-xflbx\" (UniqueName: \"kubernetes.io/projected/12635fcb-b16d-47d3-b03d-c173ffd4e612-kube-api-access-xflbx\") pod \"12635fcb-b16d-47d3-b03d-c173ffd4e612\" (UID: \"12635fcb-b16d-47d3-b03d-c173ffd4e612\") " Oct 31 00:36:27.032670 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:36:27.032643 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12635fcb-b16d-47d3-b03d-c173ffd4e612-maas-api-openshift-ca" pod "12635fcb-b16d-47d3-b03d-c173ffd4e612" (UID: "12635fcb-b16d-47d3-b03d-c173ffd4e612"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:36:27.034282 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:36:27.034255 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/12635fcb-b16d-47d3-b03d-c173ffd4e612-maas-api-xks-ca" pod "12635fcb-b16d-47d3-b03d-c173ffd4e612" (UID: "12635fcb-b16d-47d3-b03d-c173ffd4e612"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:36:27.034407 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:36:27.034298 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12635fcb-b16d-47d3-b03d-c173ffd4e612-kube-api-access-xflbx" pod "12635fcb-b16d-47d3-b03d-c173ffd4e612" (UID: "12635fcb-b16d-47d3-b03d-c173ffd4e612"). InnerVolumeSpecName "kube-api-access-xflbx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:36:27.133883 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:36:27.133795 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xflbx\" (UniqueName: \"kubernetes.io/projected/12635fcb-b16d-47d3-b03d-c173ffd4e612-kube-api-access-xflbx\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:36:27.133883 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:36:27.133834 4927 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/12635fcb-b16d-47d3-b03d-c173ffd4e612-maas-api-xks-ca\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:36:27.133883 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:36:27.133863 4927 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/12635fcb-b16d-47d3-b03d-c173ffd4e612-maas-api-openshift-ca\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:36:27.873982 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:36:27.873945 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-2nngx" Oct 31 00:36:27.873982 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:36:27.873971 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-2nngx" event={"ID":"12635fcb-b16d-47d3-b03d-c173ffd4e612","Type":"ContainerDied","Data":"a44ef9838622c5d55de8259c1f887daed72ecc56bee3ffd4a24b7cd34e845347"} Oct 31 00:36:27.874186 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:36:27.874041 4927 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a44ef9838622c5d55de8259c1f887daed72ecc56bee3ffd4a24b7cd34e845347" Oct 31 00:36:28.790271 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:36:28.790243 4927 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-pqt2c_6382dfb5-a363-47a2-bc79-0b96f7bec0c3/console-operator/3.log" Oct 31 00:36:29.815376 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:36:29.815347 4927 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-pqt2c_6382dfb5-a363-47a2-bc79-0b96f7bec0c3/console-operator/3.log" Oct 31 00:36:31.171315 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:36:31.171277 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:36:33.771193 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:36:33.771162 4927 scope.go:122] "RemoveContainer" containerID="0d80369c859d795b1b31f3bf521ebbd098d7394a31875fdcf0671b79df513690" Oct 31 00:36:33.780657 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:36:33.780636 4927 scope.go:122] "RemoveContainer" containerID="54629f8914550a4458602e3c6b5dca77b4c74e4289f7812b2b9ce7cd2e8ebec9" Oct 31 00:36:37.049732 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:36:37.049692 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:36:37.415895 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:36:37.415856 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:36:49.677310 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:36:49.677274 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sqv69" containerName="extractor" Oct 31 00:36:51.881479 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:36:51.881441 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-kgzrm"] Oct 31 00:36:51.881885 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:36:51.881870 4927 cpu_manager.go:414] "containerMap: removing container" podUID="12635fcb-b16d-47d3-b03d-c173ffd4e612" containerName="revoke-keys" Oct 31 00:36:51.881931 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:36:51.881886 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12635fcb-b16d-47d3-b03d-c173ffd4e612" containerName="revoke-keys" Oct 31 00:36:51.881979 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:36:51.881970 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="12635fcb-b16d-47d3-b03d-c173ffd4e612" containerName="revoke-keys" Oct 31 00:36:51.886691 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:36:51.886671 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-kgzrm" Oct 31 00:36:51.939859 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:36:51.939823 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-kgzrm"] Oct 31 00:36:51.940674 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:36:51.940644 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:36:51.940842 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:36:51.940702 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4fdzs\"" Oct 31 00:36:51.979628 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:36:51.979599 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-klcl4\" (UniqueName: \"kubernetes.io/projected/f57beea2-7bb2-464d-8e97-506b915456fa-kube-api-access-klcl4\") pod \"maas-api-revoke-sub-a4aef7d7-kgzrm\" (UID: \"f57beea2-7bb2-464d-8e97-506b915456fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-kgzrm" Oct 31 00:36:51.979765 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:36:51.979670 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f57beea2-7bb2-464d-8e97-506b915456fa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-kgzrm\" (UID: \"f57beea2-7bb2-464d-8e97-506b915456fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-kgzrm" Oct 31 00:36:51.979809 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:36:51.979780 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f57beea2-7bb2-464d-8e97-506b915456fa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-kgzrm\" (UID: \"f57beea2-7bb2-464d-8e97-506b915456fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-kgzrm" Oct 31 00:36:52.080744 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:36:52.080714 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f57beea2-7bb2-464d-8e97-506b915456fa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-kgzrm\" (UID: \"f57beea2-7bb2-464d-8e97-506b915456fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-kgzrm" Oct 31 00:36:52.080911 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:36:52.080763 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-klcl4\" (UniqueName: \"kubernetes.io/projected/f57beea2-7bb2-464d-8e97-506b915456fa-kube-api-access-klcl4\") pod \"maas-api-revoke-sub-a4aef7d7-kgzrm\" (UID: \"f57beea2-7bb2-464d-8e97-506b915456fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-kgzrm" Oct 31 00:36:52.080911 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:36:52.080825 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f57beea2-7bb2-464d-8e97-506b915456fa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-kgzrm\" (UID: \"f57beea2-7bb2-464d-8e97-506b915456fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-kgzrm" Oct 31 00:36:52.081549 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:36:52.081528 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f57beea2-7bb2-464d-8e97-506b915456fa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-kgzrm\" (UID: \"f57beea2-7bb2-464d-8e97-506b915456fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-kgzrm" Oct 31 00:36:52.083037 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:36:52.083018 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f57beea2-7bb2-464d-8e97-506b915456fa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-kgzrm\" (UID: \"f57beea2-7bb2-464d-8e97-506b915456fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-kgzrm" Oct 31 00:36:52.153300 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:36:52.153240 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-klcl4\" (UniqueName: \"kubernetes.io/projected/f57beea2-7bb2-464d-8e97-506b915456fa-kube-api-access-klcl4\") pod \"maas-api-revoke-sub-a4aef7d7-kgzrm\" (UID: \"f57beea2-7bb2-464d-8e97-506b915456fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-kgzrm" Oct 31 00:36:52.203159 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:36:52.203131 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-kgzrm" Oct 31 00:36:52.582002 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:36:52.581974 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-kgzrm"] Oct 31 00:36:52.582879 ip-10-0-141-180 kubenswrapper[4927]: W1031 00:36:52.582852 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf57beea2_7bb2_464d_8e97_506b915456fa.slice/crio-7868ff2065f0741b46f001a715460d567a4df70ced205a1dfcc3fe2be2a333ec WatchSource:0}: Error finding container 7868ff2065f0741b46f001a715460d567a4df70ced205a1dfcc3fe2be2a333ec: Status 404 returned error can't find the container with id 7868ff2065f0741b46f001a715460d567a4df70ced205a1dfcc3fe2be2a333ec Oct 31 00:36:53.044868 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:36:53.044825 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-kgzrm" event={"ID":"f57beea2-7bb2-464d-8e97-506b915456fa","Type":"ContainerStarted","Data":"0f008993cd63c3c2d3586b24801da0de284bff0480cd5963bf7411e9472ce76c"} Oct 31 00:36:53.044868 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:36:53.044875 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-kgzrm" event={"ID":"f57beea2-7bb2-464d-8e97-506b915456fa","Type":"ContainerStarted","Data":"7868ff2065f0741b46f001a715460d567a4df70ced205a1dfcc3fe2be2a333ec"} Oct 31 00:36:53.165732 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:36:53.165635 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-kgzrm" podStartSLOduration=2.165622509 podStartE2EDuration="2.165622509s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:36:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:36:53.163397276 +0000 UTC m=+3326.199728121" watchObservedRunningTime="2026-10-31 00:36:53.165622509 +0000 UTC m=+3326.201953330" Oct 31 00:36:54.052470 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:36:54.052430 4927 generic.go:356] "Generic (PLEG): container finished" podID="f57beea2-7bb2-464d-8e97-506b915456fa" containerID="0f008993cd63c3c2d3586b24801da0de284bff0480cd5963bf7411e9472ce76c" exitCode=0 Oct 31 00:36:54.052875 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:36:54.052519 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-kgzrm" event={"ID":"f57beea2-7bb2-464d-8e97-506b915456fa","Type":"ContainerDied","Data":"0f008993cd63c3c2d3586b24801da0de284bff0480cd5963bf7411e9472ce76c"} Oct 31 00:36:55.195483 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:36:55.195459 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-kgzrm" Oct 31 00:36:55.311824 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:36:55.311738 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f57beea2-7bb2-464d-8e97-506b915456fa-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f57beea2-7bb2-464d-8e97-506b915456fa-maas-api-openshift-ca\") pod \"f57beea2-7bb2-464d-8e97-506b915456fa\" (UID: \"f57beea2-7bb2-464d-8e97-506b915456fa\") " Oct 31 00:36:55.311824 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:36:55.311818 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f57beea2-7bb2-464d-8e97-506b915456fa-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f57beea2-7bb2-464d-8e97-506b915456fa-maas-api-xks-ca\") pod \"f57beea2-7bb2-464d-8e97-506b915456fa\" (UID: \"f57beea2-7bb2-464d-8e97-506b915456fa\") " Oct 31 00:36:55.312050 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:36:55.311864 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f57beea2-7bb2-464d-8e97-506b915456fa-kube-api-access-klcl4\" (UniqueName: \"kubernetes.io/projected/f57beea2-7bb2-464d-8e97-506b915456fa-kube-api-access-klcl4\") pod \"f57beea2-7bb2-464d-8e97-506b915456fa\" (UID: \"f57beea2-7bb2-464d-8e97-506b915456fa\") " Oct 31 00:36:55.312238 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:36:55.312193 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f57beea2-7bb2-464d-8e97-506b915456fa-maas-api-openshift-ca" pod "f57beea2-7bb2-464d-8e97-506b915456fa" (UID: "f57beea2-7bb2-464d-8e97-506b915456fa"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:36:55.314266 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:36:55.314238 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f57beea2-7bb2-464d-8e97-506b915456fa-maas-api-xks-ca" pod "f57beea2-7bb2-464d-8e97-506b915456fa" (UID: "f57beea2-7bb2-464d-8e97-506b915456fa"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:36:55.314407 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:36:55.314390 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f57beea2-7bb2-464d-8e97-506b915456fa-kube-api-access-klcl4" pod "f57beea2-7bb2-464d-8e97-506b915456fa" (UID: "f57beea2-7bb2-464d-8e97-506b915456fa"). InnerVolumeSpecName "kube-api-access-klcl4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:36:55.413128 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:36:55.413095 4927 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f57beea2-7bb2-464d-8e97-506b915456fa-maas-api-openshift-ca\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:36:55.413128 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:36:55.413125 4927 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f57beea2-7bb2-464d-8e97-506b915456fa-maas-api-xks-ca\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:36:55.413128 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:36:55.413135 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-klcl4\" (UniqueName: \"kubernetes.io/projected/f57beea2-7bb2-464d-8e97-506b915456fa-kube-api-access-klcl4\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:36:56.065621 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:36:56.065589 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-kgzrm" Oct 31 00:36:56.065621 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:36:56.065596 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-kgzrm" event={"ID":"f57beea2-7bb2-464d-8e97-506b915456fa","Type":"ContainerDied","Data":"7868ff2065f0741b46f001a715460d567a4df70ced205a1dfcc3fe2be2a333ec"} Oct 31 00:36:56.065855 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:36:56.065660 4927 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7868ff2065f0741b46f001a715460d567a4df70ced205a1dfcc3fe2be2a333ec" Oct 31 00:37:14.075509 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:37:14.075468 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5014c66a-mg9r4"] Oct 31 00:37:14.103679 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:37:14.103641 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5014c66a-mg9r4"] Oct 31 00:37:15.085708 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:37:15.085673 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d0c77474-1554-4f73-84ef-524855a5160a" path="/var/lib/kubelet/pods/d0c77474-1554-4f73-84ef-524855a5160a/volumes" Oct 31 00:37:19.677642 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:37:19.677603 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sqv69" containerName="extractor" Oct 31 00:37:23.735689 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:37:23.735654 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-7csdn"] Oct 31 00:37:23.736090 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:37:23.736068 4927 cpu_manager.go:414] "containerMap: removing container" podUID="f57beea2-7bb2-464d-8e97-506b915456fa" containerName="revoke-keys" Oct 31 00:37:23.736090 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:37:23.736079 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f57beea2-7bb2-464d-8e97-506b915456fa" containerName="revoke-keys" Oct 31 00:37:23.736177 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:37:23.736162 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="f57beea2-7bb2-464d-8e97-506b915456fa" containerName="revoke-keys" Oct 31 00:37:23.739607 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:37:23.739591 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-7csdn" Oct 31 00:37:23.794993 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:37:23.794959 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-7csdn"] Oct 31 00:37:23.800064 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:37:23.800035 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e2f5f51f-6279-4993-960e-46bdceb23e84-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-7csdn\" (UID: \"e2f5f51f-6279-4993-960e-46bdceb23e84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-7csdn" Oct 31 00:37:23.800179 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:37:23.800093 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e2f5f51f-6279-4993-960e-46bdceb23e84-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-7csdn\" (UID: \"e2f5f51f-6279-4993-960e-46bdceb23e84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-7csdn" Oct 31 00:37:23.800272 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:37:23.800245 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l6b5l\" (UniqueName: \"kubernetes.io/projected/e2f5f51f-6279-4993-960e-46bdceb23e84-kube-api-access-l6b5l\") pod \"maas-api-revoke-sub-79805ca9-7csdn\" (UID: \"e2f5f51f-6279-4993-960e-46bdceb23e84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-7csdn" Oct 31 00:37:23.842153 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:37:23.842124 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4fdzs\"" Oct 31 00:37:23.842153 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:37:23.842124 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:37:23.901116 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:37:23.901082 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e2f5f51f-6279-4993-960e-46bdceb23e84-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-7csdn\" (UID: \"e2f5f51f-6279-4993-960e-46bdceb23e84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-7csdn" Oct 31 00:37:23.901363 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:37:23.901178 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6b5l\" (UniqueName: \"kubernetes.io/projected/e2f5f51f-6279-4993-960e-46bdceb23e84-kube-api-access-l6b5l\") pod \"maas-api-revoke-sub-79805ca9-7csdn\" (UID: \"e2f5f51f-6279-4993-960e-46bdceb23e84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-7csdn" Oct 31 00:37:23.901363 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:37:23.901245 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e2f5f51f-6279-4993-960e-46bdceb23e84-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-7csdn\" (UID: \"e2f5f51f-6279-4993-960e-46bdceb23e84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-7csdn" Oct 31 00:37:23.901934 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:37:23.901908 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e2f5f51f-6279-4993-960e-46bdceb23e84-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-7csdn\" (UID: \"e2f5f51f-6279-4993-960e-46bdceb23e84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-7csdn" Oct 31 00:37:23.903658 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:37:23.903635 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e2f5f51f-6279-4993-960e-46bdceb23e84-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-7csdn\" (UID: \"e2f5f51f-6279-4993-960e-46bdceb23e84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-7csdn" Oct 31 00:37:23.964060 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:37:23.964020 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6b5l\" (UniqueName: \"kubernetes.io/projected/e2f5f51f-6279-4993-960e-46bdceb23e84-kube-api-access-l6b5l\") pod \"maas-api-revoke-sub-79805ca9-7csdn\" (UID: \"e2f5f51f-6279-4993-960e-46bdceb23e84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-7csdn" Oct 31 00:37:24.009442 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:37:24.009353 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:37:24.053029 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:37:24.053001 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-7csdn" Oct 31 00:37:24.429080 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:37:24.429051 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-7csdn"] Oct 31 00:37:24.430353 ip-10-0-141-180 kubenswrapper[4927]: W1031 00:37:24.430318 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2f5f51f_6279_4993_960e_46bdceb23e84.slice/crio-3dd70a7a4dd43730f11e7f8021bf53f2a1bb3810b98df15a80511d492aacf06f WatchSource:0}: Error finding container 3dd70a7a4dd43730f11e7f8021bf53f2a1bb3810b98df15a80511d492aacf06f: Status 404 returned error can't find the container with id 3dd70a7a4dd43730f11e7f8021bf53f2a1bb3810b98df15a80511d492aacf06f Oct 31 00:37:25.269882 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:37:25.269842 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-7csdn" event={"ID":"e2f5f51f-6279-4993-960e-46bdceb23e84","Type":"ContainerStarted","Data":"93fb83fa0b8a20b7fdbec018b78496c7c4761fa731cdeee28d7dcce8514910dc"} Oct 31 00:37:25.270275 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:37:25.269904 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-7csdn" event={"ID":"e2f5f51f-6279-4993-960e-46bdceb23e84","Type":"ContainerStarted","Data":"3dd70a7a4dd43730f11e7f8021bf53f2a1bb3810b98df15a80511d492aacf06f"} Oct 31 00:37:25.391065 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:37:25.391017 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-7csdn" podStartSLOduration=2.391003955 podStartE2EDuration="2.391003955s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:37:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:37:25.387659847 +0000 UTC m=+3358.423990682" watchObservedRunningTime="2026-10-31 00:37:25.391003955 +0000 UTC m=+3358.427334777" Oct 31 00:37:26.276794 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:37:26.276755 4927 generic.go:356] "Generic (PLEG): container finished" podID="e2f5f51f-6279-4993-960e-46bdceb23e84" containerID="93fb83fa0b8a20b7fdbec018b78496c7c4761fa731cdeee28d7dcce8514910dc" exitCode=0 Oct 31 00:37:26.277217 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:37:26.276841 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-7csdn" event={"ID":"e2f5f51f-6279-4993-960e-46bdceb23e84","Type":"ContainerDied","Data":"93fb83fa0b8a20b7fdbec018b78496c7c4761fa731cdeee28d7dcce8514910dc"} Oct 31 00:37:27.423057 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:37:27.423027 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-7csdn" Oct 31 00:37:27.533078 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:37:27.533044 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e2f5f51f-6279-4993-960e-46bdceb23e84-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e2f5f51f-6279-4993-960e-46bdceb23e84-maas-api-openshift-ca\") pod \"e2f5f51f-6279-4993-960e-46bdceb23e84\" (UID: \"e2f5f51f-6279-4993-960e-46bdceb23e84\") " Oct 31 00:37:27.533285 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:37:27.533098 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2f5f51f-6279-4993-960e-46bdceb23e84-kube-api-access-l6b5l\" (UniqueName: \"kubernetes.io/projected/e2f5f51f-6279-4993-960e-46bdceb23e84-kube-api-access-l6b5l\") pod \"e2f5f51f-6279-4993-960e-46bdceb23e84\" (UID: \"e2f5f51f-6279-4993-960e-46bdceb23e84\") " Oct 31 00:37:27.533285 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:37:27.533212 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e2f5f51f-6279-4993-960e-46bdceb23e84-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e2f5f51f-6279-4993-960e-46bdceb23e84-maas-api-xks-ca\") pod \"e2f5f51f-6279-4993-960e-46bdceb23e84\" (UID: \"e2f5f51f-6279-4993-960e-46bdceb23e84\") " Oct 31 00:37:27.533459 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:37:27.533421 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2f5f51f-6279-4993-960e-46bdceb23e84-maas-api-openshift-ca" pod "e2f5f51f-6279-4993-960e-46bdceb23e84" (UID: "e2f5f51f-6279-4993-960e-46bdceb23e84"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:37:27.535263 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:37:27.535213 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e2f5f51f-6279-4993-960e-46bdceb23e84-maas-api-xks-ca" pod "e2f5f51f-6279-4993-960e-46bdceb23e84" (UID: "e2f5f51f-6279-4993-960e-46bdceb23e84"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:37:27.535414 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:37:27.535393 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2f5f51f-6279-4993-960e-46bdceb23e84-kube-api-access-l6b5l" pod "e2f5f51f-6279-4993-960e-46bdceb23e84" (UID: "e2f5f51f-6279-4993-960e-46bdceb23e84"). InnerVolumeSpecName "kube-api-access-l6b5l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:37:27.634056 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:37:27.634026 4927 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e2f5f51f-6279-4993-960e-46bdceb23e84-maas-api-xks-ca\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:37:27.634056 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:37:27.634055 4927 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e2f5f51f-6279-4993-960e-46bdceb23e84-maas-api-openshift-ca\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:37:27.634056 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:37:27.634068 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l6b5l\" (UniqueName: \"kubernetes.io/projected/e2f5f51f-6279-4993-960e-46bdceb23e84-kube-api-access-l6b5l\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:37:28.291416 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:37:28.291382 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-7csdn" Oct 31 00:37:28.291416 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:37:28.291404 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-7csdn" event={"ID":"e2f5f51f-6279-4993-960e-46bdceb23e84","Type":"ContainerDied","Data":"3dd70a7a4dd43730f11e7f8021bf53f2a1bb3810b98df15a80511d492aacf06f"} Oct 31 00:37:28.291706 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:37:28.291454 4927 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3dd70a7a4dd43730f11e7f8021bf53f2a1bb3810b98df15a80511d492aacf06f" Oct 31 00:37:33.814869 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:37:33.814836 4927 scope.go:122] "RemoveContainer" containerID="f04a85df51ad55a3594b00a5fd8efe5c2a78aa3c1df675ee335f7147a10d3748" Oct 31 00:37:33.824264 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:37:33.824242 4927 scope.go:122] "RemoveContainer" containerID="27b24f6894c7d6e9fc5ee17f849524d8952a27c86b292bb053b06cc7a0faecf9" Oct 31 00:37:33.832435 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:37:33.832411 4927 scope.go:122] "RemoveContainer" containerID="0ba35cb0714250ebc84d5c6163ce3349ce8d3baf1e41ffeb76cc5d797a164e6c" Oct 31 00:37:33.840398 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:37:33.840376 4927 scope.go:122] "RemoveContainer" containerID="3ae428042f01c0e007043ffcf8edbd2765401d8237bbe76f40f1d46e9c5848fc" Oct 31 00:37:37.390977 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:37:37.390890 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:37:47.641640 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:37:47.641604 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-bgmzf"] Oct 31 00:37:47.642204 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:37:47.642097 4927 cpu_manager.go:414] "containerMap: removing container" podUID="e2f5f51f-6279-4993-960e-46bdceb23e84" containerName="revoke-keys" Oct 31 00:37:47.642204 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:37:47.642110 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2f5f51f-6279-4993-960e-46bdceb23e84" containerName="revoke-keys" Oct 31 00:37:47.642204 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:37:47.642182 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2f5f51f-6279-4993-960e-46bdceb23e84" containerName="revoke-keys" Oct 31 00:37:47.646946 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:37:47.646919 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-bgmzf" Oct 31 00:37:47.701085 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:37:47.701050 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-bgmzf"] Oct 31 00:37:47.747148 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:37:47.747113 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hd6nf\" (UniqueName: \"kubernetes.io/projected/d4544ba9-5754-41ef-bb34-e17fc37d3502-kube-api-access-hd6nf\") pod \"maas-api-revoke-sub-0b7e8edb-bgmzf\" (UID: \"d4544ba9-5754-41ef-bb34-e17fc37d3502\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-bgmzf" Oct 31 00:37:47.747148 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:37:47.747151 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4544ba9-5754-41ef-bb34-e17fc37d3502-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-bgmzf\" (UID: \"d4544ba9-5754-41ef-bb34-e17fc37d3502\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-bgmzf" Oct 31 00:37:47.747364 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:37:47.747288 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4544ba9-5754-41ef-bb34-e17fc37d3502-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-bgmzf\" (UID: \"d4544ba9-5754-41ef-bb34-e17fc37d3502\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-bgmzf" Oct 31 00:37:47.748213 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:37:47.748188 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:37:47.748315 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:37:47.748303 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4fdzs\"" Oct 31 00:37:47.847779 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:37:47.847744 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hd6nf\" (UniqueName: \"kubernetes.io/projected/d4544ba9-5754-41ef-bb34-e17fc37d3502-kube-api-access-hd6nf\") pod \"maas-api-revoke-sub-0b7e8edb-bgmzf\" (UID: \"d4544ba9-5754-41ef-bb34-e17fc37d3502\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-bgmzf" Oct 31 00:37:47.847779 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:37:47.847779 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4544ba9-5754-41ef-bb34-e17fc37d3502-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-bgmzf\" (UID: \"d4544ba9-5754-41ef-bb34-e17fc37d3502\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-bgmzf" Oct 31 00:37:47.847992 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:37:47.847828 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4544ba9-5754-41ef-bb34-e17fc37d3502-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-bgmzf\" (UID: \"d4544ba9-5754-41ef-bb34-e17fc37d3502\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-bgmzf" Oct 31 00:37:47.848475 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:37:47.848452 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4544ba9-5754-41ef-bb34-e17fc37d3502-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-bgmzf\" (UID: \"d4544ba9-5754-41ef-bb34-e17fc37d3502\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-bgmzf" Oct 31 00:37:47.850222 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:37:47.850195 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4544ba9-5754-41ef-bb34-e17fc37d3502-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-bgmzf\" (UID: \"d4544ba9-5754-41ef-bb34-e17fc37d3502\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-bgmzf" Oct 31 00:37:47.910787 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:37:47.910754 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hd6nf\" (UniqueName: \"kubernetes.io/projected/d4544ba9-5754-41ef-bb34-e17fc37d3502-kube-api-access-hd6nf\") pod \"maas-api-revoke-sub-0b7e8edb-bgmzf\" (UID: \"d4544ba9-5754-41ef-bb34-e17fc37d3502\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-bgmzf" Oct 31 00:37:47.964359 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:37:47.964319 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-bgmzf" Oct 31 00:37:48.340849 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:37:48.340816 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-bgmzf"] Oct 31 00:37:48.342398 ip-10-0-141-180 kubenswrapper[4927]: W1031 00:37:48.342365 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4544ba9_5754_41ef_bb34_e17fc37d3502.slice/crio-fafb4c1a9df051490e58ba1217799b5fc219efd2ce7395b13ec9beca8f4b08f2 WatchSource:0}: Error finding container fafb4c1a9df051490e58ba1217799b5fc219efd2ce7395b13ec9beca8f4b08f2: Status 404 returned error can't find the container with id fafb4c1a9df051490e58ba1217799b5fc219efd2ce7395b13ec9beca8f4b08f2 Oct 31 00:37:48.441728 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:37:48.441692 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-bgmzf" event={"ID":"d4544ba9-5754-41ef-bb34-e17fc37d3502","Type":"ContainerStarted","Data":"fafb4c1a9df051490e58ba1217799b5fc219efd2ce7395b13ec9beca8f4b08f2"} Oct 31 00:37:49.449720 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:37:49.449635 4927 generic.go:356] "Generic (PLEG): container finished" podID="d4544ba9-5754-41ef-bb34-e17fc37d3502" containerID="540a283d32b798e41c28614bc6c1aee1823d3d81a0ff8db9179ba2bdf7e6a6e1" exitCode=0 Oct 31 00:37:49.450086 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:37:49.449712 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-bgmzf" event={"ID":"d4544ba9-5754-41ef-bb34-e17fc37d3502","Type":"ContainerDied","Data":"540a283d32b798e41c28614bc6c1aee1823d3d81a0ff8db9179ba2bdf7e6a6e1"} Oct 31 00:37:49.677429 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:37:49.677391 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sqv69" containerName="extractor" Oct 31 00:37:50.593056 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:37:50.593031 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-bgmzf" Oct 31 00:37:50.670021 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:37:50.669989 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d4544ba9-5754-41ef-bb34-e17fc37d3502-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4544ba9-5754-41ef-bb34-e17fc37d3502-maas-api-openshift-ca\") pod \"d4544ba9-5754-41ef-bb34-e17fc37d3502\" (UID: \"d4544ba9-5754-41ef-bb34-e17fc37d3502\") " Oct 31 00:37:50.670202 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:37:50.670054 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d4544ba9-5754-41ef-bb34-e17fc37d3502-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4544ba9-5754-41ef-bb34-e17fc37d3502-maas-api-xks-ca\") pod \"d4544ba9-5754-41ef-bb34-e17fc37d3502\" (UID: \"d4544ba9-5754-41ef-bb34-e17fc37d3502\") " Oct 31 00:37:50.670202 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:37:50.670108 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4544ba9-5754-41ef-bb34-e17fc37d3502-kube-api-access-hd6nf\" (UniqueName: \"kubernetes.io/projected/d4544ba9-5754-41ef-bb34-e17fc37d3502-kube-api-access-hd6nf\") pod \"d4544ba9-5754-41ef-bb34-e17fc37d3502\" (UID: \"d4544ba9-5754-41ef-bb34-e17fc37d3502\") " Oct 31 00:37:50.670434 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:37:50.670407 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4544ba9-5754-41ef-bb34-e17fc37d3502-maas-api-openshift-ca" pod "d4544ba9-5754-41ef-bb34-e17fc37d3502" (UID: "d4544ba9-5754-41ef-bb34-e17fc37d3502"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:37:50.672142 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:37:50.672115 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4544ba9-5754-41ef-bb34-e17fc37d3502-kube-api-access-hd6nf" pod "d4544ba9-5754-41ef-bb34-e17fc37d3502" (UID: "d4544ba9-5754-41ef-bb34-e17fc37d3502"). InnerVolumeSpecName "kube-api-access-hd6nf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:37:50.672238 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:37:50.672143 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d4544ba9-5754-41ef-bb34-e17fc37d3502-maas-api-xks-ca" pod "d4544ba9-5754-41ef-bb34-e17fc37d3502" (UID: "d4544ba9-5754-41ef-bb34-e17fc37d3502"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:37:50.771311 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:37:50.771227 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hd6nf\" (UniqueName: \"kubernetes.io/projected/d4544ba9-5754-41ef-bb34-e17fc37d3502-kube-api-access-hd6nf\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:37:50.771311 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:37:50.771255 4927 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4544ba9-5754-41ef-bb34-e17fc37d3502-maas-api-openshift-ca\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:37:50.771311 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:37:50.771266 4927 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4544ba9-5754-41ef-bb34-e17fc37d3502-maas-api-xks-ca\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:37:51.462496 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:37:51.462465 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-bgmzf" Oct 31 00:37:51.462684 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:37:51.462503 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-bgmzf" event={"ID":"d4544ba9-5754-41ef-bb34-e17fc37d3502","Type":"ContainerDied","Data":"fafb4c1a9df051490e58ba1217799b5fc219efd2ce7395b13ec9beca8f4b08f2"} Oct 31 00:37:51.462684 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:37:51.462552 4927 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fafb4c1a9df051490e58ba1217799b5fc219efd2ce7395b13ec9beca8f4b08f2" Oct 31 00:37:58.977720 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:37:58.977682 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vnkm2"] Oct 31 00:37:58.985128 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:37:58.985080 4927 cpu_manager.go:414] "containerMap: removing container" podUID="d4544ba9-5754-41ef-bb34-e17fc37d3502" containerName="revoke-keys" Oct 31 00:37:58.985397 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:37:58.985373 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4544ba9-5754-41ef-bb34-e17fc37d3502" containerName="revoke-keys" Oct 31 00:37:58.985854 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:37:58.985833 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4544ba9-5754-41ef-bb34-e17fc37d3502" containerName="revoke-keys" Oct 31 00:37:58.990494 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:37:58.990470 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vnkm2" Oct 31 00:37:59.038118 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:37:59.038077 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vnkm2"] Oct 31 00:37:59.038732 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:37:59.038702 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0da8834d-da8f-4792-97c8-f3a15d66a9c9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-vnkm2\" (UID: \"0da8834d-da8f-4792-97c8-f3a15d66a9c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vnkm2" Oct 31 00:37:59.038887 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:37:59.038866 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jnfqk\" (UniqueName: \"kubernetes.io/projected/0da8834d-da8f-4792-97c8-f3a15d66a9c9-kube-api-access-jnfqk\") pod \"maas-api-revoke-sub-3d4156f1-vnkm2\" (UID: \"0da8834d-da8f-4792-97c8-f3a15d66a9c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vnkm2" Oct 31 00:37:59.038963 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:37:59.038936 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0da8834d-da8f-4792-97c8-f3a15d66a9c9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-vnkm2\" (UID: \"0da8834d-da8f-4792-97c8-f3a15d66a9c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vnkm2" Oct 31 00:37:59.084186 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:37:59.084155 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:37:59.084366 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:37:59.084302 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4fdzs\"" Oct 31 00:37:59.139456 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:37:59.139419 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0da8834d-da8f-4792-97c8-f3a15d66a9c9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-vnkm2\" (UID: \"0da8834d-da8f-4792-97c8-f3a15d66a9c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vnkm2" Oct 31 00:37:59.139607 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:37:59.139474 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0da8834d-da8f-4792-97c8-f3a15d66a9c9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-vnkm2\" (UID: \"0da8834d-da8f-4792-97c8-f3a15d66a9c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vnkm2" Oct 31 00:37:59.139731 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:37:59.139692 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jnfqk\" (UniqueName: \"kubernetes.io/projected/0da8834d-da8f-4792-97c8-f3a15d66a9c9-kube-api-access-jnfqk\") pod \"maas-api-revoke-sub-3d4156f1-vnkm2\" (UID: \"0da8834d-da8f-4792-97c8-f3a15d66a9c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vnkm2" Oct 31 00:37:59.140117 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:37:59.140098 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0da8834d-da8f-4792-97c8-f3a15d66a9c9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-vnkm2\" (UID: \"0da8834d-da8f-4792-97c8-f3a15d66a9c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vnkm2" Oct 31 00:37:59.141854 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:37:59.141836 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0da8834d-da8f-4792-97c8-f3a15d66a9c9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-vnkm2\" (UID: \"0da8834d-da8f-4792-97c8-f3a15d66a9c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vnkm2" Oct 31 00:37:59.202227 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:37:59.202194 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jnfqk\" (UniqueName: \"kubernetes.io/projected/0da8834d-da8f-4792-97c8-f3a15d66a9c9-kube-api-access-jnfqk\") pod \"maas-api-revoke-sub-3d4156f1-vnkm2\" (UID: \"0da8834d-da8f-4792-97c8-f3a15d66a9c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vnkm2" Oct 31 00:37:59.303317 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:37:59.303238 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vnkm2" Oct 31 00:37:59.694068 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:37:59.694033 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vnkm2"] Oct 31 00:37:59.694520 ip-10-0-141-180 kubenswrapper[4927]: W1031 00:37:59.694484 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0da8834d_da8f_4792_97c8_f3a15d66a9c9.slice/crio-c0531e7fb9597758f12ccda832693d8ce57cf492003a1177478f975683dbf237 WatchSource:0}: Error finding container c0531e7fb9597758f12ccda832693d8ce57cf492003a1177478f975683dbf237: Status 404 returned error can't find the container with id c0531e7fb9597758f12ccda832693d8ce57cf492003a1177478f975683dbf237 Oct 31 00:38:00.522325 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:38:00.522290 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vnkm2" event={"ID":"0da8834d-da8f-4792-97c8-f3a15d66a9c9","Type":"ContainerStarted","Data":"347cbe9b221acbe24700340924cf6ecdc1655bd9d0f6a6318369144b37400a9c"} Oct 31 00:38:00.522720 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:38:00.522369 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vnkm2" event={"ID":"0da8834d-da8f-4792-97c8-f3a15d66a9c9","Type":"ContainerStarted","Data":"c0531e7fb9597758f12ccda832693d8ce57cf492003a1177478f975683dbf237"} Oct 31 00:38:00.641274 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:38:00.641224 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vnkm2" podStartSLOduration=2.64121171 podStartE2EDuration="2.64121171s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:37:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:38:00.640632792 +0000 UTC m=+3393.676963635" watchObservedRunningTime="2026-10-31 00:38:00.64121171 +0000 UTC m=+3393.677542532" Oct 31 00:38:01.551357 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:38:01.551274 4927 generic.go:356] "Generic (PLEG): container finished" podID="0da8834d-da8f-4792-97c8-f3a15d66a9c9" containerID="347cbe9b221acbe24700340924cf6ecdc1655bd9d0f6a6318369144b37400a9c" exitCode=0 Oct 31 00:38:01.551907 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:38:01.551591 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vnkm2" event={"ID":"0da8834d-da8f-4792-97c8-f3a15d66a9c9","Type":"ContainerDied","Data":"347cbe9b221acbe24700340924cf6ecdc1655bd9d0f6a6318369144b37400a9c"} Oct 31 00:38:02.732773 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:38:02.732744 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vnkm2" Oct 31 00:38:02.798062 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:38:02.798030 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0da8834d-da8f-4792-97c8-f3a15d66a9c9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0da8834d-da8f-4792-97c8-f3a15d66a9c9-maas-api-openshift-ca\") pod \"0da8834d-da8f-4792-97c8-f3a15d66a9c9\" (UID: \"0da8834d-da8f-4792-97c8-f3a15d66a9c9\") " Oct 31 00:38:02.798254 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:38:02.798099 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0da8834d-da8f-4792-97c8-f3a15d66a9c9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0da8834d-da8f-4792-97c8-f3a15d66a9c9-maas-api-xks-ca\") pod \"0da8834d-da8f-4792-97c8-f3a15d66a9c9\" (UID: \"0da8834d-da8f-4792-97c8-f3a15d66a9c9\") " Oct 31 00:38:02.798254 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:38:02.798153 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0da8834d-da8f-4792-97c8-f3a15d66a9c9-kube-api-access-jnfqk\" (UniqueName: \"kubernetes.io/projected/0da8834d-da8f-4792-97c8-f3a15d66a9c9-kube-api-access-jnfqk\") pod \"0da8834d-da8f-4792-97c8-f3a15d66a9c9\" (UID: \"0da8834d-da8f-4792-97c8-f3a15d66a9c9\") " Oct 31 00:38:02.798524 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:38:02.798489 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0da8834d-da8f-4792-97c8-f3a15d66a9c9-maas-api-openshift-ca" pod "0da8834d-da8f-4792-97c8-f3a15d66a9c9" (UID: "0da8834d-da8f-4792-97c8-f3a15d66a9c9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:38:02.800244 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:38:02.800222 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0da8834d-da8f-4792-97c8-f3a15d66a9c9-maas-api-xks-ca" pod "0da8834d-da8f-4792-97c8-f3a15d66a9c9" (UID: "0da8834d-da8f-4792-97c8-f3a15d66a9c9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:38:02.800346 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:38:02.800312 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0da8834d-da8f-4792-97c8-f3a15d66a9c9-kube-api-access-jnfqk" pod "0da8834d-da8f-4792-97c8-f3a15d66a9c9" (UID: "0da8834d-da8f-4792-97c8-f3a15d66a9c9"). InnerVolumeSpecName "kube-api-access-jnfqk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:38:02.899245 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:38:02.899154 4927 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0da8834d-da8f-4792-97c8-f3a15d66a9c9-maas-api-xks-ca\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:38:02.899245 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:38:02.899189 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jnfqk\" (UniqueName: \"kubernetes.io/projected/0da8834d-da8f-4792-97c8-f3a15d66a9c9-kube-api-access-jnfqk\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:38:02.899245 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:38:02.899203 4927 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0da8834d-da8f-4792-97c8-f3a15d66a9c9-maas-api-openshift-ca\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:38:03.574291 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:38:03.574256 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vnkm2" Oct 31 00:38:03.574533 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:38:03.574301 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vnkm2" event={"ID":"0da8834d-da8f-4792-97c8-f3a15d66a9c9","Type":"ContainerDied","Data":"c0531e7fb9597758f12ccda832693d8ce57cf492003a1177478f975683dbf237"} Oct 31 00:38:03.574533 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:38:03.574370 4927 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c0531e7fb9597758f12ccda832693d8ce57cf492003a1177478f975683dbf237" Oct 31 00:38:10.083160 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:38:10.083123 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-gzhjj"] Oct 31 00:38:10.131851 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:38:10.131809 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-gzhjj"] Oct 31 00:38:11.085314 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:38:11.085277 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="792ee66f-793e-40cb-90a3-f2d49af46273" path="/var/lib/kubelet/pods/792ee66f-793e-40cb-90a3-f2d49af46273/volumes" Oct 31 00:38:19.680634 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:38:19.680589 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sqv69" containerName="extractor" Oct 31 00:38:32.674662 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:38:32.674620 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vnkm2"] Oct 31 00:38:32.724956 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:38:32.724926 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vnkm2"] Oct 31 00:38:32.801254 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:38:32.801220 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xzfss"] Oct 31 00:38:32.801736 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:38:32.801721 4927 cpu_manager.go:414] "containerMap: removing container" podUID="0da8834d-da8f-4792-97c8-f3a15d66a9c9" containerName="revoke-keys" Oct 31 00:38:32.801794 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:38:32.801738 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0da8834d-da8f-4792-97c8-f3a15d66a9c9" containerName="revoke-keys" Oct 31 00:38:32.801832 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:38:32.801827 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="0da8834d-da8f-4792-97c8-f3a15d66a9c9" containerName="revoke-keys" Oct 31 00:38:32.805262 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:38:32.805245 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xzfss" Oct 31 00:38:32.836180 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:38:32.836147 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1145e8c-f4c0-4d92-8f24-edaf51c86cc4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-xzfss\" (UID: \"c1145e8c-f4c0-4d92-8f24-edaf51c86cc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xzfss" Oct 31 00:38:32.836313 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:38:32.836211 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1145e8c-f4c0-4d92-8f24-edaf51c86cc4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-xzfss\" (UID: \"c1145e8c-f4c0-4d92-8f24-edaf51c86cc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xzfss" Oct 31 00:38:32.836382 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:38:32.836339 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmbfs\" (UniqueName: \"kubernetes.io/projected/c1145e8c-f4c0-4d92-8f24-edaf51c86cc4-kube-api-access-qmbfs\") pod \"maas-api-revoke-sub-3d4156f1-xzfss\" (UID: \"c1145e8c-f4c0-4d92-8f24-edaf51c86cc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xzfss" Oct 31 00:38:32.858703 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:38:32.858673 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:38:32.858848 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:38:32.858792 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4fdzs\"" Oct 31 00:38:32.862151 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:38:32.862117 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xzfss"] Oct 31 00:38:32.937191 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:38:32.937115 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1145e8c-f4c0-4d92-8f24-edaf51c86cc4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-xzfss\" (UID: \"c1145e8c-f4c0-4d92-8f24-edaf51c86cc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xzfss" Oct 31 00:38:32.937191 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:38:32.937162 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1145e8c-f4c0-4d92-8f24-edaf51c86cc4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-xzfss\" (UID: \"c1145e8c-f4c0-4d92-8f24-edaf51c86cc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xzfss" Oct 31 00:38:32.937415 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:38:32.937270 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmbfs\" (UniqueName: \"kubernetes.io/projected/c1145e8c-f4c0-4d92-8f24-edaf51c86cc4-kube-api-access-qmbfs\") pod \"maas-api-revoke-sub-3d4156f1-xzfss\" (UID: \"c1145e8c-f4c0-4d92-8f24-edaf51c86cc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xzfss" Oct 31 00:38:32.937788 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:38:32.937770 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1145e8c-f4c0-4d92-8f24-edaf51c86cc4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-xzfss\" (UID: \"c1145e8c-f4c0-4d92-8f24-edaf51c86cc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xzfss" Oct 31 00:38:32.939531 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:38:32.939503 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1145e8c-f4c0-4d92-8f24-edaf51c86cc4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-xzfss\" (UID: \"c1145e8c-f4c0-4d92-8f24-edaf51c86cc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xzfss" Oct 31 00:38:32.999674 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:38:32.999646 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmbfs\" (UniqueName: \"kubernetes.io/projected/c1145e8c-f4c0-4d92-8f24-edaf51c86cc4-kube-api-access-qmbfs\") pod \"maas-api-revoke-sub-3d4156f1-xzfss\" (UID: \"c1145e8c-f4c0-4d92-8f24-edaf51c86cc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xzfss" Oct 31 00:38:33.086728 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:38:33.086689 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0da8834d-da8f-4792-97c8-f3a15d66a9c9" path="/var/lib/kubelet/pods/0da8834d-da8f-4792-97c8-f3a15d66a9c9/volumes" Oct 31 00:38:33.117972 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:38:33.117938 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xzfss" Oct 31 00:38:33.492897 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:38:33.492865 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xzfss"] Oct 31 00:38:33.494596 ip-10-0-141-180 kubenswrapper[4927]: W1031 00:38:33.494567 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1145e8c_f4c0_4d92_8f24_edaf51c86cc4.slice/crio-b03efc9816bb69ddf97acc1bc6cbbf064bad02b7e5e020aad076c6adead9d2fc WatchSource:0}: Error finding container b03efc9816bb69ddf97acc1bc6cbbf064bad02b7e5e020aad076c6adead9d2fc: Status 404 returned error can't find the container with id b03efc9816bb69ddf97acc1bc6cbbf064bad02b7e5e020aad076c6adead9d2fc Oct 31 00:38:33.768357 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:38:33.768270 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xzfss" event={"ID":"c1145e8c-f4c0-4d92-8f24-edaf51c86cc4","Type":"ContainerStarted","Data":"b03efc9816bb69ddf97acc1bc6cbbf064bad02b7e5e020aad076c6adead9d2fc"} Oct 31 00:38:33.873963 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:38:33.873938 4927 scope.go:122] "RemoveContainer" containerID="af6517da8f87f0cd9fdd668c6516e20bf89109e715d7dc97aebc3811149c7c72" Oct 31 00:38:33.882649 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:38:33.882628 4927 scope.go:122] "RemoveContainer" containerID="d7c747cdf240a8304fa94db7b1f3b468917aa96a8436afc6cbf1d3726ebb5441" Oct 31 00:38:34.776077 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:38:34.776041 4927 generic.go:356] "Generic (PLEG): container finished" podID="c1145e8c-f4c0-4d92-8f24-edaf51c86cc4" containerID="19adbe013cb8aae63119b1abb54309837b022aa34414fa9414977044dd8ef5cd" exitCode=0 Oct 31 00:38:34.776495 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:38:34.776129 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xzfss" event={"ID":"c1145e8c-f4c0-4d92-8f24-edaf51c86cc4","Type":"ContainerDied","Data":"19adbe013cb8aae63119b1abb54309837b022aa34414fa9414977044dd8ef5cd"} Oct 31 00:38:35.924892 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:38:35.924867 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xzfss" Oct 31 00:38:35.966249 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:38:35.966225 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c1145e8c-f4c0-4d92-8f24-edaf51c86cc4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1145e8c-f4c0-4d92-8f24-edaf51c86cc4-maas-api-openshift-ca\") pod \"c1145e8c-f4c0-4d92-8f24-edaf51c86cc4\" (UID: \"c1145e8c-f4c0-4d92-8f24-edaf51c86cc4\") " Oct 31 00:38:35.966383 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:38:35.966320 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1145e8c-f4c0-4d92-8f24-edaf51c86cc4-kube-api-access-qmbfs\" (UniqueName: \"kubernetes.io/projected/c1145e8c-f4c0-4d92-8f24-edaf51c86cc4-kube-api-access-qmbfs\") pod \"c1145e8c-f4c0-4d92-8f24-edaf51c86cc4\" (UID: \"c1145e8c-f4c0-4d92-8f24-edaf51c86cc4\") " Oct 31 00:38:35.966383 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:38:35.966359 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c1145e8c-f4c0-4d92-8f24-edaf51c86cc4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1145e8c-f4c0-4d92-8f24-edaf51c86cc4-maas-api-xks-ca\") pod \"c1145e8c-f4c0-4d92-8f24-edaf51c86cc4\" (UID: \"c1145e8c-f4c0-4d92-8f24-edaf51c86cc4\") " Oct 31 00:38:35.966564 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:38:35.966544 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1145e8c-f4c0-4d92-8f24-edaf51c86cc4-maas-api-openshift-ca" pod "c1145e8c-f4c0-4d92-8f24-edaf51c86cc4" (UID: "c1145e8c-f4c0-4d92-8f24-edaf51c86cc4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:38:35.968284 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:38:35.968254 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1145e8c-f4c0-4d92-8f24-edaf51c86cc4-maas-api-xks-ca" pod "c1145e8c-f4c0-4d92-8f24-edaf51c86cc4" (UID: "c1145e8c-f4c0-4d92-8f24-edaf51c86cc4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:38:35.968426 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:38:35.968406 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1145e8c-f4c0-4d92-8f24-edaf51c86cc4-kube-api-access-qmbfs" pod "c1145e8c-f4c0-4d92-8f24-edaf51c86cc4" (UID: "c1145e8c-f4c0-4d92-8f24-edaf51c86cc4"). InnerVolumeSpecName "kube-api-access-qmbfs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:38:36.067436 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:38:36.067360 4927 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1145e8c-f4c0-4d92-8f24-edaf51c86cc4-maas-api-openshift-ca\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:38:36.067436 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:38:36.067392 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qmbfs\" (UniqueName: \"kubernetes.io/projected/c1145e8c-f4c0-4d92-8f24-edaf51c86cc4-kube-api-access-qmbfs\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:38:36.067436 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:38:36.067404 4927 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1145e8c-f4c0-4d92-8f24-edaf51c86cc4-maas-api-xks-ca\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:38:36.788284 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:38:36.788251 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xzfss" Oct 31 00:38:36.788482 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:38:36.788282 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xzfss" event={"ID":"c1145e8c-f4c0-4d92-8f24-edaf51c86cc4","Type":"ContainerDied","Data":"b03efc9816bb69ddf97acc1bc6cbbf064bad02b7e5e020aad076c6adead9d2fc"} Oct 31 00:38:36.788482 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:38:36.788327 4927 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b03efc9816bb69ddf97acc1bc6cbbf064bad02b7e5e020aad076c6adead9d2fc" Oct 31 00:38:37.774435 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:38:37.774368 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:38:37.995106 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:38:37.995071 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:38:49.678140 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:38:49.678100 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sqv69" containerName="extractor" Oct 31 00:38:51.401031 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:38:51.400994 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:39:19.677495 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:39:19.677456 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sqv69" containerName="extractor" Oct 31 00:39:20.053109 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:39:20.053073 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-dgbqb"] Oct 31 00:39:20.103945 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:39:20.103905 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-dgbqb"] Oct 31 00:39:21.085808 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:39:21.085769 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a1fe65e4-7f84-42e7-9721-401adb8493b2" path="/var/lib/kubelet/pods/a1fe65e4-7f84-42e7-9721-401adb8493b2/volumes" Oct 31 00:39:23.183185 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:39:23.183153 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-qcrvb"] Oct 31 00:39:23.183611 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:39:23.183596 4927 cpu_manager.go:414] "containerMap: removing container" podUID="c1145e8c-f4c0-4d92-8f24-edaf51c86cc4" containerName="revoke-keys" Oct 31 00:39:23.183658 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:39:23.183611 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1145e8c-f4c0-4d92-8f24-edaf51c86cc4" containerName="revoke-keys" Oct 31 00:39:23.183696 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:39:23.183677 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1145e8c-f4c0-4d92-8f24-edaf51c86cc4" containerName="revoke-keys" Oct 31 00:39:23.190116 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:39:23.190077 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-qcrvb" Oct 31 00:39:23.233483 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:39:23.233449 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00356130-32fb-4b92-9952-25d0466e4415-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-qcrvb\" (UID: \"00356130-32fb-4b92-9952-25d0466e4415\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-qcrvb" Oct 31 00:39:23.233616 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:39:23.233493 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dcrrw\" (UniqueName: \"kubernetes.io/projected/00356130-32fb-4b92-9952-25d0466e4415-kube-api-access-dcrrw\") pod \"maas-api-revoke-sub-e2fc9c5b-qcrvb\" (UID: \"00356130-32fb-4b92-9952-25d0466e4415\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-qcrvb" Oct 31 00:39:23.233672 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:39:23.233649 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00356130-32fb-4b92-9952-25d0466e4415-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-qcrvb\" (UID: \"00356130-32fb-4b92-9952-25d0466e4415\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-qcrvb" Oct 31 00:39:23.241962 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:39:23.241931 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-qcrvb"] Oct 31 00:39:23.289738 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:39:23.289708 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:39:23.289893 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:39:23.289785 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4fdzs\"" Oct 31 00:39:23.334747 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:39:23.334720 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00356130-32fb-4b92-9952-25d0466e4415-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-qcrvb\" (UID: \"00356130-32fb-4b92-9952-25d0466e4415\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-qcrvb" Oct 31 00:39:23.334891 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:39:23.334758 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dcrrw\" (UniqueName: \"kubernetes.io/projected/00356130-32fb-4b92-9952-25d0466e4415-kube-api-access-dcrrw\") pod \"maas-api-revoke-sub-e2fc9c5b-qcrvb\" (UID: \"00356130-32fb-4b92-9952-25d0466e4415\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-qcrvb" Oct 31 00:39:23.334891 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:39:23.334819 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00356130-32fb-4b92-9952-25d0466e4415-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-qcrvb\" (UID: \"00356130-32fb-4b92-9952-25d0466e4415\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-qcrvb" Oct 31 00:39:23.335520 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:39:23.335490 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00356130-32fb-4b92-9952-25d0466e4415-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-qcrvb\" (UID: \"00356130-32fb-4b92-9952-25d0466e4415\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-qcrvb" Oct 31 00:39:23.337356 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:39:23.337303 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00356130-32fb-4b92-9952-25d0466e4415-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-qcrvb\" (UID: \"00356130-32fb-4b92-9952-25d0466e4415\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-qcrvb" Oct 31 00:39:23.399397 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:39:23.399369 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dcrrw\" (UniqueName: \"kubernetes.io/projected/00356130-32fb-4b92-9952-25d0466e4415-kube-api-access-dcrrw\") pod \"maas-api-revoke-sub-e2fc9c5b-qcrvb\" (UID: \"00356130-32fb-4b92-9952-25d0466e4415\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-qcrvb" Oct 31 00:39:23.505526 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:39:23.505454 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-qcrvb" Oct 31 00:39:23.882399 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:39:23.882373 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-qcrvb"] Oct 31 00:39:23.884127 ip-10-0-141-180 kubenswrapper[4927]: W1031 00:39:23.884090 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00356130_32fb_4b92_9952_25d0466e4415.slice/crio-0327243c1d8bb1ac9c3248781dc2d398c5dfb0281a171a2524c87a4d21240c7e WatchSource:0}: Error finding container 0327243c1d8bb1ac9c3248781dc2d398c5dfb0281a171a2524c87a4d21240c7e: Status 404 returned error can't find the container with id 0327243c1d8bb1ac9c3248781dc2d398c5dfb0281a171a2524c87a4d21240c7e Oct 31 00:39:24.107759 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:39:24.107722 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-qcrvb" event={"ID":"00356130-32fb-4b92-9952-25d0466e4415","Type":"ContainerStarted","Data":"0327243c1d8bb1ac9c3248781dc2d398c5dfb0281a171a2524c87a4d21240c7e"} Oct 31 00:39:25.114452 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:39:25.114372 4927 generic.go:356] "Generic (PLEG): container finished" podID="00356130-32fb-4b92-9952-25d0466e4415" containerID="00210cb868cf533febb7ad5f3253e45d9a421d386877e53b5569fd842434678b" exitCode=0 Oct 31 00:39:25.114833 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:39:25.114469 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-qcrvb" event={"ID":"00356130-32fb-4b92-9952-25d0466e4415","Type":"ContainerDied","Data":"00210cb868cf533febb7ad5f3253e45d9a421d386877e53b5569fd842434678b"} Oct 31 00:39:26.258653 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:39:26.258629 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-qcrvb" Oct 31 00:39:26.361584 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:39:26.361557 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/00356130-32fb-4b92-9952-25d0466e4415-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00356130-32fb-4b92-9952-25d0466e4415-maas-api-openshift-ca\") pod \"00356130-32fb-4b92-9952-25d0466e4415\" (UID: \"00356130-32fb-4b92-9952-25d0466e4415\") " Oct 31 00:39:26.361716 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:39:26.361679 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00356130-32fb-4b92-9952-25d0466e4415-kube-api-access-dcrrw\" (UniqueName: \"kubernetes.io/projected/00356130-32fb-4b92-9952-25d0466e4415-kube-api-access-dcrrw\") pod \"00356130-32fb-4b92-9952-25d0466e4415\" (UID: \"00356130-32fb-4b92-9952-25d0466e4415\") " Oct 31 00:39:26.361763 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:39:26.361737 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/00356130-32fb-4b92-9952-25d0466e4415-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00356130-32fb-4b92-9952-25d0466e4415-maas-api-xks-ca\") pod \"00356130-32fb-4b92-9952-25d0466e4415\" (UID: \"00356130-32fb-4b92-9952-25d0466e4415\") " Oct 31 00:39:26.361975 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:39:26.361945 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00356130-32fb-4b92-9952-25d0466e4415-maas-api-openshift-ca" pod "00356130-32fb-4b92-9952-25d0466e4415" (UID: "00356130-32fb-4b92-9952-25d0466e4415"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:39:26.362042 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:39:26.362020 4927 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00356130-32fb-4b92-9952-25d0466e4415-maas-api-openshift-ca\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:39:26.363754 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:39:26.363731 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/00356130-32fb-4b92-9952-25d0466e4415-maas-api-xks-ca" pod "00356130-32fb-4b92-9952-25d0466e4415" (UID: "00356130-32fb-4b92-9952-25d0466e4415"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:39:26.363890 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:39:26.363873 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00356130-32fb-4b92-9952-25d0466e4415-kube-api-access-dcrrw" pod "00356130-32fb-4b92-9952-25d0466e4415" (UID: "00356130-32fb-4b92-9952-25d0466e4415"). InnerVolumeSpecName "kube-api-access-dcrrw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:39:26.462635 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:39:26.462600 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dcrrw\" (UniqueName: \"kubernetes.io/projected/00356130-32fb-4b92-9952-25d0466e4415-kube-api-access-dcrrw\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:39:26.462635 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:39:26.462634 4927 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00356130-32fb-4b92-9952-25d0466e4415-maas-api-xks-ca\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:39:27.126675 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:39:27.126653 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-qcrvb" Oct 31 00:39:27.126811 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:39:27.126647 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-qcrvb" event={"ID":"00356130-32fb-4b92-9952-25d0466e4415","Type":"ContainerDied","Data":"0327243c1d8bb1ac9c3248781dc2d398c5dfb0281a171a2524c87a4d21240c7e"} Oct 31 00:39:27.126811 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:39:27.126777 4927 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0327243c1d8bb1ac9c3248781dc2d398c5dfb0281a171a2524c87a4d21240c7e" Oct 31 00:39:33.909386 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:39:33.909353 4927 scope.go:122] "RemoveContainer" containerID="e32104edc7fc9f328b1f03fc1b0660480e5d713fa2b27965fc983ed2f633be7f" Oct 31 00:39:33.955621 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:39:33.955583 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-dqf9g"] Oct 31 00:39:33.956420 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:39:33.956396 4927 cpu_manager.go:414] "containerMap: removing container" podUID="00356130-32fb-4b92-9952-25d0466e4415" containerName="revoke-keys" Oct 31 00:39:33.956479 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:39:33.956423 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00356130-32fb-4b92-9952-25d0466e4415" containerName="revoke-keys" Oct 31 00:39:33.956625 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:39:33.956612 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="00356130-32fb-4b92-9952-25d0466e4415" containerName="revoke-keys" Oct 31 00:39:33.961928 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:39:33.961903 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-dqf9g" Oct 31 00:39:34.013841 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:39:34.013807 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-dqf9g"] Oct 31 00:39:34.033889 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:39:34.033855 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3189e4b8-6b86-4499-abce-f27620456397-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-dqf9g\" (UID: \"3189e4b8-6b86-4499-abce-f27620456397\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-dqf9g" Oct 31 00:39:34.033889 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:39:34.033894 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3189e4b8-6b86-4499-abce-f27620456397-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-dqf9g\" (UID: \"3189e4b8-6b86-4499-abce-f27620456397\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-dqf9g" Oct 31 00:39:34.034147 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:39:34.033911 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29t6w\" (UniqueName: \"kubernetes.io/projected/3189e4b8-6b86-4499-abce-f27620456397-kube-api-access-29t6w\") pod \"maas-api-revoke-sub-5a29efb3-dqf9g\" (UID: \"3189e4b8-6b86-4499-abce-f27620456397\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-dqf9g" Oct 31 00:39:34.061079 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:39:34.061045 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:39:34.061079 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:39:34.061060 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4fdzs\"" Oct 31 00:39:34.134898 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:39:34.134858 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3189e4b8-6b86-4499-abce-f27620456397-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-dqf9g\" (UID: \"3189e4b8-6b86-4499-abce-f27620456397\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-dqf9g" Oct 31 00:39:34.134898 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:39:34.134902 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3189e4b8-6b86-4499-abce-f27620456397-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-dqf9g\" (UID: \"3189e4b8-6b86-4499-abce-f27620456397\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-dqf9g" Oct 31 00:39:34.135180 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:39:34.134930 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29t6w\" (UniqueName: \"kubernetes.io/projected/3189e4b8-6b86-4499-abce-f27620456397-kube-api-access-29t6w\") pod \"maas-api-revoke-sub-5a29efb3-dqf9g\" (UID: \"3189e4b8-6b86-4499-abce-f27620456397\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-dqf9g" Oct 31 00:39:34.135592 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:39:34.135564 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3189e4b8-6b86-4499-abce-f27620456397-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-dqf9g\" (UID: \"3189e4b8-6b86-4499-abce-f27620456397\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-dqf9g" Oct 31 00:39:34.137373 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:39:34.137325 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3189e4b8-6b86-4499-abce-f27620456397-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-dqf9g\" (UID: \"3189e4b8-6b86-4499-abce-f27620456397\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-dqf9g" Oct 31 00:39:34.197829 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:39:34.197762 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-29t6w\" (UniqueName: \"kubernetes.io/projected/3189e4b8-6b86-4499-abce-f27620456397-kube-api-access-29t6w\") pod \"maas-api-revoke-sub-5a29efb3-dqf9g\" (UID: \"3189e4b8-6b86-4499-abce-f27620456397\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-dqf9g" Oct 31 00:39:34.275082 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:39:34.275039 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-dqf9g" Oct 31 00:39:34.659161 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:39:34.659129 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-dqf9g"] Oct 31 00:39:34.660746 ip-10-0-141-180 kubenswrapper[4927]: W1031 00:39:34.660720 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3189e4b8_6b86_4499_abce_f27620456397.slice/crio-8727ef68ebd14d11a73fb4db8dfc389f32ead792ad35bd67827e54611ef9f301 WatchSource:0}: Error finding container 8727ef68ebd14d11a73fb4db8dfc389f32ead792ad35bd67827e54611ef9f301: Status 404 returned error can't find the container with id 8727ef68ebd14d11a73fb4db8dfc389f32ead792ad35bd67827e54611ef9f301 Oct 31 00:39:35.178900 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:39:35.178817 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-dqf9g" event={"ID":"3189e4b8-6b86-4499-abce-f27620456397","Type":"ContainerStarted","Data":"e951c3312e1f98ea08084aeb72c061acc2f26c78b8c0d4ecd034edb5c33d7423"} Oct 31 00:39:35.178900 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:39:35.178871 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-dqf9g" event={"ID":"3189e4b8-6b86-4499-abce-f27620456397","Type":"ContainerStarted","Data":"8727ef68ebd14d11a73fb4db8dfc389f32ead792ad35bd67827e54611ef9f301"} Oct 31 00:39:35.300399 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:39:35.300326 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-dqf9g" podStartSLOduration=2.300312863 podStartE2EDuration="2.300312863s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:39:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:39:35.29715187 +0000 UTC m=+3488.333482692" watchObservedRunningTime="2026-10-31 00:39:35.300312863 +0000 UTC m=+3488.336643684" Oct 31 00:39:36.188431 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:39:36.188388 4927 generic.go:356] "Generic (PLEG): container finished" podID="3189e4b8-6b86-4499-abce-f27620456397" containerID="e951c3312e1f98ea08084aeb72c061acc2f26c78b8c0d4ecd034edb5c33d7423" exitCode=0 Oct 31 00:39:36.188934 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:39:36.188470 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-dqf9g" event={"ID":"3189e4b8-6b86-4499-abce-f27620456397","Type":"ContainerDied","Data":"e951c3312e1f98ea08084aeb72c061acc2f26c78b8c0d4ecd034edb5c33d7423"} Oct 31 00:39:37.647452 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:39:37.647108 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-dqf9g" Oct 31 00:39:37.809076 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:39:37.807906 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3189e4b8-6b86-4499-abce-f27620456397-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3189e4b8-6b86-4499-abce-f27620456397-maas-api-xks-ca\") pod \"3189e4b8-6b86-4499-abce-f27620456397\" (UID: \"3189e4b8-6b86-4499-abce-f27620456397\") " Oct 31 00:39:37.809076 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:39:37.807963 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3189e4b8-6b86-4499-abce-f27620456397-kube-api-access-29t6w\" (UniqueName: \"kubernetes.io/projected/3189e4b8-6b86-4499-abce-f27620456397-kube-api-access-29t6w\") pod \"3189e4b8-6b86-4499-abce-f27620456397\" (UID: \"3189e4b8-6b86-4499-abce-f27620456397\") " Oct 31 00:39:37.809076 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:39:37.808120 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3189e4b8-6b86-4499-abce-f27620456397-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3189e4b8-6b86-4499-abce-f27620456397-maas-api-openshift-ca\") pod \"3189e4b8-6b86-4499-abce-f27620456397\" (UID: \"3189e4b8-6b86-4499-abce-f27620456397\") " Oct 31 00:39:37.809997 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:39:37.809686 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3189e4b8-6b86-4499-abce-f27620456397-maas-api-openshift-ca" pod "3189e4b8-6b86-4499-abce-f27620456397" (UID: "3189e4b8-6b86-4499-abce-f27620456397"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:39:37.855649 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:39:37.855524 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3189e4b8-6b86-4499-abce-f27620456397-maas-api-xks-ca" pod "3189e4b8-6b86-4499-abce-f27620456397" (UID: "3189e4b8-6b86-4499-abce-f27620456397"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:39:37.862549 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:39:37.862489 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3189e4b8-6b86-4499-abce-f27620456397-kube-api-access-29t6w" pod "3189e4b8-6b86-4499-abce-f27620456397" (UID: "3189e4b8-6b86-4499-abce-f27620456397"). InnerVolumeSpecName "kube-api-access-29t6w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:39:37.916357 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:39:37.911968 4927 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3189e4b8-6b86-4499-abce-f27620456397-maas-api-xks-ca\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:39:37.916357 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:39:37.912013 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-29t6w\" (UniqueName: \"kubernetes.io/projected/3189e4b8-6b86-4499-abce-f27620456397-kube-api-access-29t6w\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:39:37.916357 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:39:37.912030 4927 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3189e4b8-6b86-4499-abce-f27620456397-maas-api-openshift-ca\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:39:38.162411 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:39:38.141499 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:39:38.204260 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:39:38.204234 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-dqf9g" Oct 31 00:39:38.204459 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:39:38.204234 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-dqf9g" event={"ID":"3189e4b8-6b86-4499-abce-f27620456397","Type":"ContainerDied","Data":"8727ef68ebd14d11a73fb4db8dfc389f32ead792ad35bd67827e54611ef9f301"} Oct 31 00:39:38.204459 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:39:38.204376 4927 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8727ef68ebd14d11a73fb4db8dfc389f32ead792ad35bd67827e54611ef9f301" Oct 31 00:39:38.547410 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:39:38.547158 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:39:49.678487 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:39:49.678441 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sqv69" containerName="extractor" Oct 31 00:39:50.600939 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:39:50.600901 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xzfss"] Oct 31 00:39:50.651680 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:39:50.651643 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xzfss"] Oct 31 00:39:50.727032 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:39:50.727001 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zk6cp"] Oct 31 00:39:50.727473 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:39:50.727458 4927 cpu_manager.go:414] "containerMap: removing container" podUID="3189e4b8-6b86-4499-abce-f27620456397" containerName="revoke-keys" Oct 31 00:39:50.727524 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:39:50.727473 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3189e4b8-6b86-4499-abce-f27620456397" containerName="revoke-keys" Oct 31 00:39:50.727562 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:39:50.727543 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="3189e4b8-6b86-4499-abce-f27620456397" containerName="revoke-keys" Oct 31 00:39:50.731050 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:39:50.731031 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zk6cp" Oct 31 00:39:50.762316 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:39:50.762279 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c96e602a-b8ab-48f2-a269-c4f6205d55a0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-zk6cp\" (UID: \"c96e602a-b8ab-48f2-a269-c4f6205d55a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zk6cp" Oct 31 00:39:50.762513 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:39:50.762398 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c96e602a-b8ab-48f2-a269-c4f6205d55a0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-zk6cp\" (UID: \"c96e602a-b8ab-48f2-a269-c4f6205d55a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zk6cp" Oct 31 00:39:50.762513 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:39:50.762458 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mtqrr\" (UniqueName: \"kubernetes.io/projected/c96e602a-b8ab-48f2-a269-c4f6205d55a0-kube-api-access-mtqrr\") pod \"maas-api-revoke-sub-3d4156f1-zk6cp\" (UID: \"c96e602a-b8ab-48f2-a269-c4f6205d55a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zk6cp" Oct 31 00:39:50.788064 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:39:50.788032 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4fdzs\"" Oct 31 00:39:50.788473 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:39:50.788452 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:39:50.791375 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:39:50.791347 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zk6cp"] Oct 31 00:39:50.863832 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:39:50.863761 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c96e602a-b8ab-48f2-a269-c4f6205d55a0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-zk6cp\" (UID: \"c96e602a-b8ab-48f2-a269-c4f6205d55a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zk6cp" Oct 31 00:39:50.863832 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:39:50.863802 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c96e602a-b8ab-48f2-a269-c4f6205d55a0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-zk6cp\" (UID: \"c96e602a-b8ab-48f2-a269-c4f6205d55a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zk6cp" Oct 31 00:39:50.864057 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:39:50.863930 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtqrr\" (UniqueName: \"kubernetes.io/projected/c96e602a-b8ab-48f2-a269-c4f6205d55a0-kube-api-access-mtqrr\") pod \"maas-api-revoke-sub-3d4156f1-zk6cp\" (UID: \"c96e602a-b8ab-48f2-a269-c4f6205d55a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zk6cp" Oct 31 00:39:50.864569 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:39:50.864547 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c96e602a-b8ab-48f2-a269-c4f6205d55a0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-zk6cp\" (UID: \"c96e602a-b8ab-48f2-a269-c4f6205d55a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zk6cp" Oct 31 00:39:50.866062 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:39:50.866043 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c96e602a-b8ab-48f2-a269-c4f6205d55a0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-zk6cp\" (UID: \"c96e602a-b8ab-48f2-a269-c4f6205d55a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zk6cp" Oct 31 00:39:50.926680 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:39:50.926640 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mtqrr\" (UniqueName: \"kubernetes.io/projected/c96e602a-b8ab-48f2-a269-c4f6205d55a0-kube-api-access-mtqrr\") pod \"maas-api-revoke-sub-3d4156f1-zk6cp\" (UID: \"c96e602a-b8ab-48f2-a269-c4f6205d55a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zk6cp" Oct 31 00:39:51.043836 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:39:51.043800 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zk6cp" Oct 31 00:39:51.053125 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:39:51.053080 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-mqxxc"] Oct 31 00:39:51.057605 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:39:51.057574 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-mqxxc"] Oct 31 00:39:51.085378 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:39:51.084932 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c2731fb-f9c4-4962-8637-2aa44d6acf15" path="/var/lib/kubelet/pods/0c2731fb-f9c4-4962-8637-2aa44d6acf15/volumes" Oct 31 00:39:51.085512 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:39:51.085389 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c1145e8c-f4c0-4d92-8f24-edaf51c86cc4" path="/var/lib/kubelet/pods/c1145e8c-f4c0-4d92-8f24-edaf51c86cc4/volumes" Oct 31 00:39:51.426841 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:39:51.426812 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zk6cp"] Oct 31 00:39:51.427483 ip-10-0-141-180 kubenswrapper[4927]: W1031 00:39:51.427454 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc96e602a_b8ab_48f2_a269_c4f6205d55a0.slice/crio-2b6618379934fd7f8c42a823be4dc99383cd0c47f5e7e9d27571d51a2c2fc083 WatchSource:0}: Error finding container 2b6618379934fd7f8c42a823be4dc99383cd0c47f5e7e9d27571d51a2c2fc083: Status 404 returned error can't find the container with id 2b6618379934fd7f8c42a823be4dc99383cd0c47f5e7e9d27571d51a2c2fc083 Oct 31 00:39:52.302754 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:39:52.302722 4927 generic.go:356] "Generic (PLEG): container finished" podID="c96e602a-b8ab-48f2-a269-c4f6205d55a0" containerID="83b4fd2ffc3e526cd930ec5e8baec58635a36049e2d06dc4618ab708a605e136" exitCode=0 Oct 31 00:39:52.303138 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:39:52.302806 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zk6cp" event={"ID":"c96e602a-b8ab-48f2-a269-c4f6205d55a0","Type":"ContainerDied","Data":"83b4fd2ffc3e526cd930ec5e8baec58635a36049e2d06dc4618ab708a605e136"} Oct 31 00:39:52.303138 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:39:52.302865 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zk6cp" event={"ID":"c96e602a-b8ab-48f2-a269-c4f6205d55a0","Type":"ContainerStarted","Data":"2b6618379934fd7f8c42a823be4dc99383cd0c47f5e7e9d27571d51a2c2fc083"} Oct 31 00:39:53.461022 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:39:53.460997 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zk6cp" Oct 31 00:39:53.491169 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:39:53.491142 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c96e602a-b8ab-48f2-a269-c4f6205d55a0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c96e602a-b8ab-48f2-a269-c4f6205d55a0-maas-api-xks-ca\") pod \"c96e602a-b8ab-48f2-a269-c4f6205d55a0\" (UID: \"c96e602a-b8ab-48f2-a269-c4f6205d55a0\") " Oct 31 00:39:53.491282 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:39:53.491201 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c96e602a-b8ab-48f2-a269-c4f6205d55a0-kube-api-access-mtqrr\" (UniqueName: \"kubernetes.io/projected/c96e602a-b8ab-48f2-a269-c4f6205d55a0-kube-api-access-mtqrr\") pod \"c96e602a-b8ab-48f2-a269-c4f6205d55a0\" (UID: \"c96e602a-b8ab-48f2-a269-c4f6205d55a0\") " Oct 31 00:39:53.491282 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:39:53.491235 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c96e602a-b8ab-48f2-a269-c4f6205d55a0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c96e602a-b8ab-48f2-a269-c4f6205d55a0-maas-api-openshift-ca\") pod \"c96e602a-b8ab-48f2-a269-c4f6205d55a0\" (UID: \"c96e602a-b8ab-48f2-a269-c4f6205d55a0\") " Oct 31 00:39:53.491740 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:39:53.491709 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c96e602a-b8ab-48f2-a269-c4f6205d55a0-maas-api-openshift-ca" pod "c96e602a-b8ab-48f2-a269-c4f6205d55a0" (UID: "c96e602a-b8ab-48f2-a269-c4f6205d55a0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:39:53.493227 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:39:53.493199 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c96e602a-b8ab-48f2-a269-c4f6205d55a0-maas-api-xks-ca" pod "c96e602a-b8ab-48f2-a269-c4f6205d55a0" (UID: "c96e602a-b8ab-48f2-a269-c4f6205d55a0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:39:53.493397 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:39:53.493273 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c96e602a-b8ab-48f2-a269-c4f6205d55a0-kube-api-access-mtqrr" pod "c96e602a-b8ab-48f2-a269-c4f6205d55a0" (UID: "c96e602a-b8ab-48f2-a269-c4f6205d55a0"). InnerVolumeSpecName "kube-api-access-mtqrr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:39:53.593003 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:39:53.592924 4927 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c96e602a-b8ab-48f2-a269-c4f6205d55a0-maas-api-xks-ca\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:39:53.593003 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:39:53.592953 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mtqrr\" (UniqueName: \"kubernetes.io/projected/c96e602a-b8ab-48f2-a269-c4f6205d55a0-kube-api-access-mtqrr\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:39:53.593003 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:39:53.592964 4927 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c96e602a-b8ab-48f2-a269-c4f6205d55a0-maas-api-openshift-ca\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:39:54.316058 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:39:54.316027 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zk6cp" Oct 31 00:39:54.316234 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:39:54.316064 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zk6cp" event={"ID":"c96e602a-b8ab-48f2-a269-c4f6205d55a0","Type":"ContainerDied","Data":"2b6618379934fd7f8c42a823be4dc99383cd0c47f5e7e9d27571d51a2c2fc083"} Oct 31 00:39:54.316234 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:39:54.316130 4927 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2b6618379934fd7f8c42a823be4dc99383cd0c47f5e7e9d27571d51a2c2fc083" Oct 31 00:40:04.053845 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:40:04.053803 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-k2s8w"] Oct 31 00:40:04.107097 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:40:04.107055 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-k2s8w"] Oct 31 00:40:05.085469 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:40:05.085423 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f157e2ce-caed-406b-a000-7bd759cb94df" path="/var/lib/kubelet/pods/f157e2ce-caed-406b-a000-7bd759cb94df/volumes" Oct 31 00:40:19.677953 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:40:19.677910 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sqv69" containerName="extractor" Oct 31 00:40:22.600696 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:40:22.600657 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:40:23.100462 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:40:23.100430 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-0e6572-openshift-default-5c475f9cbf97qjl"] Oct 31 00:40:23.100902 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:40:23.100888 4927 cpu_manager.go:414] "containerMap: removing container" podUID="c96e602a-b8ab-48f2-a269-c4f6205d55a0" containerName="revoke-keys" Oct 31 00:40:23.100959 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:40:23.100903 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c96e602a-b8ab-48f2-a269-c4f6205d55a0" containerName="revoke-keys" Oct 31 00:40:23.100998 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:40:23.100984 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="c96e602a-b8ab-48f2-a269-c4f6205d55a0" containerName="revoke-keys" Oct 31 00:40:23.104512 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:40:23.104492 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-0e6572-openshift-default-5c475f9cbf97qjl" Oct 31 00:40:23.117022 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:40:23.116993 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d768f7f8-eb55-4419-802a-9a4f2e6e57b8-istio-token\") pod \"e2e-worker-single-main-0e6572-openshift-default-5c475f9cbf97qjl\" (UID: \"d768f7f8-eb55-4419-802a-9a4f2e6e57b8\") " pod="openshift-ingress/e2e-worker-single-main-0e6572-openshift-default-5c475f9cbf97qjl" Oct 31 00:40:23.117164 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:40:23.117033 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5j2mj\" (UniqueName: \"kubernetes.io/projected/d768f7f8-eb55-4419-802a-9a4f2e6e57b8-kube-api-access-5j2mj\") pod \"e2e-worker-single-main-0e6572-openshift-default-5c475f9cbf97qjl\" (UID: \"d768f7f8-eb55-4419-802a-9a4f2e6e57b8\") " pod="openshift-ingress/e2e-worker-single-main-0e6572-openshift-default-5c475f9cbf97qjl" Oct 31 00:40:23.117164 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:40:23.117061 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d768f7f8-eb55-4419-802a-9a4f2e6e57b8-istiod-ca-cert\") pod \"e2e-worker-single-main-0e6572-openshift-default-5c475f9cbf97qjl\" (UID: \"d768f7f8-eb55-4419-802a-9a4f2e6e57b8\") " pod="openshift-ingress/e2e-worker-single-main-0e6572-openshift-default-5c475f9cbf97qjl" Oct 31 00:40:23.117283 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:40:23.117186 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d768f7f8-eb55-4419-802a-9a4f2e6e57b8-workload-socket\") pod \"e2e-worker-single-main-0e6572-openshift-default-5c475f9cbf97qjl\" (UID: \"d768f7f8-eb55-4419-802a-9a4f2e6e57b8\") " pod="openshift-ingress/e2e-worker-single-main-0e6572-openshift-default-5c475f9cbf97qjl" Oct 31 00:40:23.117283 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:40:23.117233 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d768f7f8-eb55-4419-802a-9a4f2e6e57b8-istio-podinfo\") pod \"e2e-worker-single-main-0e6572-openshift-default-5c475f9cbf97qjl\" (UID: \"d768f7f8-eb55-4419-802a-9a4f2e6e57b8\") " pod="openshift-ingress/e2e-worker-single-main-0e6572-openshift-default-5c475f9cbf97qjl" Oct 31 00:40:23.117398 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:40:23.117288 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d768f7f8-eb55-4419-802a-9a4f2e6e57b8-workload-certs\") pod \"e2e-worker-single-main-0e6572-openshift-default-5c475f9cbf97qjl\" (UID: \"d768f7f8-eb55-4419-802a-9a4f2e6e57b8\") " pod="openshift-ingress/e2e-worker-single-main-0e6572-openshift-default-5c475f9cbf97qjl" Oct 31 00:40:23.117398 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:40:23.117321 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d768f7f8-eb55-4419-802a-9a4f2e6e57b8-istio-envoy\") pod \"e2e-worker-single-main-0e6572-openshift-default-5c475f9cbf97qjl\" (UID: \"d768f7f8-eb55-4419-802a-9a4f2e6e57b8\") " pod="openshift-ingress/e2e-worker-single-main-0e6572-openshift-default-5c475f9cbf97qjl" Oct 31 00:40:23.117471 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:40:23.117391 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d768f7f8-eb55-4419-802a-9a4f2e6e57b8-istio-data\") pod \"e2e-worker-single-main-0e6572-openshift-default-5c475f9cbf97qjl\" (UID: \"d768f7f8-eb55-4419-802a-9a4f2e6e57b8\") " pod="openshift-ingress/e2e-worker-single-main-0e6572-openshift-default-5c475f9cbf97qjl" Oct 31 00:40:23.117471 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:40:23.117426 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d768f7f8-eb55-4419-802a-9a4f2e6e57b8-credential-socket\") pod \"e2e-worker-single-main-0e6572-openshift-default-5c475f9cbf97qjl\" (UID: \"d768f7f8-eb55-4419-802a-9a4f2e6e57b8\") " pod="openshift-ingress/e2e-worker-single-main-0e6572-openshift-default-5c475f9cbf97qjl" Oct 31 00:40:23.166488 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:40:23.166452 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-0e6572-openshift-default-dockercfg-zmpnd\"" Oct 31 00:40:23.206302 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:40:23.206264 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-0e6572-openshift-default-5c475f9cbf97qjl"] Oct 31 00:40:23.218026 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:40:23.217990 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5j2mj\" (UniqueName: \"kubernetes.io/projected/d768f7f8-eb55-4419-802a-9a4f2e6e57b8-kube-api-access-5j2mj\") pod \"e2e-worker-single-main-0e6572-openshift-default-5c475f9cbf97qjl\" (UID: \"d768f7f8-eb55-4419-802a-9a4f2e6e57b8\") " pod="openshift-ingress/e2e-worker-single-main-0e6572-openshift-default-5c475f9cbf97qjl" Oct 31 00:40:23.218223 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:40:23.218041 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d768f7f8-eb55-4419-802a-9a4f2e6e57b8-istiod-ca-cert\") pod \"e2e-worker-single-main-0e6572-openshift-default-5c475f9cbf97qjl\" (UID: \"d768f7f8-eb55-4419-802a-9a4f2e6e57b8\") " pod="openshift-ingress/e2e-worker-single-main-0e6572-openshift-default-5c475f9cbf97qjl" Oct 31 00:40:23.218223 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:40:23.218091 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d768f7f8-eb55-4419-802a-9a4f2e6e57b8-workload-socket\") pod \"e2e-worker-single-main-0e6572-openshift-default-5c475f9cbf97qjl\" (UID: \"d768f7f8-eb55-4419-802a-9a4f2e6e57b8\") " pod="openshift-ingress/e2e-worker-single-main-0e6572-openshift-default-5c475f9cbf97qjl" Oct 31 00:40:23.218223 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:40:23.218118 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d768f7f8-eb55-4419-802a-9a4f2e6e57b8-istio-podinfo\") pod \"e2e-worker-single-main-0e6572-openshift-default-5c475f9cbf97qjl\" (UID: \"d768f7f8-eb55-4419-802a-9a4f2e6e57b8\") " pod="openshift-ingress/e2e-worker-single-main-0e6572-openshift-default-5c475f9cbf97qjl" Oct 31 00:40:23.218223 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:40:23.218143 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d768f7f8-eb55-4419-802a-9a4f2e6e57b8-workload-certs\") pod \"e2e-worker-single-main-0e6572-openshift-default-5c475f9cbf97qjl\" (UID: \"d768f7f8-eb55-4419-802a-9a4f2e6e57b8\") " pod="openshift-ingress/e2e-worker-single-main-0e6572-openshift-default-5c475f9cbf97qjl" Oct 31 00:40:23.218223 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:40:23.218206 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d768f7f8-eb55-4419-802a-9a4f2e6e57b8-istio-envoy\") pod \"e2e-worker-single-main-0e6572-openshift-default-5c475f9cbf97qjl\" (UID: \"d768f7f8-eb55-4419-802a-9a4f2e6e57b8\") " pod="openshift-ingress/e2e-worker-single-main-0e6572-openshift-default-5c475f9cbf97qjl" Oct 31 00:40:23.218614 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:40:23.218590 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d768f7f8-eb55-4419-802a-9a4f2e6e57b8-istio-data\") pod \"e2e-worker-single-main-0e6572-openshift-default-5c475f9cbf97qjl\" (UID: \"d768f7f8-eb55-4419-802a-9a4f2e6e57b8\") " pod="openshift-ingress/e2e-worker-single-main-0e6572-openshift-default-5c475f9cbf97qjl" Oct 31 00:40:23.218714 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:40:23.218650 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d768f7f8-eb55-4419-802a-9a4f2e6e57b8-workload-socket\") pod \"e2e-worker-single-main-0e6572-openshift-default-5c475f9cbf97qjl\" (UID: \"d768f7f8-eb55-4419-802a-9a4f2e6e57b8\") " pod="openshift-ingress/e2e-worker-single-main-0e6572-openshift-default-5c475f9cbf97qjl" Oct 31 00:40:23.218714 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:40:23.218610 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d768f7f8-eb55-4419-802a-9a4f2e6e57b8-workload-certs\") pod \"e2e-worker-single-main-0e6572-openshift-default-5c475f9cbf97qjl\" (UID: \"d768f7f8-eb55-4419-802a-9a4f2e6e57b8\") " pod="openshift-ingress/e2e-worker-single-main-0e6572-openshift-default-5c475f9cbf97qjl" Oct 31 00:40:23.218842 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:40:23.218693 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d768f7f8-eb55-4419-802a-9a4f2e6e57b8-credential-socket\") pod \"e2e-worker-single-main-0e6572-openshift-default-5c475f9cbf97qjl\" (UID: \"d768f7f8-eb55-4419-802a-9a4f2e6e57b8\") " pod="openshift-ingress/e2e-worker-single-main-0e6572-openshift-default-5c475f9cbf97qjl" Oct 31 00:40:23.218842 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:40:23.218737 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d768f7f8-eb55-4419-802a-9a4f2e6e57b8-istio-data\") pod \"e2e-worker-single-main-0e6572-openshift-default-5c475f9cbf97qjl\" (UID: \"d768f7f8-eb55-4419-802a-9a4f2e6e57b8\") " pod="openshift-ingress/e2e-worker-single-main-0e6572-openshift-default-5c475f9cbf97qjl" Oct 31 00:40:23.218842 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:40:23.218812 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d768f7f8-eb55-4419-802a-9a4f2e6e57b8-istio-token\") pod \"e2e-worker-single-main-0e6572-openshift-default-5c475f9cbf97qjl\" (UID: \"d768f7f8-eb55-4419-802a-9a4f2e6e57b8\") " pod="openshift-ingress/e2e-worker-single-main-0e6572-openshift-default-5c475f9cbf97qjl" Oct 31 00:40:23.219084 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:40:23.219054 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d768f7f8-eb55-4419-802a-9a4f2e6e57b8-istiod-ca-cert\") pod \"e2e-worker-single-main-0e6572-openshift-default-5c475f9cbf97qjl\" (UID: \"d768f7f8-eb55-4419-802a-9a4f2e6e57b8\") " pod="openshift-ingress/e2e-worker-single-main-0e6572-openshift-default-5c475f9cbf97qjl" Oct 31 00:40:23.219279 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:40:23.219258 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d768f7f8-eb55-4419-802a-9a4f2e6e57b8-credential-socket\") pod \"e2e-worker-single-main-0e6572-openshift-default-5c475f9cbf97qjl\" (UID: \"d768f7f8-eb55-4419-802a-9a4f2e6e57b8\") " pod="openshift-ingress/e2e-worker-single-main-0e6572-openshift-default-5c475f9cbf97qjl" Oct 31 00:40:23.220950 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:40:23.220922 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d768f7f8-eb55-4419-802a-9a4f2e6e57b8-istio-envoy\") pod \"e2e-worker-single-main-0e6572-openshift-default-5c475f9cbf97qjl\" (UID: \"d768f7f8-eb55-4419-802a-9a4f2e6e57b8\") " pod="openshift-ingress/e2e-worker-single-main-0e6572-openshift-default-5c475f9cbf97qjl" Oct 31 00:40:23.221646 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:40:23.221587 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d768f7f8-eb55-4419-802a-9a4f2e6e57b8-istio-podinfo\") pod \"e2e-worker-single-main-0e6572-openshift-default-5c475f9cbf97qjl\" (UID: \"d768f7f8-eb55-4419-802a-9a4f2e6e57b8\") " pod="openshift-ingress/e2e-worker-single-main-0e6572-openshift-default-5c475f9cbf97qjl" Oct 31 00:40:23.221911 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:40:23.221886 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:40:23.283696 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:40:23.283662 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d768f7f8-eb55-4419-802a-9a4f2e6e57b8-istio-token\") pod \"e2e-worker-single-main-0e6572-openshift-default-5c475f9cbf97qjl\" (UID: \"d768f7f8-eb55-4419-802a-9a4f2e6e57b8\") " pod="openshift-ingress/e2e-worker-single-main-0e6572-openshift-default-5c475f9cbf97qjl" Oct 31 00:40:23.283879 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:40:23.283832 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5j2mj\" (UniqueName: \"kubernetes.io/projected/d768f7f8-eb55-4419-802a-9a4f2e6e57b8-kube-api-access-5j2mj\") pod \"e2e-worker-single-main-0e6572-openshift-default-5c475f9cbf97qjl\" (UID: \"d768f7f8-eb55-4419-802a-9a4f2e6e57b8\") " pod="openshift-ingress/e2e-worker-single-main-0e6572-openshift-default-5c475f9cbf97qjl" Oct 31 00:40:23.417557 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:40:23.417521 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-0e6572-openshift-default-5c475f9cbf97qjl" Oct 31 00:40:23.475474 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:40:23.475434 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:40:23.866679 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:40:23.866655 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-0e6572-openshift-default-5c475f9cbf97qjl"] Oct 31 00:40:23.867811 ip-10-0-141-180 kubenswrapper[4927]: W1031 00:40:23.867783 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd768f7f8_eb55_4419_802a_9a4f2e6e57b8.slice/crio-d1c8107633a00c0fa62489ef24df090ad5728026fe7a0bd7e54e48228e237f99 WatchSource:0}: Error finding container d1c8107633a00c0fa62489ef24df090ad5728026fe7a0bd7e54e48228e237f99: Status 404 returned error can't find the container with id d1c8107633a00c0fa62489ef24df090ad5728026fe7a0bd7e54e48228e237f99 Oct 31 00:40:23.869849 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:40:23.869817 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:40:23.869938 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:40:23.869894 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:40:23.869938 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:40:23.869927 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:40:24.547700 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:40:24.547667 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-0e6572-openshift-default-5c475f9cbf97qjl" event={"ID":"d768f7f8-eb55-4419-802a-9a4f2e6e57b8","Type":"ContainerStarted","Data":"1e976479aec8fe79960a4e1399f8f64751cbb2b2191396ec1795f6741965a2b9"} Oct 31 00:40:24.547879 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:40:24.547745 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-0e6572-openshift-default-5c475f9cbf97qjl" event={"ID":"d768f7f8-eb55-4419-802a-9a4f2e6e57b8","Type":"ContainerStarted","Data":"d1c8107633a00c0fa62489ef24df090ad5728026fe7a0bd7e54e48228e237f99"} Oct 31 00:40:24.673283 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:40:24.673228 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-0e6572-openshift-default-5c475f9cbf97qjl" podStartSLOduration=1.6732134749999998 podStartE2EDuration="1.673213475s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:40:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:40:24.670027441 +0000 UTC m=+3537.706358264" watchObservedRunningTime="2026-10-31 00:40:24.673213475 +0000 UTC m=+3537.709544365" Oct 31 00:40:25.418485 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:40:25.418450 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-0e6572-openshift-default-5c475f9cbf97qjl" Oct 31 00:40:25.423657 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:40:25.423625 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-0e6572-openshift-default-5c475f9cbf97qjl" Oct 31 00:40:25.554413 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:40:25.554383 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-0e6572-openshift-default-5c475f9cbf97qjl" Oct 31 00:40:25.555540 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:40:25.555520 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-0e6572-openshift-default-5c475f9cbf97qjl" Oct 31 00:40:27.619164 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:40:27.619128 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:40:27.780424 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:40:27.780383 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:40:27.933565 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:40:27.933534 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:40:30.753488 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:40:30.753444 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:40:30.925150 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:40:30.925104 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:40:33.931965 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:40:33.931925 4927 scope.go:122] "RemoveContainer" containerID="1392615bec61d83fcd3e76956e0c847e641612ddfa5a7ebb4e2217f2825e284d" Oct 31 00:40:34.009594 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:40:34.009558 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-0e6572-5ffbcf6485-b2hsr"] Oct 31 00:40:34.014702 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:40:34.014678 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-0e6572-5ffbcf6485-b2hsr" Oct 31 00:40:34.017461 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:40:34.017436 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/294f16fb-d757-4f90-ba41-b5c1a9180ab7-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-0e6572-5ffbcf6485-b2hsr\" (UID: \"294f16fb-d757-4f90-ba41-b5c1a9180ab7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-0e6572-5ffbcf6485-b2hsr" Oct 31 00:40:34.017562 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:40:34.017496 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/294f16fb-d757-4f90-ba41-b5c1a9180ab7-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-0e6572-5ffbcf6485-b2hsr\" (UID: \"294f16fb-d757-4f90-ba41-b5c1a9180ab7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-0e6572-5ffbcf6485-b2hsr" Oct 31 00:40:34.017604 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:40:34.017572 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fsbb5\" (UniqueName: \"kubernetes.io/projected/294f16fb-d757-4f90-ba41-b5c1a9180ab7-kube-api-access-fsbb5\") pod \"maas-api-e2e-worker-single-main-0e6572-5ffbcf6485-b2hsr\" (UID: \"294f16fb-d757-4f90-ba41-b5c1a9180ab7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-0e6572-5ffbcf6485-b2hsr" Oct 31 00:40:34.017688 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:40:34.017673 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/294f16fb-d757-4f90-ba41-b5c1a9180ab7-metrics-certs\") pod \"maas-api-e2e-worker-single-main-0e6572-5ffbcf6485-b2hsr\" (UID: \"294f16fb-d757-4f90-ba41-b5c1a9180ab7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-0e6572-5ffbcf6485-b2hsr" Oct 31 00:40:34.068924 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:40:34.068891 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-0e6572-5ffbcf6485-b2hsr"] Oct 31 00:40:34.116776 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:40:34.116736 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-0e6572-serving-cert\"" Oct 31 00:40:34.116776 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:40:34.116768 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 31 00:40:34.117022 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:40:34.116971 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-2nk58\"" Oct 31 00:40:34.118321 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:40:34.118301 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/294f16fb-d757-4f90-ba41-b5c1a9180ab7-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-0e6572-5ffbcf6485-b2hsr\" (UID: \"294f16fb-d757-4f90-ba41-b5c1a9180ab7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-0e6572-5ffbcf6485-b2hsr" Oct 31 00:40:34.118397 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:40:34.118357 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/294f16fb-d757-4f90-ba41-b5c1a9180ab7-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-0e6572-5ffbcf6485-b2hsr\" (UID: \"294f16fb-d757-4f90-ba41-b5c1a9180ab7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-0e6572-5ffbcf6485-b2hsr" Oct 31 00:40:34.118397 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:40:34.118390 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fsbb5\" (UniqueName: \"kubernetes.io/projected/294f16fb-d757-4f90-ba41-b5c1a9180ab7-kube-api-access-fsbb5\") pod \"maas-api-e2e-worker-single-main-0e6572-5ffbcf6485-b2hsr\" (UID: \"294f16fb-d757-4f90-ba41-b5c1a9180ab7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-0e6572-5ffbcf6485-b2hsr" Oct 31 00:40:34.118484 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:40:34.118431 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/294f16fb-d757-4f90-ba41-b5c1a9180ab7-metrics-certs\") pod \"maas-api-e2e-worker-single-main-0e6572-5ffbcf6485-b2hsr\" (UID: \"294f16fb-d757-4f90-ba41-b5c1a9180ab7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-0e6572-5ffbcf6485-b2hsr" Oct 31 00:40:34.118484 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:40:34.118449 4927 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-single-main-0e6572-serving-cert: secret "maas-api-e2e-worker-single-main-0e6572-serving-cert" not found Oct 31 00:40:34.118571 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:40:34.118517 4927 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/294f16fb-d757-4f90-ba41-b5c1a9180ab7-maas-api-tls podName:294f16fb-d757-4f90-ba41-b5c1a9180ab7 nodeName:}" failed. No retries permitted until 2026-10-31 00:40:34.618498303 +0000 UTC m=+3547.654829116 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/294f16fb-d757-4f90-ba41-b5c1a9180ab7-maas-api-tls") pod "maas-api-e2e-worker-single-main-0e6572-5ffbcf6485-b2hsr" (UID: "294f16fb-d757-4f90-ba41-b5c1a9180ab7") : secret "maas-api-e2e-worker-single-main-0e6572-serving-cert" not found Oct 31 00:40:34.119115 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:40:34.119094 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/294f16fb-d757-4f90-ba41-b5c1a9180ab7-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-0e6572-5ffbcf6485-b2hsr\" (UID: \"294f16fb-d757-4f90-ba41-b5c1a9180ab7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-0e6572-5ffbcf6485-b2hsr" Oct 31 00:40:34.120896 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:40:34.120879 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/294f16fb-d757-4f90-ba41-b5c1a9180ab7-metrics-certs\") pod \"maas-api-e2e-worker-single-main-0e6572-5ffbcf6485-b2hsr\" (UID: \"294f16fb-d757-4f90-ba41-b5c1a9180ab7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-0e6572-5ffbcf6485-b2hsr" Oct 31 00:40:34.180962 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:40:34.180933 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fsbb5\" (UniqueName: \"kubernetes.io/projected/294f16fb-d757-4f90-ba41-b5c1a9180ab7-kube-api-access-fsbb5\") pod \"maas-api-e2e-worker-single-main-0e6572-5ffbcf6485-b2hsr\" (UID: \"294f16fb-d757-4f90-ba41-b5c1a9180ab7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-0e6572-5ffbcf6485-b2hsr" Oct 31 00:40:34.213542 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:40:34.213420 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:40:34.390646 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:40:34.390612 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:40:34.549230 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:40:34.549139 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:40:34.623198 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:40:34.623166 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/294f16fb-d757-4f90-ba41-b5c1a9180ab7-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-0e6572-5ffbcf6485-b2hsr\" (UID: \"294f16fb-d757-4f90-ba41-b5c1a9180ab7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-0e6572-5ffbcf6485-b2hsr" Oct 31 00:40:34.625723 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:40:34.625698 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/294f16fb-d757-4f90-ba41-b5c1a9180ab7-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-0e6572-5ffbcf6485-b2hsr\" (UID: \"294f16fb-d757-4f90-ba41-b5c1a9180ab7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-0e6572-5ffbcf6485-b2hsr" Oct 31 00:40:34.629444 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:40:34.629418 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-0e6572-5ffbcf6485-b2hsr" Oct 31 00:40:35.011148 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:40:35.011116 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-0e6572-5ffbcf6485-b2hsr"] Oct 31 00:40:35.012167 ip-10-0-141-180 kubenswrapper[4927]: W1031 00:40:35.012142 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod294f16fb_d757_4f90_ba41_b5c1a9180ab7.slice/crio-6963820ab1ab385f8eef4a187ca492f7d111fa123db493fe5455053f1acd6cec WatchSource:0}: Error finding container 6963820ab1ab385f8eef4a187ca492f7d111fa123db493fe5455053f1acd6cec: Status 404 returned error can't find the container with id 6963820ab1ab385f8eef4a187ca492f7d111fa123db493fe5455053f1acd6cec Oct 31 00:40:35.013553 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:40:35.013534 4927 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 00:40:35.615013 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:40:35.614975 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-0e6572-5ffbcf6485-b2hsr" event={"ID":"294f16fb-d757-4f90-ba41-b5c1a9180ab7","Type":"ContainerStarted","Data":"6963820ab1ab385f8eef4a187ca492f7d111fa123db493fe5455053f1acd6cec"} Oct 31 00:40:36.622234 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:40:36.622190 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-0e6572-5ffbcf6485-b2hsr" event={"ID":"294f16fb-d757-4f90-ba41-b5c1a9180ab7","Type":"ContainerStarted","Data":"b5d58c38aabb4dd8f37c897c890f7469b8132afb1bf1c2a75740b649dc3d099d"} Oct 31 00:40:36.622694 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:40:36.622370 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-0e6572-5ffbcf6485-b2hsr" Oct 31 00:40:36.749665 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:40:36.749547 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-0e6572-5ffbcf6485-b2hsr" podStartSLOduration=2.794457326 podStartE2EDuration="3.74953324s" totalImagesPullingTime="955.075914ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:40:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:40:36.748019026 +0000 UTC m=+3549.784349849" watchObservedRunningTime="2026-10-31 00:40:36.74953324 +0000 UTC m=+3549.785864061" Oct 31 00:40:38.491189 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:40:38.491135 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:40:39.052079 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:40:39.052045 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-4rx5g"] Oct 31 00:40:39.103113 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:40:39.103078 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-4rx5g"] Oct 31 00:40:39.391150 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:40:39.391063 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:40:41.085228 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:40:41.085196 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce97800b-46eb-4f70-940a-c5d6e46457dc" path="/var/lib/kubelet/pods/ce97800b-46eb-4f70-940a-c5d6e46457dc/volumes" Oct 31 00:40:42.642443 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:40:42.642409 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-0e6572-5ffbcf6485-b2hsr" Oct 31 00:40:44.453542 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:40:44.453506 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:40:49.477493 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:40:49.477451 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:40:49.677678 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:40:49.677636 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sqv69" containerName="extractor" Oct 31 00:40:50.050172 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:40:50.050137 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-6nz9m"] Oct 31 00:40:50.101439 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:40:50.101408 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-6nz9m"] Oct 31 00:40:51.086472 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:40:51.086438 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7015ec94-e3c9-45c9-891b-7569c390bec7" path="/var/lib/kubelet/pods/7015ec94-e3c9-45c9-891b-7569c390bec7/volumes" Oct 31 00:40:51.753227 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:40:51.753188 4927 generic.go:356] "Generic (PLEG): container finished" podID="d768f7f8-eb55-4419-802a-9a4f2e6e57b8" containerID="1e976479aec8fe79960a4e1399f8f64751cbb2b2191396ec1795f6741965a2b9" exitCode=0 Oct 31 00:40:51.753476 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:40:51.753260 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-0e6572-openshift-default-5c475f9cbf97qjl" event={"ID":"d768f7f8-eb55-4419-802a-9a4f2e6e57b8","Type":"ContainerDied","Data":"1e976479aec8fe79960a4e1399f8f64751cbb2b2191396ec1795f6741965a2b9"} Oct 31 00:40:51.753693 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:40:51.753678 4927 scope.go:122] "RemoveContainer" containerID="1e976479aec8fe79960a4e1399f8f64751cbb2b2191396ec1795f6741965a2b9" Oct 31 00:40:51.754916 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:40:51.754888 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:40:51.754994 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:40:51.754971 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:40:51.755048 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:40:51.755016 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:40:52.354254 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:40:52.354217 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:40:52.761589 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:40:52.761553 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-0e6572-openshift-default-5c475f9cbf97qjl" event={"ID":"d768f7f8-eb55-4419-802a-9a4f2e6e57b8","Type":"ContainerStarted","Data":"fcfc5ee368ea577dfb6d8d113befa0a945617b1747099f6f7f3683900569ce93"} Oct 31 00:40:52.904747 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:40:52.904703 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:40:53.099662 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:40:53.099574 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:40:53.419687 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:40:53.419656 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-0e6572-openshift-default-5c475f9cbf97qjl" Oct 31 00:40:53.432500 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:40:53.432462 4927 patch_prober.go:29] interesting pod/e2e-worker-single-main-0e6572-openshift-default-5c475f9cbf97qjl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.32:15021/healthz/ready\": dial tcp 10.132.1.32:15021: connect: connection refused" start-of-body= Oct 31 00:40:53.432663 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:40:53.432527 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-0e6572-openshift-default-5c475f9cbf97qjl" podUID="d768f7f8-eb55-4419-802a-9a4f2e6e57b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.32:15021/healthz/ready\": dial tcp 10.132.1.32:15021: connect: connection refused" Oct 31 00:40:54.428356 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:40:54.418636 4927 patch_prober.go:29] interesting pod/e2e-worker-single-main-0e6572-openshift-default-5c475f9cbf97qjl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.32:15021/healthz/ready\": dial tcp 10.132.1.32:15021: connect: connection refused" start-of-body= Oct 31 00:40:54.428356 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:40:54.418697 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-0e6572-openshift-default-5c475f9cbf97qjl" podUID="d768f7f8-eb55-4419-802a-9a4f2e6e57b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.32:15021/healthz/ready\": dial tcp 10.132.1.32:15021: connect: connection refused" Oct 31 00:40:54.791010 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:40:54.790925 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:40:55.421737 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:40:55.421695 4927 patch_prober.go:29] interesting pod/e2e-worker-single-main-0e6572-openshift-default-5c475f9cbf97qjl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.32:15021/healthz/ready\": dial tcp 10.132.1.32:15021: connect: connection refused" start-of-body= Oct 31 00:40:55.422033 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:40:55.421999 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-0e6572-openshift-default-5c475f9cbf97qjl" podUID="d768f7f8-eb55-4419-802a-9a4f2e6e57b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.32:15021/healthz/ready\": dial tcp 10.132.1.32:15021: connect: connection refused" Oct 31 00:40:55.554999 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:40:55.554962 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-0e6572-openshift-default-5c475f9cbf97qjl" Oct 31 00:40:56.279995 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:40:56.279957 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:40:56.421460 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:40:56.421421 4927 patch_prober.go:29] interesting pod/e2e-worker-single-main-0e6572-openshift-default-5c475f9cbf97qjl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.32:15021/healthz/ready\": dial tcp 10.132.1.32:15021: connect: connection refused" start-of-body= Oct 31 00:40:56.421743 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:40:56.421716 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-0e6572-openshift-default-5c475f9cbf97qjl" podUID="d768f7f8-eb55-4419-802a-9a4f2e6e57b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.32:15021/healthz/ready\": dial tcp 10.132.1.32:15021: connect: connection refused" Oct 31 00:40:57.424789 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:40:57.424746 4927 patch_prober.go:29] interesting pod/e2e-worker-single-main-0e6572-openshift-default-5c475f9cbf97qjl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.32:15021/healthz/ready\": dial tcp 10.132.1.32:15021: connect: connection refused" start-of-body= Oct 31 00:40:57.425453 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:40:57.425418 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-0e6572-openshift-default-5c475f9cbf97qjl" podUID="d768f7f8-eb55-4419-802a-9a4f2e6e57b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.32:15021/healthz/ready\": dial tcp 10.132.1.32:15021: connect: connection refused" Oct 31 00:40:58.422966 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:40:58.422934 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-0e6572-openshift-default-5c475f9cbf97qjl" Oct 31 00:40:58.423961 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:40:58.423940 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-0e6572-openshift-default-5c475f9cbf97qjl" Oct 31 00:40:59.780092 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:40:59.780061 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:41:04.894063 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:41:04.894028 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:41:09.875478 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:41:09.875441 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:41:14.964371 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:41:14.964315 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:41:19.677242 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:41:19.677203 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sqv69" containerName="extractor" Oct 31 00:41:19.993296 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:41:19.993242 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:41:25.071028 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:41:25.070988 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:41:27.049915 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:41:27.049860 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-2nngx"] Oct 31 00:41:27.101204 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:41:27.101173 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-2nngx"] Oct 31 00:41:27.135023 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:41:27.134977 4927 status_manager.go:1045] "Failed to get status for pod" err="pods \"maas-api-revoke-sub-ae594979-2nngx\" is forbidden: User \"system:node:ip-10-0-141-180.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"odh-ai-gateway-infra\": no relationship found between node 'ip-10-0-141-180.us-west-2.compute.internal' and this object" podUID="12635fcb-b16d-47d3-b03d-c173ffd4e612" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-2nngx" Oct 31 00:41:28.844720 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:41:28.844690 4927 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-pqt2c_6382dfb5-a363-47a2-bc79-0b96f7bec0c3/console-operator/3.log" Oct 31 00:41:29.085568 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:41:29.085530 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="12635fcb-b16d-47d3-b03d-c173ffd4e612" path="/var/lib/kubelet/pods/12635fcb-b16d-47d3-b03d-c173ffd4e612/volumes" Oct 31 00:41:29.871470 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:41:29.871442 4927 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-pqt2c_6382dfb5-a363-47a2-bc79-0b96f7bec0c3/console-operator/3.log" Oct 31 00:41:30.091556 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:41:30.091516 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:41:33.958642 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:41:33.958610 4927 scope.go:122] "RemoveContainer" containerID="caee348f9db214de9716c2e9bbeddd5e835949f2f03b66424c5bafaa8e189563" Oct 31 00:41:33.970230 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:41:33.970211 4927 scope.go:122] "RemoveContainer" containerID="537913eaa90ad68dd091a12007c37050a9bc9d9b1e9c650d6d3f910ed9758426" Oct 31 00:41:35.162213 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:41:35.162176 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:41:38.833400 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:41:38.833327 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:41:40.204297 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:41:40.204255 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:41:45.268210 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:41:45.268170 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:41:49.678829 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:41:49.678785 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sqv69" containerName="extractor" Oct 31 00:41:50.281619 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:41:50.281582 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:41:52.422676 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:41:52.422641 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:41:55.686391 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:41:55.686345 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:41:56.051135 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:41:56.051047 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-kgzrm"] Oct 31 00:41:56.101839 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:41:56.101805 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-kgzrm"] Oct 31 00:41:57.091219 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:41:57.091176 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f57beea2-7bb2-464d-8e97-506b915456fa" path="/var/lib/kubelet/pods/f57beea2-7bb2-464d-8e97-506b915456fa/volumes" Oct 31 00:41:57.467512 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:41:57.467473 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:42:00.784185 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:42:00.784154 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:42:04.722175 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:42:04.722138 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:42:04.868034 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:42:04.867996 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:42:05.086959 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:42:05.086878 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-0e6572-5ffbcf6485-b2hsr"] Oct 31 00:42:05.087141 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:42:05.087055 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-0e6572-5ffbcf6485-b2hsr" podUID="294f16fb-d757-4f90-ba41-b5c1a9180ab7" containerName="maas-api" containerID="cri-o://b5d58c38aabb4dd8f37c897c890f7469b8132afb1bf1c2a75740b649dc3d099d" gracePeriod=30 Oct 31 00:42:05.194143 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:42:05.194106 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:42:06.233140 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:42:06.233091 4927 generic.go:356] "Generic (PLEG): container finished" podID="294f16fb-d757-4f90-ba41-b5c1a9180ab7" containerID="b5d58c38aabb4dd8f37c897c890f7469b8132afb1bf1c2a75740b649dc3d099d" exitCode=0 Oct 31 00:42:06.233648 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:42:06.233300 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-0e6572-5ffbcf6485-b2hsr" event={"ID":"294f16fb-d757-4f90-ba41-b5c1a9180ab7","Type":"ContainerDied","Data":"b5d58c38aabb4dd8f37c897c890f7469b8132afb1bf1c2a75740b649dc3d099d"} Oct 31 00:42:06.340783 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:42:06.340757 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-0e6572-5ffbcf6485-b2hsr" Oct 31 00:42:06.509139 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:42:06.509094 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/294f16fb-d757-4f90-ba41-b5c1a9180ab7-kube-api-access-fsbb5\" (UniqueName: \"kubernetes.io/projected/294f16fb-d757-4f90-ba41-b5c1a9180ab7-kube-api-access-fsbb5\") pod \"294f16fb-d757-4f90-ba41-b5c1a9180ab7\" (UID: \"294f16fb-d757-4f90-ba41-b5c1a9180ab7\") " Oct 31 00:42:06.509416 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:42:06.509158 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/294f16fb-d757-4f90-ba41-b5c1a9180ab7-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/294f16fb-d757-4f90-ba41-b5c1a9180ab7-maas-api-tls\") pod \"294f16fb-d757-4f90-ba41-b5c1a9180ab7\" (UID: \"294f16fb-d757-4f90-ba41-b5c1a9180ab7\") " Oct 31 00:42:06.509416 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:42:06.509226 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/294f16fb-d757-4f90-ba41-b5c1a9180ab7-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/294f16fb-d757-4f90-ba41-b5c1a9180ab7-openshift-service-ca\") pod \"294f16fb-d757-4f90-ba41-b5c1a9180ab7\" (UID: \"294f16fb-d757-4f90-ba41-b5c1a9180ab7\") " Oct 31 00:42:06.509416 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:42:06.509303 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/294f16fb-d757-4f90-ba41-b5c1a9180ab7-metrics-certs\" (UniqueName: \"kubernetes.io/secret/294f16fb-d757-4f90-ba41-b5c1a9180ab7-metrics-certs\") pod \"294f16fb-d757-4f90-ba41-b5c1a9180ab7\" (UID: \"294f16fb-d757-4f90-ba41-b5c1a9180ab7\") " Oct 31 00:42:06.509726 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:42:06.509696 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/294f16fb-d757-4f90-ba41-b5c1a9180ab7-openshift-service-ca" pod "294f16fb-d757-4f90-ba41-b5c1a9180ab7" (UID: "294f16fb-d757-4f90-ba41-b5c1a9180ab7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:42:06.511433 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:42:06.511407 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/294f16fb-d757-4f90-ba41-b5c1a9180ab7-maas-api-tls" pod "294f16fb-d757-4f90-ba41-b5c1a9180ab7" (UID: "294f16fb-d757-4f90-ba41-b5c1a9180ab7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:42:06.511544 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:42:06.511498 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/294f16fb-d757-4f90-ba41-b5c1a9180ab7-kube-api-access-fsbb5" pod "294f16fb-d757-4f90-ba41-b5c1a9180ab7" (UID: "294f16fb-d757-4f90-ba41-b5c1a9180ab7"). InnerVolumeSpecName "kube-api-access-fsbb5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:42:06.511606 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:42:06.511591 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/294f16fb-d757-4f90-ba41-b5c1a9180ab7-metrics-certs" pod "294f16fb-d757-4f90-ba41-b5c1a9180ab7" (UID: "294f16fb-d757-4f90-ba41-b5c1a9180ab7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:42:06.610276 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:42:06.610236 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fsbb5\" (UniqueName: \"kubernetes.io/projected/294f16fb-d757-4f90-ba41-b5c1a9180ab7-kube-api-access-fsbb5\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:42:06.610276 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:42:06.610271 4927 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/294f16fb-d757-4f90-ba41-b5c1a9180ab7-maas-api-tls\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:42:06.610276 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:42:06.610281 4927 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/294f16fb-d757-4f90-ba41-b5c1a9180ab7-openshift-service-ca\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:42:06.610557 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:42:06.610291 4927 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/294f16fb-d757-4f90-ba41-b5c1a9180ab7-metrics-certs\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:42:07.242024 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:42:07.241992 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-0e6572-5ffbcf6485-b2hsr" Oct 31 00:42:07.242499 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:42:07.241997 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-0e6572-5ffbcf6485-b2hsr" event={"ID":"294f16fb-d757-4f90-ba41-b5c1a9180ab7","Type":"ContainerDied","Data":"6963820ab1ab385f8eef4a187ca492f7d111fa123db493fe5455053f1acd6cec"} Oct 31 00:42:07.242499 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:42:07.242137 4927 scope.go:122] "RemoveContainer" containerID="b5d58c38aabb4dd8f37c897c890f7469b8132afb1bf1c2a75740b649dc3d099d" Oct 31 00:42:07.412601 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:42:07.412564 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-0e6572-5ffbcf6485-b2hsr"] Oct 31 00:42:07.461741 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:42:07.461699 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-0e6572-5ffbcf6485-b2hsr"] Oct 31 00:42:08.787553 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:42:08.787520 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:42:08.989972 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:42:08.989937 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:42:09.086147 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:42:09.086061 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="294f16fb-d757-4f90-ba41-b5c1a9180ab7" path="/var/lib/kubelet/pods/294f16fb-d757-4f90-ba41-b5c1a9180ab7/volumes" Oct 31 00:42:09.789034 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:42:09.789002 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:42:15.570093 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:42:15.570055 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:42:15.767340 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:42:15.767289 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:42:15.899501 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:42:15.899420 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-0e6572-openshift-default-5c475f9cbf97qjl"] Oct 31 00:42:15.899679 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:42:15.899637 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-0e6572-openshift-default-5c475f9cbf97qjl" podUID="d768f7f8-eb55-4419-802a-9a4f2e6e57b8" containerName="istio-proxy" containerID="cri-o://fcfc5ee368ea577dfb6d8d113befa0a945617b1747099f6f7f3683900569ce93" gracePeriod=30 Oct 31 00:42:15.938015 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:42:15.937981 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:42:17.444930 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:42:17.444884 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zk6cp"] Oct 31 00:42:17.495900 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:42:17.495863 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zk6cp"] Oct 31 00:42:17.577401 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:42:17.577364 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zmqlp"] Oct 31 00:42:17.577812 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:42:17.577799 4927 cpu_manager.go:414] "containerMap: removing container" podUID="294f16fb-d757-4f90-ba41-b5c1a9180ab7" containerName="maas-api" Oct 31 00:42:17.577861 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:42:17.577813 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="294f16fb-d757-4f90-ba41-b5c1a9180ab7" containerName="maas-api" Oct 31 00:42:17.577902 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:42:17.577879 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="294f16fb-d757-4f90-ba41-b5c1a9180ab7" containerName="maas-api" Oct 31 00:42:17.581027 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:42:17.581011 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zmqlp" Oct 31 00:42:17.617287 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:42:17.617257 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff7abf47-1bc8-47ec-92ef-33e82aeb1e92-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-zmqlp\" (UID: \"ff7abf47-1bc8-47ec-92ef-33e82aeb1e92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zmqlp" Oct 31 00:42:17.617456 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:42:17.617301 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff7abf47-1bc8-47ec-92ef-33e82aeb1e92-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-zmqlp\" (UID: \"ff7abf47-1bc8-47ec-92ef-33e82aeb1e92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zmqlp" Oct 31 00:42:17.617536 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:42:17.617458 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pg4kj\" (UniqueName: \"kubernetes.io/projected/ff7abf47-1bc8-47ec-92ef-33e82aeb1e92-kube-api-access-pg4kj\") pod \"maas-api-revoke-sub-3d4156f1-zmqlp\" (UID: \"ff7abf47-1bc8-47ec-92ef-33e82aeb1e92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zmqlp" Oct 31 00:42:17.634796 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:42:17.634770 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4fdzs\"" Oct 31 00:42:17.635251 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:42:17.635226 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:42:17.635679 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:42:17.635656 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zmqlp"] Oct 31 00:42:17.718403 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:42:17.718298 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff7abf47-1bc8-47ec-92ef-33e82aeb1e92-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-zmqlp\" (UID: \"ff7abf47-1bc8-47ec-92ef-33e82aeb1e92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zmqlp" Oct 31 00:42:17.718403 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:42:17.718354 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff7abf47-1bc8-47ec-92ef-33e82aeb1e92-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-zmqlp\" (UID: \"ff7abf47-1bc8-47ec-92ef-33e82aeb1e92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zmqlp" Oct 31 00:42:17.718646 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:42:17.718411 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pg4kj\" (UniqueName: \"kubernetes.io/projected/ff7abf47-1bc8-47ec-92ef-33e82aeb1e92-kube-api-access-pg4kj\") pod \"maas-api-revoke-sub-3d4156f1-zmqlp\" (UID: \"ff7abf47-1bc8-47ec-92ef-33e82aeb1e92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zmqlp" Oct 31 00:42:17.719025 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:42:17.719004 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff7abf47-1bc8-47ec-92ef-33e82aeb1e92-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-zmqlp\" (UID: \"ff7abf47-1bc8-47ec-92ef-33e82aeb1e92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zmqlp" Oct 31 00:42:17.720716 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:42:17.720690 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff7abf47-1bc8-47ec-92ef-33e82aeb1e92-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-zmqlp\" (UID: \"ff7abf47-1bc8-47ec-92ef-33e82aeb1e92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zmqlp" Oct 31 00:42:17.780920 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:42:17.780887 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pg4kj\" (UniqueName: \"kubernetes.io/projected/ff7abf47-1bc8-47ec-92ef-33e82aeb1e92-kube-api-access-pg4kj\") pod \"maas-api-revoke-sub-3d4156f1-zmqlp\" (UID: \"ff7abf47-1bc8-47ec-92ef-33e82aeb1e92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zmqlp" Oct 31 00:42:17.893144 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:42:17.893101 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zmqlp" Oct 31 00:42:18.275275 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:42:18.275239 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zmqlp"] Oct 31 00:42:18.277071 ip-10-0-141-180 kubenswrapper[4927]: W1031 00:42:18.277037 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff7abf47_1bc8_47ec_92ef_33e82aeb1e92.slice/crio-2aacf573ab156f6f9723069da2961e8801a96bcb3735dbbcb68d5536523f479e WatchSource:0}: Error finding container 2aacf573ab156f6f9723069da2961e8801a96bcb3735dbbcb68d5536523f479e: Status 404 returned error can't find the container with id 2aacf573ab156f6f9723069da2961e8801a96bcb3735dbbcb68d5536523f479e Oct 31 00:42:18.311984 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:42:18.311956 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zmqlp" event={"ID":"ff7abf47-1bc8-47ec-92ef-33e82aeb1e92","Type":"ContainerStarted","Data":"2aacf573ab156f6f9723069da2961e8801a96bcb3735dbbcb68d5536523f479e"} Oct 31 00:42:19.084794 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:42:19.084759 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c96e602a-b8ab-48f2-a269-c4f6205d55a0" path="/var/lib/kubelet/pods/c96e602a-b8ab-48f2-a269-c4f6205d55a0/volumes" Oct 31 00:42:19.320093 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:42:19.320063 4927 generic.go:356] "Generic (PLEG): container finished" podID="ff7abf47-1bc8-47ec-92ef-33e82aeb1e92" containerID="b2ba3ec6c398097ef182613f93f4ed52c0dd385c620fe383e5a69a56f644f442" exitCode=0 Oct 31 00:42:19.320246 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:42:19.320149 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zmqlp" event={"ID":"ff7abf47-1bc8-47ec-92ef-33e82aeb1e92","Type":"ContainerDied","Data":"b2ba3ec6c398097ef182613f93f4ed52c0dd385c620fe383e5a69a56f644f442"} Oct 31 00:42:19.677483 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:42:19.677447 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sqv69" containerName="extractor" Oct 31 00:42:20.459850 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:42:20.459825 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zmqlp" Oct 31 00:42:20.543244 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:42:20.543215 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff7abf47-1bc8-47ec-92ef-33e82aeb1e92-kube-api-access-pg4kj\" (UniqueName: \"kubernetes.io/projected/ff7abf47-1bc8-47ec-92ef-33e82aeb1e92-kube-api-access-pg4kj\") pod \"ff7abf47-1bc8-47ec-92ef-33e82aeb1e92\" (UID: \"ff7abf47-1bc8-47ec-92ef-33e82aeb1e92\") " Oct 31 00:42:20.543430 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:42:20.543291 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff7abf47-1bc8-47ec-92ef-33e82aeb1e92-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff7abf47-1bc8-47ec-92ef-33e82aeb1e92-maas-api-openshift-ca\") pod \"ff7abf47-1bc8-47ec-92ef-33e82aeb1e92\" (UID: \"ff7abf47-1bc8-47ec-92ef-33e82aeb1e92\") " Oct 31 00:42:20.543430 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:42:20.543317 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ff7abf47-1bc8-47ec-92ef-33e82aeb1e92-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff7abf47-1bc8-47ec-92ef-33e82aeb1e92-maas-api-xks-ca\") pod \"ff7abf47-1bc8-47ec-92ef-33e82aeb1e92\" (UID: \"ff7abf47-1bc8-47ec-92ef-33e82aeb1e92\") " Oct 31 00:42:20.543635 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:42:20.543613 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff7abf47-1bc8-47ec-92ef-33e82aeb1e92-maas-api-openshift-ca" pod "ff7abf47-1bc8-47ec-92ef-33e82aeb1e92" (UID: "ff7abf47-1bc8-47ec-92ef-33e82aeb1e92"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:42:20.545548 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:42:20.545520 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff7abf47-1bc8-47ec-92ef-33e82aeb1e92-maas-api-xks-ca" pod "ff7abf47-1bc8-47ec-92ef-33e82aeb1e92" (UID: "ff7abf47-1bc8-47ec-92ef-33e82aeb1e92"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:42:20.545548 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:42:20.545541 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff7abf47-1bc8-47ec-92ef-33e82aeb1e92-kube-api-access-pg4kj" pod "ff7abf47-1bc8-47ec-92ef-33e82aeb1e92" (UID: "ff7abf47-1bc8-47ec-92ef-33e82aeb1e92"). InnerVolumeSpecName "kube-api-access-pg4kj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:42:20.644741 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:42:20.644654 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pg4kj\" (UniqueName: \"kubernetes.io/projected/ff7abf47-1bc8-47ec-92ef-33e82aeb1e92-kube-api-access-pg4kj\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:42:20.644741 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:42:20.644685 4927 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff7abf47-1bc8-47ec-92ef-33e82aeb1e92-maas-api-openshift-ca\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:42:20.644741 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:42:20.644700 4927 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff7abf47-1bc8-47ec-92ef-33e82aeb1e92-maas-api-xks-ca\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:42:21.139033 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:42:21.139007 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-0e6572-openshift-default-5c475f9cbf97qjl" Oct 31 00:42:21.250896 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:42:21.250823 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d768f7f8-eb55-4419-802a-9a4f2e6e57b8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d768f7f8-eb55-4419-802a-9a4f2e6e57b8-istiod-ca-cert\") pod \"d768f7f8-eb55-4419-802a-9a4f2e6e57b8\" (UID: \"d768f7f8-eb55-4419-802a-9a4f2e6e57b8\") " Oct 31 00:42:21.250896 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:42:21.250885 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d768f7f8-eb55-4419-802a-9a4f2e6e57b8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d768f7f8-eb55-4419-802a-9a4f2e6e57b8-workload-socket\") pod \"d768f7f8-eb55-4419-802a-9a4f2e6e57b8\" (UID: \"d768f7f8-eb55-4419-802a-9a4f2e6e57b8\") " Oct 31 00:42:21.251094 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:42:21.250905 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d768f7f8-eb55-4419-802a-9a4f2e6e57b8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d768f7f8-eb55-4419-802a-9a4f2e6e57b8-istio-data\") pod \"d768f7f8-eb55-4419-802a-9a4f2e6e57b8\" (UID: \"d768f7f8-eb55-4419-802a-9a4f2e6e57b8\") " Oct 31 00:42:21.251094 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:42:21.250940 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d768f7f8-eb55-4419-802a-9a4f2e6e57b8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d768f7f8-eb55-4419-802a-9a4f2e6e57b8-istio-envoy\") pod \"d768f7f8-eb55-4419-802a-9a4f2e6e57b8\" (UID: \"d768f7f8-eb55-4419-802a-9a4f2e6e57b8\") " Oct 31 00:42:21.251094 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:42:21.250982 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d768f7f8-eb55-4419-802a-9a4f2e6e57b8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d768f7f8-eb55-4419-802a-9a4f2e6e57b8-istio-podinfo\") pod \"d768f7f8-eb55-4419-802a-9a4f2e6e57b8\" (UID: \"d768f7f8-eb55-4419-802a-9a4f2e6e57b8\") " Oct 31 00:42:21.251094 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:42:21.251015 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d768f7f8-eb55-4419-802a-9a4f2e6e57b8-kube-api-access-5j2mj\" (UniqueName: \"kubernetes.io/projected/d768f7f8-eb55-4419-802a-9a4f2e6e57b8-kube-api-access-5j2mj\") pod \"d768f7f8-eb55-4419-802a-9a4f2e6e57b8\" (UID: \"d768f7f8-eb55-4419-802a-9a4f2e6e57b8\") " Oct 31 00:42:21.251094 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:42:21.251056 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d768f7f8-eb55-4419-802a-9a4f2e6e57b8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d768f7f8-eb55-4419-802a-9a4f2e6e57b8-credential-socket\") pod \"d768f7f8-eb55-4419-802a-9a4f2e6e57b8\" (UID: \"d768f7f8-eb55-4419-802a-9a4f2e6e57b8\") " Oct 31 00:42:21.251424 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:42:21.251111 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d768f7f8-eb55-4419-802a-9a4f2e6e57b8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d768f7f8-eb55-4419-802a-9a4f2e6e57b8-workload-certs\") pod \"d768f7f8-eb55-4419-802a-9a4f2e6e57b8\" (UID: \"d768f7f8-eb55-4419-802a-9a4f2e6e57b8\") " Oct 31 00:42:21.251424 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:42:21.251148 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d768f7f8-eb55-4419-802a-9a4f2e6e57b8-istio-token\" (UniqueName: \"kubernetes.io/projected/d768f7f8-eb55-4419-802a-9a4f2e6e57b8-istio-token\") pod \"d768f7f8-eb55-4419-802a-9a4f2e6e57b8\" (UID: \"d768f7f8-eb55-4419-802a-9a4f2e6e57b8\") " Oct 31 00:42:21.251424 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:42:21.251237 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d768f7f8-eb55-4419-802a-9a4f2e6e57b8-workload-socket" pod "d768f7f8-eb55-4419-802a-9a4f2e6e57b8" (UID: "d768f7f8-eb55-4419-802a-9a4f2e6e57b8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:42:21.251424 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:42:21.251254 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d768f7f8-eb55-4419-802a-9a4f2e6e57b8-istio-data" pod "d768f7f8-eb55-4419-802a-9a4f2e6e57b8" (UID: "d768f7f8-eb55-4419-802a-9a4f2e6e57b8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:42:21.251424 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:42:21.251285 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d768f7f8-eb55-4419-802a-9a4f2e6e57b8-istiod-ca-cert" pod "d768f7f8-eb55-4419-802a-9a4f2e6e57b8" (UID: "d768f7f8-eb55-4419-802a-9a4f2e6e57b8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:42:21.251697 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:42:21.251475 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d768f7f8-eb55-4419-802a-9a4f2e6e57b8-credential-socket" pod "d768f7f8-eb55-4419-802a-9a4f2e6e57b8" (UID: "d768f7f8-eb55-4419-802a-9a4f2e6e57b8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:42:21.251697 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:42:21.251518 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d768f7f8-eb55-4419-802a-9a4f2e6e57b8-workload-certs" pod "d768f7f8-eb55-4419-802a-9a4f2e6e57b8" (UID: "d768f7f8-eb55-4419-802a-9a4f2e6e57b8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:42:21.251697 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:42:21.251609 4927 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d768f7f8-eb55-4419-802a-9a4f2e6e57b8-credential-socket\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:42:21.251697 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:42:21.251622 4927 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d768f7f8-eb55-4419-802a-9a4f2e6e57b8-workload-certs\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:42:21.251697 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:42:21.251632 4927 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d768f7f8-eb55-4419-802a-9a4f2e6e57b8-istiod-ca-cert\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:42:21.251697 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:42:21.251642 4927 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d768f7f8-eb55-4419-802a-9a4f2e6e57b8-workload-socket\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:42:21.251697 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:42:21.251650 4927 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d768f7f8-eb55-4419-802a-9a4f2e6e57b8-istio-data\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:42:21.253275 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:42:21.253252 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d768f7f8-eb55-4419-802a-9a4f2e6e57b8-istio-envoy" pod "d768f7f8-eb55-4419-802a-9a4f2e6e57b8" (UID: "d768f7f8-eb55-4419-802a-9a4f2e6e57b8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:42:21.253410 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:42:21.253323 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d768f7f8-eb55-4419-802a-9a4f2e6e57b8-istio-token" pod "d768f7f8-eb55-4419-802a-9a4f2e6e57b8" (UID: "d768f7f8-eb55-4419-802a-9a4f2e6e57b8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:42:21.253410 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:42:21.253382 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d768f7f8-eb55-4419-802a-9a4f2e6e57b8-kube-api-access-5j2mj" pod "d768f7f8-eb55-4419-802a-9a4f2e6e57b8" (UID: "d768f7f8-eb55-4419-802a-9a4f2e6e57b8"). InnerVolumeSpecName "kube-api-access-5j2mj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:42:21.253410 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:42:21.253382 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d768f7f8-eb55-4419-802a-9a4f2e6e57b8-istio-podinfo" pod "d768f7f8-eb55-4419-802a-9a4f2e6e57b8" (UID: "d768f7f8-eb55-4419-802a-9a4f2e6e57b8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:42:21.334456 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:42:21.334422 4927 generic.go:356] "Generic (PLEG): container finished" podID="d768f7f8-eb55-4419-802a-9a4f2e6e57b8" containerID="fcfc5ee368ea577dfb6d8d113befa0a945617b1747099f6f7f3683900569ce93" exitCode=0 Oct 31 00:42:21.334613 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:42:21.334477 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-0e6572-openshift-default-5c475f9cbf97qjl" Oct 31 00:42:21.334613 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:42:21.334502 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-0e6572-openshift-default-5c475f9cbf97qjl" event={"ID":"d768f7f8-eb55-4419-802a-9a4f2e6e57b8","Type":"ContainerDied","Data":"fcfc5ee368ea577dfb6d8d113befa0a945617b1747099f6f7f3683900569ce93"} Oct 31 00:42:21.334613 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:42:21.334564 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-0e6572-openshift-default-5c475f9cbf97qjl" event={"ID":"d768f7f8-eb55-4419-802a-9a4f2e6e57b8","Type":"ContainerDied","Data":"d1c8107633a00c0fa62489ef24df090ad5728026fe7a0bd7e54e48228e237f99"} Oct 31 00:42:21.334613 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:42:21.334588 4927 scope.go:122] "RemoveContainer" containerID="fcfc5ee368ea577dfb6d8d113befa0a945617b1747099f6f7f3683900569ce93" Oct 31 00:42:21.336307 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:42:21.336190 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zmqlp" event={"ID":"ff7abf47-1bc8-47ec-92ef-33e82aeb1e92","Type":"ContainerDied","Data":"2aacf573ab156f6f9723069da2961e8801a96bcb3735dbbcb68d5536523f479e"} Oct 31 00:42:21.336307 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:42:21.336233 4927 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2aacf573ab156f6f9723069da2961e8801a96bcb3735dbbcb68d5536523f479e" Oct 31 00:42:21.336307 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:42:21.336240 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zmqlp" Oct 31 00:42:21.347837 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:42:21.347815 4927 scope.go:122] "RemoveContainer" containerID="1e976479aec8fe79960a4e1399f8f64751cbb2b2191396ec1795f6741965a2b9" Oct 31 00:42:21.352270 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:42:21.352246 4927 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d768f7f8-eb55-4419-802a-9a4f2e6e57b8-istio-envoy\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:42:21.352410 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:42:21.352273 4927 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d768f7f8-eb55-4419-802a-9a4f2e6e57b8-istio-podinfo\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:42:21.352410 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:42:21.352284 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5j2mj\" (UniqueName: \"kubernetes.io/projected/d768f7f8-eb55-4419-802a-9a4f2e6e57b8-kube-api-access-5j2mj\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:42:21.352410 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:42:21.352294 4927 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d768f7f8-eb55-4419-802a-9a4f2e6e57b8-istio-token\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:42:21.356417 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:42:21.356401 4927 scope.go:122] "RemoveContainer" containerID="fcfc5ee368ea577dfb6d8d113befa0a945617b1747099f6f7f3683900569ce93" Oct 31 00:42:21.356705 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:42:21.356683 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fcfc5ee368ea577dfb6d8d113befa0a945617b1747099f6f7f3683900569ce93\": container with ID starting with fcfc5ee368ea577dfb6d8d113befa0a945617b1747099f6f7f3683900569ce93 not found: ID does not exist" containerID="fcfc5ee368ea577dfb6d8d113befa0a945617b1747099f6f7f3683900569ce93" Oct 31 00:42:21.356771 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:42:21.356712 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fcfc5ee368ea577dfb6d8d113befa0a945617b1747099f6f7f3683900569ce93"} err="failed to get container status \"fcfc5ee368ea577dfb6d8d113befa0a945617b1747099f6f7f3683900569ce93\": rpc error: code = NotFound desc = could not find container \"fcfc5ee368ea577dfb6d8d113befa0a945617b1747099f6f7f3683900569ce93\": container with ID starting with fcfc5ee368ea577dfb6d8d113befa0a945617b1747099f6f7f3683900569ce93 not found: ID does not exist" Oct 31 00:42:21.356771 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:42:21.356729 4927 scope.go:122] "RemoveContainer" containerID="1e976479aec8fe79960a4e1399f8f64751cbb2b2191396ec1795f6741965a2b9" Oct 31 00:42:21.356977 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:42:21.356960 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e976479aec8fe79960a4e1399f8f64751cbb2b2191396ec1795f6741965a2b9\": container with ID starting with 1e976479aec8fe79960a4e1399f8f64751cbb2b2191396ec1795f6741965a2b9 not found: ID does not exist" containerID="1e976479aec8fe79960a4e1399f8f64751cbb2b2191396ec1795f6741965a2b9" Oct 31 00:42:21.357026 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:42:21.356983 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e976479aec8fe79960a4e1399f8f64751cbb2b2191396ec1795f6741965a2b9"} err="failed to get container status \"1e976479aec8fe79960a4e1399f8f64751cbb2b2191396ec1795f6741965a2b9\": rpc error: code = NotFound desc = could not find container \"1e976479aec8fe79960a4e1399f8f64751cbb2b2191396ec1795f6741965a2b9\": container with ID starting with 1e976479aec8fe79960a4e1399f8f64751cbb2b2191396ec1795f6741965a2b9 not found: ID does not exist" Oct 31 00:42:21.504430 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:42:21.504361 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-0e6572-openshift-default-5c475f9cbf97qjl"] Oct 31 00:42:21.510078 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:42:21.510056 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-0e6572-openshift-default-5c475f9cbf97qjl"] Oct 31 00:42:23.086400 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:42:23.086364 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d768f7f8-eb55-4419-802a-9a4f2e6e57b8" path="/var/lib/kubelet/pods/d768f7f8-eb55-4419-802a-9a4f2e6e57b8/volumes" Oct 31 00:42:27.394670 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:42:27.394635 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:42:28.049675 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:42:28.049642 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-7csdn"] Oct 31 00:42:28.100471 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:42:28.100434 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-7csdn"] Oct 31 00:42:29.085518 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:42:29.085481 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e2f5f51f-6279-4993-960e-46bdceb23e84" path="/var/lib/kubelet/pods/e2f5f51f-6279-4993-960e-46bdceb23e84/volumes" Oct 31 00:42:33.789211 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:42:33.789173 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:42:33.999179 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:42:33.999147 4927 scope.go:122] "RemoveContainer" containerID="0dd237469a44b2255386c7e6d036c069ca460d1caa6b2c02b745c46e465460bb" Oct 31 00:42:34.036354 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:42:34.033492 4927 scope.go:122] "RemoveContainer" containerID="6893c10707da6b8a4066ef0bf66e7f4891096d87adcb123b53ec280ed4002809" Oct 31 00:42:34.088127 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:42:34.088101 4927 scope.go:122] "RemoveContainer" containerID="466122ee70bc030cc0e6947e1986710968a71186a7a6b25ec4f25f193b09e3d7" Oct 31 00:42:34.821779 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:42:34.821743 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:42:39.197484 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:42:39.197451 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:42:39.351594 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:42:39.351557 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:42:40.726295 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:42:40.726234 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:42:49.677654 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:42:49.677615 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sqv69" containerName="extractor" Oct 31 00:42:51.052464 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:42:51.052420 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-bgmzf"] Oct 31 00:42:51.103254 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:42:51.103220 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-bgmzf"] Oct 31 00:42:52.101656 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:42:52.101620 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-fvgn8"] Oct 31 00:42:52.102064 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:42:52.101800 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-fvgn8" podUID="89673660-8db3-4428-90cb-0ce72cf178e3" containerName="manager" containerID="cri-o://4284f55772c235305bf2dd1f4c18bfb6e4db105b551f1b96c4970d9d9843979d" gracePeriod=10 Oct 31 00:42:52.359921 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:42:52.359862 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-fvgn8" Oct 31 00:42:52.473183 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:42:52.473149 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89673660-8db3-4428-90cb-0ce72cf178e3-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/89673660-8db3-4428-90cb-0ce72cf178e3-extensions-socket-volume\") pod \"89673660-8db3-4428-90cb-0ce72cf178e3\" (UID: \"89673660-8db3-4428-90cb-0ce72cf178e3\") " Oct 31 00:42:52.473374 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:42:52.473224 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89673660-8db3-4428-90cb-0ce72cf178e3-kube-api-access-q2krb\" (UniqueName: \"kubernetes.io/projected/89673660-8db3-4428-90cb-0ce72cf178e3-kube-api-access-q2krb\") pod \"89673660-8db3-4428-90cb-0ce72cf178e3\" (UID: \"89673660-8db3-4428-90cb-0ce72cf178e3\") " Oct 31 00:42:52.473563 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:42:52.473538 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89673660-8db3-4428-90cb-0ce72cf178e3-extensions-socket-volume" pod "89673660-8db3-4428-90cb-0ce72cf178e3" (UID: "89673660-8db3-4428-90cb-0ce72cf178e3"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:42:52.475297 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:42:52.475276 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89673660-8db3-4428-90cb-0ce72cf178e3-kube-api-access-q2krb" pod "89673660-8db3-4428-90cb-0ce72cf178e3" (UID: "89673660-8db3-4428-90cb-0ce72cf178e3"). InnerVolumeSpecName "kube-api-access-q2krb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:42:52.545128 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:42:52.545076 4927 generic.go:356] "Generic (PLEG): container finished" podID="89673660-8db3-4428-90cb-0ce72cf178e3" containerID="4284f55772c235305bf2dd1f4c18bfb6e4db105b551f1b96c4970d9d9843979d" exitCode=0 Oct 31 00:42:52.545301 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:42:52.545197 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-fvgn8" Oct 31 00:42:52.545301 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:42:52.545217 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-fvgn8" event={"ID":"89673660-8db3-4428-90cb-0ce72cf178e3","Type":"ContainerDied","Data":"4284f55772c235305bf2dd1f4c18bfb6e4db105b551f1b96c4970d9d9843979d"} Oct 31 00:42:52.545416 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:42:52.545310 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-fvgn8" event={"ID":"89673660-8db3-4428-90cb-0ce72cf178e3","Type":"ContainerDied","Data":"d72434e48fa34140748dda722de3b56348f76599c171dc6bba651a476ac4bb99"} Oct 31 00:42:52.545416 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:42:52.545354 4927 scope.go:122] "RemoveContainer" containerID="4284f55772c235305bf2dd1f4c18bfb6e4db105b551f1b96c4970d9d9843979d" Oct 31 00:42:52.557049 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:42:52.557029 4927 scope.go:122] "RemoveContainer" containerID="4284f55772c235305bf2dd1f4c18bfb6e4db105b551f1b96c4970d9d9843979d" Oct 31 00:42:52.557465 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:42:52.557437 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4284f55772c235305bf2dd1f4c18bfb6e4db105b551f1b96c4970d9d9843979d\": container with ID starting with 4284f55772c235305bf2dd1f4c18bfb6e4db105b551f1b96c4970d9d9843979d not found: ID does not exist" containerID="4284f55772c235305bf2dd1f4c18bfb6e4db105b551f1b96c4970d9d9843979d" Oct 31 00:42:52.557572 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:42:52.557476 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4284f55772c235305bf2dd1f4c18bfb6e4db105b551f1b96c4970d9d9843979d"} err="failed to get container status \"4284f55772c235305bf2dd1f4c18bfb6e4db105b551f1b96c4970d9d9843979d\": rpc error: code = NotFound desc = could not find container \"4284f55772c235305bf2dd1f4c18bfb6e4db105b551f1b96c4970d9d9843979d\": container with ID starting with 4284f55772c235305bf2dd1f4c18bfb6e4db105b551f1b96c4970d9d9843979d not found: ID does not exist" Oct 31 00:42:52.574574 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:42:52.574537 4927 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/89673660-8db3-4428-90cb-0ce72cf178e3-extensions-socket-volume\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:42:52.574574 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:42:52.574566 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q2krb\" (UniqueName: \"kubernetes.io/projected/89673660-8db3-4428-90cb-0ce72cf178e3-kube-api-access-q2krb\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:42:52.716798 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:42:52.716766 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-fvgn8"] Oct 31 00:42:52.720423 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:42:52.720395 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-fvgn8"] Oct 31 00:42:53.085170 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:42:53.085086 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="89673660-8db3-4428-90cb-0ce72cf178e3" path="/var/lib/kubelet/pods/89673660-8db3-4428-90cb-0ce72cf178e3/volumes" Oct 31 00:42:53.085552 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:42:53.085537 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d4544ba9-5754-41ef-bb34-e17fc37d3502" path="/var/lib/kubelet/pods/d4544ba9-5754-41ef-bb34-e17fc37d3502/volumes" Oct 31 00:43:19.676878 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:43:19.676840 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sqv69" containerName="extractor" Oct 31 00:43:34.198844 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:43:34.198806 4927 scope.go:122] "RemoveContainer" containerID="0f008993cd63c3c2d3586b24801da0de284bff0480cd5963bf7411e9472ce76c" Oct 31 00:43:34.208122 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:43:34.208095 4927 scope.go:122] "RemoveContainer" containerID="93fb83fa0b8a20b7fdbec018b78496c7c4761fa731cdeee28d7dcce8514910dc" Oct 31 00:43:39.559594 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:43:39.559402 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:49.678149 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:43:49.678105 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sqv69" containerName="extractor" Oct 31 00:43:55.636121 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:43:55.636084 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-v7w7j"] Oct 31 00:43:55.636569 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:43:55.636507 4927 cpu_manager.go:414] "containerMap: removing container" podUID="89673660-8db3-4428-90cb-0ce72cf178e3" containerName="manager" Oct 31 00:43:55.636569 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:43:55.636519 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89673660-8db3-4428-90cb-0ce72cf178e3" containerName="manager" Oct 31 00:43:55.636569 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:43:55.636529 4927 cpu_manager.go:414] "containerMap: removing container" podUID="d768f7f8-eb55-4419-802a-9a4f2e6e57b8" containerName="istio-proxy" Oct 31 00:43:55.636569 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:43:55.636535 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d768f7f8-eb55-4419-802a-9a4f2e6e57b8" containerName="istio-proxy" Oct 31 00:43:55.636569 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:43:55.636548 4927 cpu_manager.go:414] "containerMap: removing container" podUID="d768f7f8-eb55-4419-802a-9a4f2e6e57b8" containerName="istio-proxy" Oct 31 00:43:55.636569 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:43:55.636554 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d768f7f8-eb55-4419-802a-9a4f2e6e57b8" containerName="istio-proxy" Oct 31 00:43:55.636806 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:43:55.636578 4927 cpu_manager.go:414] "containerMap: removing container" podUID="ff7abf47-1bc8-47ec-92ef-33e82aeb1e92" containerName="revoke-keys" Oct 31 00:43:55.636806 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:43:55.636583 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff7abf47-1bc8-47ec-92ef-33e82aeb1e92" containerName="revoke-keys" Oct 31 00:43:55.636806 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:43:55.636643 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="d768f7f8-eb55-4419-802a-9a4f2e6e57b8" containerName="istio-proxy" Oct 31 00:43:55.636806 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:43:55.636655 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff7abf47-1bc8-47ec-92ef-33e82aeb1e92" containerName="revoke-keys" Oct 31 00:43:55.636806 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:43:55.636662 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="89673660-8db3-4428-90cb-0ce72cf178e3" containerName="manager" Oct 31 00:43:55.641317 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:43:55.641297 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-v7w7j" Oct 31 00:43:55.654144 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:43:55.654118 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/98cf959f-bca4-41b2-aa33-51b8f3b3e117-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-v7w7j\" (UID: \"98cf959f-bca4-41b2-aa33-51b8f3b3e117\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-v7w7j" Oct 31 00:43:55.654267 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:43:55.654238 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jn5pp\" (UniqueName: \"kubernetes.io/projected/98cf959f-bca4-41b2-aa33-51b8f3b3e117-kube-api-access-jn5pp\") pod \"kuadrant-operator-controller-manager-678d87766f-v7w7j\" (UID: \"98cf959f-bca4-41b2-aa33-51b8f3b3e117\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-v7w7j" Oct 31 00:43:55.686793 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:43:55.686760 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-v7w7j"] Oct 31 00:43:55.734770 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:43:55.734740 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-7l4tw\"" Oct 31 00:43:55.755164 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:43:55.755134 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/98cf959f-bca4-41b2-aa33-51b8f3b3e117-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-v7w7j\" (UID: \"98cf959f-bca4-41b2-aa33-51b8f3b3e117\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-v7w7j" Oct 31 00:43:55.755281 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:43:55.755184 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jn5pp\" (UniqueName: \"kubernetes.io/projected/98cf959f-bca4-41b2-aa33-51b8f3b3e117-kube-api-access-jn5pp\") pod \"kuadrant-operator-controller-manager-678d87766f-v7w7j\" (UID: \"98cf959f-bca4-41b2-aa33-51b8f3b3e117\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-v7w7j" Oct 31 00:43:55.755544 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:43:55.755525 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/98cf959f-bca4-41b2-aa33-51b8f3b3e117-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-v7w7j\" (UID: \"98cf959f-bca4-41b2-aa33-51b8f3b3e117\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-v7w7j" Oct 31 00:43:55.818307 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:43:55.818274 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jn5pp\" (UniqueName: \"kubernetes.io/projected/98cf959f-bca4-41b2-aa33-51b8f3b3e117-kube-api-access-jn5pp\") pod \"kuadrant-operator-controller-manager-678d87766f-v7w7j\" (UID: \"98cf959f-bca4-41b2-aa33-51b8f3b3e117\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-v7w7j" Oct 31 00:43:55.954409 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:43:55.954372 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-v7w7j" Oct 31 00:43:56.436425 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:43:56.436397 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-v7w7j"] Oct 31 00:43:56.438776 ip-10-0-141-180 kubenswrapper[4927]: W1031 00:43:56.438745 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98cf959f_bca4_41b2_aa33_51b8f3b3e117.slice/crio-a70061d51f6135d32928855439db57786697b65df28d64f282fb887da7aabd2e WatchSource:0}: Error finding container a70061d51f6135d32928855439db57786697b65df28d64f282fb887da7aabd2e: Status 404 returned error can't find the container with id a70061d51f6135d32928855439db57786697b65df28d64f282fb887da7aabd2e Oct 31 00:43:56.916430 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:43:56.916389 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-v7w7j" event={"ID":"98cf959f-bca4-41b2-aa33-51b8f3b3e117","Type":"ContainerStarted","Data":"224c71f8f5660ff5855c62ba366e56dd293b907cd9a861029bd54b1e45e06a14"} Oct 31 00:43:56.916858 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:43:56.916454 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-v7w7j" event={"ID":"98cf959f-bca4-41b2-aa33-51b8f3b3e117","Type":"ContainerStarted","Data":"a70061d51f6135d32928855439db57786697b65df28d64f282fb887da7aabd2e"} Oct 31 00:43:56.916858 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:43:56.916597 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-v7w7j" Oct 31 00:43:57.092140 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:43:57.092085 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-v7w7j" podStartSLOduration=2.09207166 podStartE2EDuration="2.09207166s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:43:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:43:57.091412139 +0000 UTC m=+3750.127742961" watchObservedRunningTime="2026-10-31 00:43:57.09207166 +0000 UTC m=+3750.128402482" Oct 31 00:44:07.924971 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:44:07.924937 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-v7w7j" Oct 31 00:44:19.677712 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:44:19.677667 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sqv69" containerName="extractor" Oct 31 00:44:20.699910 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:44:20.699873 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-mr9m7"] Oct 31 00:44:20.700425 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:44:20.700411 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="d768f7f8-eb55-4419-802a-9a4f2e6e57b8" containerName="istio-proxy" Oct 31 00:44:20.703765 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:44:20.703749 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-mr9m7" Oct 31 00:44:20.758273 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:44:20.758243 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-mr9m7"] Oct 31 00:44:20.806412 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:44:20.806374 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:44:20.806587 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:44:20.806419 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4fdzs\"" Oct 31 00:44:20.816838 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:44:20.816807 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6ffaa04-f498-4957-b1fb-e5d7ab62c1be-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-mr9m7\" (UID: \"c6ffaa04-f498-4957-b1fb-e5d7ab62c1be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-mr9m7" Oct 31 00:44:20.817060 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:44:20.816855 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sh26z\" (UniqueName: \"kubernetes.io/projected/c6ffaa04-f498-4957-b1fb-e5d7ab62c1be-kube-api-access-sh26z\") pod \"maas-api-revoke-sub-d576e68b-mr9m7\" (UID: \"c6ffaa04-f498-4957-b1fb-e5d7ab62c1be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-mr9m7" Oct 31 00:44:20.817060 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:44:20.816949 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6ffaa04-f498-4957-b1fb-e5d7ab62c1be-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-mr9m7\" (UID: \"c6ffaa04-f498-4957-b1fb-e5d7ab62c1be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-mr9m7" Oct 31 00:44:20.917771 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:44:20.917732 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6ffaa04-f498-4957-b1fb-e5d7ab62c1be-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-mr9m7\" (UID: \"c6ffaa04-f498-4957-b1fb-e5d7ab62c1be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-mr9m7" Oct 31 00:44:20.917968 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:44:20.917794 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sh26z\" (UniqueName: \"kubernetes.io/projected/c6ffaa04-f498-4957-b1fb-e5d7ab62c1be-kube-api-access-sh26z\") pod \"maas-api-revoke-sub-d576e68b-mr9m7\" (UID: \"c6ffaa04-f498-4957-b1fb-e5d7ab62c1be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-mr9m7" Oct 31 00:44:20.917968 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:44:20.917829 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6ffaa04-f498-4957-b1fb-e5d7ab62c1be-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-mr9m7\" (UID: \"c6ffaa04-f498-4957-b1fb-e5d7ab62c1be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-mr9m7" Oct 31 00:44:20.918408 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:44:20.918383 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6ffaa04-f498-4957-b1fb-e5d7ab62c1be-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-mr9m7\" (UID: \"c6ffaa04-f498-4957-b1fb-e5d7ab62c1be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-mr9m7" Oct 31 00:44:20.920465 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:44:20.920440 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6ffaa04-f498-4957-b1fb-e5d7ab62c1be-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-mr9m7\" (UID: \"c6ffaa04-f498-4957-b1fb-e5d7ab62c1be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-mr9m7" Oct 31 00:44:20.980849 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:44:20.980772 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sh26z\" (UniqueName: \"kubernetes.io/projected/c6ffaa04-f498-4957-b1fb-e5d7ab62c1be-kube-api-access-sh26z\") pod \"maas-api-revoke-sub-d576e68b-mr9m7\" (UID: \"c6ffaa04-f498-4957-b1fb-e5d7ab62c1be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-mr9m7" Oct 31 00:44:21.017730 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:44:21.017692 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-mr9m7" Oct 31 00:44:21.397526 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:44:21.397495 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-mr9m7"] Oct 31 00:44:21.399405 ip-10-0-141-180 kubenswrapper[4927]: W1031 00:44:21.399372 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6ffaa04_f498_4957_b1fb_e5d7ab62c1be.slice/crio-98b4934d69ae7a131d718ba9af74c0c95a9599f7a5b8fea993b42077bf2c0b08 WatchSource:0}: Error finding container 98b4934d69ae7a131d718ba9af74c0c95a9599f7a5b8fea993b42077bf2c0b08: Status 404 returned error can't find the container with id 98b4934d69ae7a131d718ba9af74c0c95a9599f7a5b8fea993b42077bf2c0b08 Oct 31 00:44:22.099194 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:44:22.099097 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-mr9m7" event={"ID":"c6ffaa04-f498-4957-b1fb-e5d7ab62c1be","Type":"ContainerStarted","Data":"0b511dbef5f408064d1fa907830655d0382d4efc7bcd2809b6f187658c07a94b"} Oct 31 00:44:22.099194 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:44:22.099156 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-mr9m7" event={"ID":"c6ffaa04-f498-4957-b1fb-e5d7ab62c1be","Type":"ContainerStarted","Data":"98b4934d69ae7a131d718ba9af74c0c95a9599f7a5b8fea993b42077bf2c0b08"} Oct 31 00:44:22.218682 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:44:22.218630 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-mr9m7" podStartSLOduration=2.218616096 podStartE2EDuration="2.218616096s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:44:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:44:22.21651064 +0000 UTC m=+3775.252841475" watchObservedRunningTime="2026-10-31 00:44:22.218616096 +0000 UTC m=+3775.254946922" Oct 31 00:44:23.106810 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:44:23.106771 4927 generic.go:356] "Generic (PLEG): container finished" podID="c6ffaa04-f498-4957-b1fb-e5d7ab62c1be" containerID="0b511dbef5f408064d1fa907830655d0382d4efc7bcd2809b6f187658c07a94b" exitCode=0 Oct 31 00:44:23.107222 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:44:23.106861 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-mr9m7" event={"ID":"c6ffaa04-f498-4957-b1fb-e5d7ab62c1be","Type":"ContainerDied","Data":"0b511dbef5f408064d1fa907830655d0382d4efc7bcd2809b6f187658c07a94b"} Oct 31 00:44:24.253780 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:44:24.253751 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-mr9m7" Oct 31 00:44:24.355028 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:44:24.354993 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c6ffaa04-f498-4957-b1fb-e5d7ab62c1be-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6ffaa04-f498-4957-b1fb-e5d7ab62c1be-maas-api-xks-ca\") pod \"c6ffaa04-f498-4957-b1fb-e5d7ab62c1be\" (UID: \"c6ffaa04-f498-4957-b1fb-e5d7ab62c1be\") " Oct 31 00:44:24.355210 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:44:24.355048 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6ffaa04-f498-4957-b1fb-e5d7ab62c1be-kube-api-access-sh26z\" (UniqueName: \"kubernetes.io/projected/c6ffaa04-f498-4957-b1fb-e5d7ab62c1be-kube-api-access-sh26z\") pod \"c6ffaa04-f498-4957-b1fb-e5d7ab62c1be\" (UID: \"c6ffaa04-f498-4957-b1fb-e5d7ab62c1be\") " Oct 31 00:44:24.355210 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:44:24.355201 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c6ffaa04-f498-4957-b1fb-e5d7ab62c1be-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6ffaa04-f498-4957-b1fb-e5d7ab62c1be-maas-api-openshift-ca\") pod \"c6ffaa04-f498-4957-b1fb-e5d7ab62c1be\" (UID: \"c6ffaa04-f498-4957-b1fb-e5d7ab62c1be\") " Oct 31 00:44:24.355589 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:44:24.355565 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6ffaa04-f498-4957-b1fb-e5d7ab62c1be-maas-api-openshift-ca" pod "c6ffaa04-f498-4957-b1fb-e5d7ab62c1be" (UID: "c6ffaa04-f498-4957-b1fb-e5d7ab62c1be"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:44:24.357169 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:44:24.357145 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6ffaa04-f498-4957-b1fb-e5d7ab62c1be-maas-api-xks-ca" pod "c6ffaa04-f498-4957-b1fb-e5d7ab62c1be" (UID: "c6ffaa04-f498-4957-b1fb-e5d7ab62c1be"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:44:24.357265 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:44:24.357222 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6ffaa04-f498-4957-b1fb-e5d7ab62c1be-kube-api-access-sh26z" pod "c6ffaa04-f498-4957-b1fb-e5d7ab62c1be" (UID: "c6ffaa04-f498-4957-b1fb-e5d7ab62c1be"). InnerVolumeSpecName "kube-api-access-sh26z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:44:24.456483 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:44:24.456446 4927 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6ffaa04-f498-4957-b1fb-e5d7ab62c1be-maas-api-xks-ca\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:44:24.456483 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:44:24.456478 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sh26z\" (UniqueName: \"kubernetes.io/projected/c6ffaa04-f498-4957-b1fb-e5d7ab62c1be-kube-api-access-sh26z\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:44:24.456483 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:44:24.456489 4927 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6ffaa04-f498-4957-b1fb-e5d7ab62c1be-maas-api-openshift-ca\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:44:25.119443 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:44:25.119412 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-mr9m7" Oct 31 00:44:25.119621 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:44:25.119404 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-mr9m7" event={"ID":"c6ffaa04-f498-4957-b1fb-e5d7ab62c1be","Type":"ContainerDied","Data":"98b4934d69ae7a131d718ba9af74c0c95a9599f7a5b8fea993b42077bf2c0b08"} Oct 31 00:44:25.119621 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:44:25.119546 4927 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="98b4934d69ae7a131d718ba9af74c0c95a9599f7a5b8fea993b42077bf2c0b08" Oct 31 00:44:27.050446 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:44:27.050406 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-qcrvb"] Oct 31 00:44:27.103573 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:44:27.103539 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-qcrvb"] Oct 31 00:44:27.134066 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:44:27.134012 4927 status_manager.go:1045] "Failed to get status for pod" err="pods \"maas-api-revoke-sub-e2fc9c5b-qcrvb\" is forbidden: User \"system:node:ip-10-0-141-180.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"odh-ai-gateway-infra\": no relationship found between node 'ip-10-0-141-180.us-west-2.compute.internal' and this object" podUID="00356130-32fb-4b92-9952-25d0466e4415" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-qcrvb" Oct 31 00:44:27.414675 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:44:27.414644 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:44:27.610378 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:44:27.610320 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:44:29.084579 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:44:29.084536 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="00356130-32fb-4b92-9952-25d0466e4415" path="/var/lib/kubelet/pods/00356130-32fb-4b92-9952-25d0466e4415/volumes" Oct 31 00:44:34.267118 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:44:34.267080 4927 scope.go:122] "RemoveContainer" containerID="347cbe9b221acbe24700340924cf6ecdc1655bd9d0f6a6318369144b37400a9c" Oct 31 00:44:34.276405 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:44:34.276384 4927 scope.go:122] "RemoveContainer" containerID="19adbe013cb8aae63119b1abb54309837b022aa34414fa9414977044dd8ef5cd" Oct 31 00:44:34.284503 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:44:34.284481 4927 scope.go:122] "RemoveContainer" containerID="540a283d32b798e41c28614bc6c1aee1823d3d81a0ff8db9179ba2bdf7e6a6e1" Oct 31 00:44:38.049787 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:44:38.049746 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-dqf9g"] Oct 31 00:44:38.100967 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:44:38.100930 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-dqf9g"] Oct 31 00:44:39.084632 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:44:39.084597 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3189e4b8-6b86-4499-abce-f27620456397" path="/var/lib/kubelet/pods/3189e4b8-6b86-4499-abce-f27620456397/volumes" Oct 31 00:44:39.931703 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:44:39.931665 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:44:40.272591 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:44:40.272505 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:44:49.678776 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:44:49.678736 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sqv69" containerName="extractor" Oct 31 00:44:49.827326 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:44:49.827288 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-7zlnn"] Oct 31 00:44:49.827983 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:44:49.827964 4927 cpu_manager.go:414] "containerMap: removing container" podUID="c6ffaa04-f498-4957-b1fb-e5d7ab62c1be" containerName="revoke-keys" Oct 31 00:44:49.828044 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:44:49.827987 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6ffaa04-f498-4957-b1fb-e5d7ab62c1be" containerName="revoke-keys" Oct 31 00:44:49.828097 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:44:49.828085 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6ffaa04-f498-4957-b1fb-e5d7ab62c1be" containerName="revoke-keys" Oct 31 00:44:49.833250 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:44:49.833223 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-7zlnn" Oct 31 00:44:49.867509 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:44:49.867478 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/767625ba-836a-4aad-9960-5ed7bf134250-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-7zlnn\" (UID: \"767625ba-836a-4aad-9960-5ed7bf134250\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-7zlnn" Oct 31 00:44:49.867669 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:44:49.867517 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/767625ba-836a-4aad-9960-5ed7bf134250-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-7zlnn\" (UID: \"767625ba-836a-4aad-9960-5ed7bf134250\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-7zlnn" Oct 31 00:44:49.867669 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:44:49.867602 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h78fw\" (UniqueName: \"kubernetes.io/projected/767625ba-836a-4aad-9960-5ed7bf134250-kube-api-access-h78fw\") pod \"maas-api-revoke-sub-d2bc93bc-7zlnn\" (UID: \"767625ba-836a-4aad-9960-5ed7bf134250\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-7zlnn" Oct 31 00:44:49.885420 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:44:49.885381 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-7zlnn"] Oct 31 00:44:49.932303 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:44:49.932207 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:44:49.932303 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:44:49.932285 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4fdzs\"" Oct 31 00:44:49.968977 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:44:49.968936 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/767625ba-836a-4aad-9960-5ed7bf134250-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-7zlnn\" (UID: \"767625ba-836a-4aad-9960-5ed7bf134250\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-7zlnn" Oct 31 00:44:49.969184 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:44:49.968984 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/767625ba-836a-4aad-9960-5ed7bf134250-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-7zlnn\" (UID: \"767625ba-836a-4aad-9960-5ed7bf134250\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-7zlnn" Oct 31 00:44:49.969184 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:44:49.969018 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h78fw\" (UniqueName: \"kubernetes.io/projected/767625ba-836a-4aad-9960-5ed7bf134250-kube-api-access-h78fw\") pod \"maas-api-revoke-sub-d2bc93bc-7zlnn\" (UID: \"767625ba-836a-4aad-9960-5ed7bf134250\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-7zlnn" Oct 31 00:44:49.969760 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:44:49.969737 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/767625ba-836a-4aad-9960-5ed7bf134250-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-7zlnn\" (UID: \"767625ba-836a-4aad-9960-5ed7bf134250\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-7zlnn" Oct 31 00:44:49.971576 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:44:49.971552 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/767625ba-836a-4aad-9960-5ed7bf134250-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-7zlnn\" (UID: \"767625ba-836a-4aad-9960-5ed7bf134250\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-7zlnn" Oct 31 00:44:50.031493 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:44:50.031453 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h78fw\" (UniqueName: \"kubernetes.io/projected/767625ba-836a-4aad-9960-5ed7bf134250-kube-api-access-h78fw\") pod \"maas-api-revoke-sub-d2bc93bc-7zlnn\" (UID: \"767625ba-836a-4aad-9960-5ed7bf134250\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-7zlnn" Oct 31 00:44:50.146062 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:44:50.146028 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-7zlnn" Oct 31 00:44:50.520789 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:44:50.520758 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-7zlnn"] Oct 31 00:44:50.521990 ip-10-0-141-180 kubenswrapper[4927]: W1031 00:44:50.521967 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod767625ba_836a_4aad_9960_5ed7bf134250.slice/crio-a7e27ba0bb5d273e8e8130e65a092e311e7aa9317a26de3f00a29274166cd0fb WatchSource:0}: Error finding container a7e27ba0bb5d273e8e8130e65a092e311e7aa9317a26de3f00a29274166cd0fb: Status 404 returned error can't find the container with id a7e27ba0bb5d273e8e8130e65a092e311e7aa9317a26de3f00a29274166cd0fb Oct 31 00:44:51.307351 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:44:51.307301 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-7zlnn" event={"ID":"767625ba-836a-4aad-9960-5ed7bf134250","Type":"ContainerStarted","Data":"631f9133d79396e5aa3435bc0161c3b926da07feb3471026d66d30dd8b54c59f"} Oct 31 00:44:51.307805 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:44:51.307410 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-7zlnn" event={"ID":"767625ba-836a-4aad-9960-5ed7bf134250","Type":"ContainerStarted","Data":"a7e27ba0bb5d273e8e8130e65a092e311e7aa9317a26de3f00a29274166cd0fb"} Oct 31 00:44:51.426609 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:44:51.426556 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-7zlnn" podStartSLOduration=2.426543337 podStartE2EDuration="2.426543337s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:44:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:44:51.424899349 +0000 UTC m=+3804.461230173" watchObservedRunningTime="2026-10-31 00:44:51.426543337 +0000 UTC m=+3804.462874158" Oct 31 00:44:52.314952 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:44:52.314918 4927 generic.go:356] "Generic (PLEG): container finished" podID="767625ba-836a-4aad-9960-5ed7bf134250" containerID="631f9133d79396e5aa3435bc0161c3b926da07feb3471026d66d30dd8b54c59f" exitCode=0 Oct 31 00:44:52.315367 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:44:52.314998 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-7zlnn" event={"ID":"767625ba-836a-4aad-9960-5ed7bf134250","Type":"ContainerDied","Data":"631f9133d79396e5aa3435bc0161c3b926da07feb3471026d66d30dd8b54c59f"} Oct 31 00:44:53.459502 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:44:53.459474 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-7zlnn" Oct 31 00:44:53.501590 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:44:53.501561 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/767625ba-836a-4aad-9960-5ed7bf134250-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/767625ba-836a-4aad-9960-5ed7bf134250-maas-api-xks-ca\") pod \"767625ba-836a-4aad-9960-5ed7bf134250\" (UID: \"767625ba-836a-4aad-9960-5ed7bf134250\") " Oct 31 00:44:53.501695 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:44:53.501641 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/767625ba-836a-4aad-9960-5ed7bf134250-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/767625ba-836a-4aad-9960-5ed7bf134250-maas-api-openshift-ca\") pod \"767625ba-836a-4aad-9960-5ed7bf134250\" (UID: \"767625ba-836a-4aad-9960-5ed7bf134250\") " Oct 31 00:44:53.501793 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:44:53.501776 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/767625ba-836a-4aad-9960-5ed7bf134250-kube-api-access-h78fw\" (UniqueName: \"kubernetes.io/projected/767625ba-836a-4aad-9960-5ed7bf134250-kube-api-access-h78fw\") pod \"767625ba-836a-4aad-9960-5ed7bf134250\" (UID: \"767625ba-836a-4aad-9960-5ed7bf134250\") " Oct 31 00:44:53.502006 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:44:53.501977 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/767625ba-836a-4aad-9960-5ed7bf134250-maas-api-openshift-ca" pod "767625ba-836a-4aad-9960-5ed7bf134250" (UID: "767625ba-836a-4aad-9960-5ed7bf134250"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:44:53.502117 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:44:53.502098 4927 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/767625ba-836a-4aad-9960-5ed7bf134250-maas-api-openshift-ca\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:44:53.503792 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:44:53.503764 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/767625ba-836a-4aad-9960-5ed7bf134250-maas-api-xks-ca" pod "767625ba-836a-4aad-9960-5ed7bf134250" (UID: "767625ba-836a-4aad-9960-5ed7bf134250"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:44:53.503884 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:44:53.503813 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/767625ba-836a-4aad-9960-5ed7bf134250-kube-api-access-h78fw" pod "767625ba-836a-4aad-9960-5ed7bf134250" (UID: "767625ba-836a-4aad-9960-5ed7bf134250"). InnerVolumeSpecName "kube-api-access-h78fw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:44:53.603352 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:44:53.603249 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h78fw\" (UniqueName: \"kubernetes.io/projected/767625ba-836a-4aad-9960-5ed7bf134250-kube-api-access-h78fw\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:44:53.603352 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:44:53.603287 4927 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/767625ba-836a-4aad-9960-5ed7bf134250-maas-api-xks-ca\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:44:54.329464 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:44:54.329429 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-7zlnn" Oct 31 00:44:54.329637 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:44:54.329435 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-7zlnn" event={"ID":"767625ba-836a-4aad-9960-5ed7bf134250","Type":"ContainerDied","Data":"a7e27ba0bb5d273e8e8130e65a092e311e7aa9317a26de3f00a29274166cd0fb"} Oct 31 00:44:54.329637 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:44:54.329566 4927 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a7e27ba0bb5d273e8e8130e65a092e311e7aa9317a26de3f00a29274166cd0fb" Oct 31 00:45:00.162550 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:45:00.162514 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890125-4twzz"] Oct 31 00:45:00.162949 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:45:00.162935 4927 cpu_manager.go:414] "containerMap: removing container" podUID="767625ba-836a-4aad-9960-5ed7bf134250" containerName="revoke-keys" Oct 31 00:45:00.163000 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:45:00.162949 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="767625ba-836a-4aad-9960-5ed7bf134250" containerName="revoke-keys" Oct 31 00:45:00.163051 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:45:00.163042 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="767625ba-836a-4aad-9960-5ed7bf134250" containerName="revoke-keys" Oct 31 00:45:00.166462 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:45:00.166443 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890125-4twzz" Oct 31 00:45:00.219673 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:45:00.219618 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890125-4twzz"] Oct 31 00:45:00.267984 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:45:00.267951 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4fdzs\"" Oct 31 00:45:00.268224 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:45:00.268196 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/794a46cb-f783-4a10-806e-b1fe3fb28c1b-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890125-4twzz\" (UID: \"794a46cb-f783-4a10-806e-b1fe3fb28c1b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890125-4twzz" Oct 31 00:45:00.268384 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:45:00.268363 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmrpm\" (UniqueName: \"kubernetes.io/projected/794a46cb-f783-4a10-806e-b1fe3fb28c1b-kube-api-access-mmrpm\") pod \"maas-api-key-cleanup-29890125-4twzz\" (UID: \"794a46cb-f783-4a10-806e-b1fe3fb28c1b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890125-4twzz" Oct 31 00:45:00.369345 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:45:00.369299 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmrpm\" (UniqueName: \"kubernetes.io/projected/794a46cb-f783-4a10-806e-b1fe3fb28c1b-kube-api-access-mmrpm\") pod \"maas-api-key-cleanup-29890125-4twzz\" (UID: \"794a46cb-f783-4a10-806e-b1fe3fb28c1b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890125-4twzz" Oct 31 00:45:00.369524 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:45:00.369406 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/794a46cb-f783-4a10-806e-b1fe3fb28c1b-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890125-4twzz\" (UID: \"794a46cb-f783-4a10-806e-b1fe3fb28c1b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890125-4twzz" Oct 31 00:45:00.370081 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:45:00.370061 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/794a46cb-f783-4a10-806e-b1fe3fb28c1b-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890125-4twzz\" (UID: \"794a46cb-f783-4a10-806e-b1fe3fb28c1b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890125-4twzz" Oct 31 00:45:00.431611 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:45:00.431537 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmrpm\" (UniqueName: \"kubernetes.io/projected/794a46cb-f783-4a10-806e-b1fe3fb28c1b-kube-api-access-mmrpm\") pod \"maas-api-key-cleanup-29890125-4twzz\" (UID: \"794a46cb-f783-4a10-806e-b1fe3fb28c1b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890125-4twzz" Oct 31 00:45:00.480106 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:45:00.480076 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890125-4twzz" Oct 31 00:45:00.855351 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:45:00.855305 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890125-4twzz"] Oct 31 00:45:00.856475 ip-10-0-141-180 kubenswrapper[4927]: W1031 00:45:00.856447 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod794a46cb_f783_4a10_806e_b1fe3fb28c1b.slice/crio-b502560c036410c5559fda069f5602e14bc8677d1b044b1a48c3d1ff966586a1 WatchSource:0}: Error finding container b502560c036410c5559fda069f5602e14bc8677d1b044b1a48c3d1ff966586a1: Status 404 returned error can't find the container with id b502560c036410c5559fda069f5602e14bc8677d1b044b1a48c3d1ff966586a1 Oct 31 00:45:01.372120 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:45:01.372025 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890125-4twzz" event={"ID":"794a46cb-f783-4a10-806e-b1fe3fb28c1b","Type":"ContainerStarted","Data":"b93c6d567003983377e086f4435c69e87369c18f4ae5ff2959832f68ac09a2b5"} Oct 31 00:45:01.372120 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:45:01.372081 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890125-4twzz" event={"ID":"794a46cb-f783-4a10-806e-b1fe3fb28c1b","Type":"ContainerStarted","Data":"b502560c036410c5559fda069f5602e14bc8677d1b044b1a48c3d1ff966586a1"} Oct 31 00:45:01.490700 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:45:01.490648 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890125-4twzz" podStartSLOduration=1.4906362 podStartE2EDuration="1.4906362s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:45:01.489276677 +0000 UTC m=+3814.525607501" watchObservedRunningTime="2026-10-31 00:45:01.4906362 +0000 UTC m=+3814.526967022" Oct 31 00:45:02.247226 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:45:02.247190 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:02.379172 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:45:02.379131 4927 generic.go:356] "Generic (PLEG): container finished" podID="794a46cb-f783-4a10-806e-b1fe3fb28c1b" containerID="b93c6d567003983377e086f4435c69e87369c18f4ae5ff2959832f68ac09a2b5" exitCode=60 Oct 31 00:45:02.379694 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:45:02.379214 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890125-4twzz" event={"ID":"794a46cb-f783-4a10-806e-b1fe3fb28c1b","Type":"ContainerDied","Data":"b93c6d567003983377e086f4435c69e87369c18f4ae5ff2959832f68ac09a2b5"} Oct 31 00:45:02.379694 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:45:02.379543 4927 scope.go:122] "RemoveContainer" containerID="b93c6d567003983377e086f4435c69e87369c18f4ae5ff2959832f68ac09a2b5" Oct 31 00:45:02.828729 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:45:02.828693 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-e98014-openshift-default-596b484cdf5xsgc"] Oct 31 00:45:02.832481 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:45:02.832457 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-e98014-openshift-default-596b484cdf5xsgc" Oct 31 00:45:02.890891 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:45:02.890813 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-e98014-openshift-default-596b484cdf5xsgc"] Oct 31 00:45:02.896611 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:45:02.896582 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qq9n5\" (UniqueName: \"kubernetes.io/projected/ab2d8d66-2c0e-4394-8e10-903c2868442a-kube-api-access-qq9n5\") pod \"e2e-worker-single-main-e98014-openshift-default-596b484cdf5xsgc\" (UID: \"ab2d8d66-2c0e-4394-8e10-903c2868442a\") " pod="openshift-ingress/e2e-worker-single-main-e98014-openshift-default-596b484cdf5xsgc" Oct 31 00:45:02.896780 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:45:02.896621 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab2d8d66-2c0e-4394-8e10-903c2868442a-workload-socket\") pod \"e2e-worker-single-main-e98014-openshift-default-596b484cdf5xsgc\" (UID: \"ab2d8d66-2c0e-4394-8e10-903c2868442a\") " pod="openshift-ingress/e2e-worker-single-main-e98014-openshift-default-596b484cdf5xsgc" Oct 31 00:45:02.896780 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:45:02.896643 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab2d8d66-2c0e-4394-8e10-903c2868442a-credential-socket\") pod \"e2e-worker-single-main-e98014-openshift-default-596b484cdf5xsgc\" (UID: \"ab2d8d66-2c0e-4394-8e10-903c2868442a\") " pod="openshift-ingress/e2e-worker-single-main-e98014-openshift-default-596b484cdf5xsgc" Oct 31 00:45:02.896916 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:45:02.896806 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab2d8d66-2c0e-4394-8e10-903c2868442a-istio-podinfo\") pod \"e2e-worker-single-main-e98014-openshift-default-596b484cdf5xsgc\" (UID: \"ab2d8d66-2c0e-4394-8e10-903c2868442a\") " pod="openshift-ingress/e2e-worker-single-main-e98014-openshift-default-596b484cdf5xsgc" Oct 31 00:45:02.896916 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:45:02.896864 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab2d8d66-2c0e-4394-8e10-903c2868442a-istiod-ca-cert\") pod \"e2e-worker-single-main-e98014-openshift-default-596b484cdf5xsgc\" (UID: \"ab2d8d66-2c0e-4394-8e10-903c2868442a\") " pod="openshift-ingress/e2e-worker-single-main-e98014-openshift-default-596b484cdf5xsgc" Oct 31 00:45:02.896916 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:45:02.896894 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab2d8d66-2c0e-4394-8e10-903c2868442a-workload-certs\") pod \"e2e-worker-single-main-e98014-openshift-default-596b484cdf5xsgc\" (UID: \"ab2d8d66-2c0e-4394-8e10-903c2868442a\") " pod="openshift-ingress/e2e-worker-single-main-e98014-openshift-default-596b484cdf5xsgc" Oct 31 00:45:02.897094 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:45:02.896934 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab2d8d66-2c0e-4394-8e10-903c2868442a-istio-data\") pod \"e2e-worker-single-main-e98014-openshift-default-596b484cdf5xsgc\" (UID: \"ab2d8d66-2c0e-4394-8e10-903c2868442a\") " pod="openshift-ingress/e2e-worker-single-main-e98014-openshift-default-596b484cdf5xsgc" Oct 31 00:45:02.897094 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:45:02.897003 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab2d8d66-2c0e-4394-8e10-903c2868442a-istio-envoy\") pod \"e2e-worker-single-main-e98014-openshift-default-596b484cdf5xsgc\" (UID: \"ab2d8d66-2c0e-4394-8e10-903c2868442a\") " pod="openshift-ingress/e2e-worker-single-main-e98014-openshift-default-596b484cdf5xsgc" Oct 31 00:45:02.897094 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:45:02.897059 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab2d8d66-2c0e-4394-8e10-903c2868442a-istio-token\") pod \"e2e-worker-single-main-e98014-openshift-default-596b484cdf5xsgc\" (UID: \"ab2d8d66-2c0e-4394-8e10-903c2868442a\") " pod="openshift-ingress/e2e-worker-single-main-e98014-openshift-default-596b484cdf5xsgc" Oct 31 00:45:02.901507 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:45:02.901479 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-e98014-openshift-default-dockercfg-bnrdq\"" Oct 31 00:45:02.925192 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:45:02.925160 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:03.011874 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:45:03.011832 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qq9n5\" (UniqueName: \"kubernetes.io/projected/ab2d8d66-2c0e-4394-8e10-903c2868442a-kube-api-access-qq9n5\") pod \"e2e-worker-single-main-e98014-openshift-default-596b484cdf5xsgc\" (UID: \"ab2d8d66-2c0e-4394-8e10-903c2868442a\") " pod="openshift-ingress/e2e-worker-single-main-e98014-openshift-default-596b484cdf5xsgc" Oct 31 00:45:03.012083 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:45:03.011944 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab2d8d66-2c0e-4394-8e10-903c2868442a-workload-socket\") pod \"e2e-worker-single-main-e98014-openshift-default-596b484cdf5xsgc\" (UID: \"ab2d8d66-2c0e-4394-8e10-903c2868442a\") " pod="openshift-ingress/e2e-worker-single-main-e98014-openshift-default-596b484cdf5xsgc" Oct 31 00:45:03.012083 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:45:03.011978 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab2d8d66-2c0e-4394-8e10-903c2868442a-credential-socket\") pod \"e2e-worker-single-main-e98014-openshift-default-596b484cdf5xsgc\" (UID: \"ab2d8d66-2c0e-4394-8e10-903c2868442a\") " pod="openshift-ingress/e2e-worker-single-main-e98014-openshift-default-596b484cdf5xsgc" Oct 31 00:45:03.012083 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:45:03.012024 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab2d8d66-2c0e-4394-8e10-903c2868442a-istio-podinfo\") pod \"e2e-worker-single-main-e98014-openshift-default-596b484cdf5xsgc\" (UID: \"ab2d8d66-2c0e-4394-8e10-903c2868442a\") " pod="openshift-ingress/e2e-worker-single-main-e98014-openshift-default-596b484cdf5xsgc" Oct 31 00:45:03.012083 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:45:03.012069 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab2d8d66-2c0e-4394-8e10-903c2868442a-istiod-ca-cert\") pod \"e2e-worker-single-main-e98014-openshift-default-596b484cdf5xsgc\" (UID: \"ab2d8d66-2c0e-4394-8e10-903c2868442a\") " pod="openshift-ingress/e2e-worker-single-main-e98014-openshift-default-596b484cdf5xsgc" Oct 31 00:45:03.012317 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:45:03.012094 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab2d8d66-2c0e-4394-8e10-903c2868442a-workload-certs\") pod \"e2e-worker-single-main-e98014-openshift-default-596b484cdf5xsgc\" (UID: \"ab2d8d66-2c0e-4394-8e10-903c2868442a\") " pod="openshift-ingress/e2e-worker-single-main-e98014-openshift-default-596b484cdf5xsgc" Oct 31 00:45:03.012317 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:45:03.012129 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab2d8d66-2c0e-4394-8e10-903c2868442a-istio-data\") pod \"e2e-worker-single-main-e98014-openshift-default-596b484cdf5xsgc\" (UID: \"ab2d8d66-2c0e-4394-8e10-903c2868442a\") " pod="openshift-ingress/e2e-worker-single-main-e98014-openshift-default-596b484cdf5xsgc" Oct 31 00:45:03.012317 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:45:03.012165 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab2d8d66-2c0e-4394-8e10-903c2868442a-istio-envoy\") pod \"e2e-worker-single-main-e98014-openshift-default-596b484cdf5xsgc\" (UID: \"ab2d8d66-2c0e-4394-8e10-903c2868442a\") " pod="openshift-ingress/e2e-worker-single-main-e98014-openshift-default-596b484cdf5xsgc" Oct 31 00:45:03.012317 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:45:03.012228 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab2d8d66-2c0e-4394-8e10-903c2868442a-istio-token\") pod \"e2e-worker-single-main-e98014-openshift-default-596b484cdf5xsgc\" (UID: \"ab2d8d66-2c0e-4394-8e10-903c2868442a\") " pod="openshift-ingress/e2e-worker-single-main-e98014-openshift-default-596b484cdf5xsgc" Oct 31 00:45:03.013585 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:45:03.013526 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab2d8d66-2c0e-4394-8e10-903c2868442a-istiod-ca-cert\") pod \"e2e-worker-single-main-e98014-openshift-default-596b484cdf5xsgc\" (UID: \"ab2d8d66-2c0e-4394-8e10-903c2868442a\") " pod="openshift-ingress/e2e-worker-single-main-e98014-openshift-default-596b484cdf5xsgc" Oct 31 00:45:03.013835 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:45:03.013813 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab2d8d66-2c0e-4394-8e10-903c2868442a-workload-certs\") pod \"e2e-worker-single-main-e98014-openshift-default-596b484cdf5xsgc\" (UID: \"ab2d8d66-2c0e-4394-8e10-903c2868442a\") " pod="openshift-ingress/e2e-worker-single-main-e98014-openshift-default-596b484cdf5xsgc" Oct 31 00:45:03.014197 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:45:03.014173 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab2d8d66-2c0e-4394-8e10-903c2868442a-istio-data\") pod \"e2e-worker-single-main-e98014-openshift-default-596b484cdf5xsgc\" (UID: \"ab2d8d66-2c0e-4394-8e10-903c2868442a\") " pod="openshift-ingress/e2e-worker-single-main-e98014-openshift-default-596b484cdf5xsgc" Oct 31 00:45:03.014324 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:45:03.014299 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab2d8d66-2c0e-4394-8e10-903c2868442a-workload-socket\") pod \"e2e-worker-single-main-e98014-openshift-default-596b484cdf5xsgc\" (UID: \"ab2d8d66-2c0e-4394-8e10-903c2868442a\") " pod="openshift-ingress/e2e-worker-single-main-e98014-openshift-default-596b484cdf5xsgc" Oct 31 00:45:03.014675 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:45:03.014650 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab2d8d66-2c0e-4394-8e10-903c2868442a-credential-socket\") pod \"e2e-worker-single-main-e98014-openshift-default-596b484cdf5xsgc\" (UID: \"ab2d8d66-2c0e-4394-8e10-903c2868442a\") " pod="openshift-ingress/e2e-worker-single-main-e98014-openshift-default-596b484cdf5xsgc" Oct 31 00:45:03.016922 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:45:03.015482 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab2d8d66-2c0e-4394-8e10-903c2868442a-istio-podinfo\") pod \"e2e-worker-single-main-e98014-openshift-default-596b484cdf5xsgc\" (UID: \"ab2d8d66-2c0e-4394-8e10-903c2868442a\") " pod="openshift-ingress/e2e-worker-single-main-e98014-openshift-default-596b484cdf5xsgc" Oct 31 00:45:03.016922 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:45:03.016872 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab2d8d66-2c0e-4394-8e10-903c2868442a-istio-envoy\") pod \"e2e-worker-single-main-e98014-openshift-default-596b484cdf5xsgc\" (UID: \"ab2d8d66-2c0e-4394-8e10-903c2868442a\") " pod="openshift-ingress/e2e-worker-single-main-e98014-openshift-default-596b484cdf5xsgc" Oct 31 00:45:03.086102 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:45:03.084395 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qq9n5\" (UniqueName: \"kubernetes.io/projected/ab2d8d66-2c0e-4394-8e10-903c2868442a-kube-api-access-qq9n5\") pod \"e2e-worker-single-main-e98014-openshift-default-596b484cdf5xsgc\" (UID: \"ab2d8d66-2c0e-4394-8e10-903c2868442a\") " pod="openshift-ingress/e2e-worker-single-main-e98014-openshift-default-596b484cdf5xsgc" Oct 31 00:45:03.086102 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:45:03.084814 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab2d8d66-2c0e-4394-8e10-903c2868442a-istio-token\") pod \"e2e-worker-single-main-e98014-openshift-default-596b484cdf5xsgc\" (UID: \"ab2d8d66-2c0e-4394-8e10-903c2868442a\") " pod="openshift-ingress/e2e-worker-single-main-e98014-openshift-default-596b484cdf5xsgc" Oct 31 00:45:03.112926 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:45:03.112880 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:03.146443 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:45:03.146351 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-e98014-openshift-default-596b484cdf5xsgc" Oct 31 00:45:03.387574 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:45:03.387538 4927 generic.go:356] "Generic (PLEG): container finished" podID="794a46cb-f783-4a10-806e-b1fe3fb28c1b" containerID="49415f6f2d0a945d160b571bf98bcd92cf5419655a10a3947bfd95360a541b0f" exitCode=60 Oct 31 00:45:03.388020 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:45:03.387601 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890125-4twzz" event={"ID":"794a46cb-f783-4a10-806e-b1fe3fb28c1b","Type":"ContainerDied","Data":"49415f6f2d0a945d160b571bf98bcd92cf5419655a10a3947bfd95360a541b0f"} Oct 31 00:45:03.388020 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:45:03.387668 4927 scope.go:122] "RemoveContainer" containerID="b93c6d567003983377e086f4435c69e87369c18f4ae5ff2959832f68ac09a2b5" Oct 31 00:45:03.388020 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:45:03.387929 4927 scope.go:122] "RemoveContainer" containerID="49415f6f2d0a945d160b571bf98bcd92cf5419655a10a3947bfd95360a541b0f" Oct 31 00:45:03.388201 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:45:03.388175 4927 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29890125-4twzz_odh-ai-gateway-infra(794a46cb-f783-4a10-806e-b1fe3fb28c1b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890125-4twzz" podUID="794a46cb-f783-4a10-806e-b1fe3fb28c1b" Oct 31 00:45:03.634714 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:45:03.634685 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-e98014-openshift-default-596b484cdf5xsgc"] Oct 31 00:45:03.636131 ip-10-0-141-180 kubenswrapper[4927]: W1031 00:45:03.636109 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab2d8d66_2c0e_4394_8e10_903c2868442a.slice/crio-33a983b32ef89b12f44caa9c0690a62f6a78ad7853f8abe1ab72d1e35db4aaaf WatchSource:0}: Error finding container 33a983b32ef89b12f44caa9c0690a62f6a78ad7853f8abe1ab72d1e35db4aaaf: Status 404 returned error can't find the container with id 33a983b32ef89b12f44caa9c0690a62f6a78ad7853f8abe1ab72d1e35db4aaaf Oct 31 00:45:03.638444 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:45:03.638413 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:45:03.638501 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:45:03.638476 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:45:03.638542 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:45:03.638506 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:45:04.400407 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:45:04.400293 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-e98014-openshift-default-596b484cdf5xsgc" event={"ID":"ab2d8d66-2c0e-4394-8e10-903c2868442a","Type":"ContainerStarted","Data":"533e67ff73d35afef873acd90475444dee2431a49c0c65ef7589cd7983e3e7ab"} Oct 31 00:45:04.400885 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:45:04.400414 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-e98014-openshift-default-596b484cdf5xsgc" event={"ID":"ab2d8d66-2c0e-4394-8e10-903c2868442a","Type":"ContainerStarted","Data":"33a983b32ef89b12f44caa9c0690a62f6a78ad7853f8abe1ab72d1e35db4aaaf"} Oct 31 00:45:04.403198 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:45:04.403171 4927 scope.go:122] "RemoveContainer" containerID="49415f6f2d0a945d160b571bf98bcd92cf5419655a10a3947bfd95360a541b0f" Oct 31 00:45:04.403485 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:45:04.403462 4927 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29890125-4twzz_odh-ai-gateway-infra(794a46cb-f783-4a10-806e-b1fe3fb28c1b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890125-4twzz" podUID="794a46cb-f783-4a10-806e-b1fe3fb28c1b" Oct 31 00:45:04.532515 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:45:04.532465 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-e98014-openshift-default-596b484cdf5xsgc" podStartSLOduration=2.53245083 podStartE2EDuration="2.53245083s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:45:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:45:04.529961772 +0000 UTC m=+3817.566292614" watchObservedRunningTime="2026-10-31 00:45:04.53245083 +0000 UTC m=+3817.568781651" Oct 31 00:45:05.146674 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:45:05.146631 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-e98014-openshift-default-596b484cdf5xsgc" Oct 31 00:45:05.151168 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:45:05.151143 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-e98014-openshift-default-596b484cdf5xsgc" Oct 31 00:45:05.408774 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:45:05.408747 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-e98014-openshift-default-596b484cdf5xsgc" Oct 31 00:45:05.409808 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:45:05.409786 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-e98014-openshift-default-596b484cdf5xsgc" Oct 31 00:45:07.442807 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:45:07.442717 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:07.613915 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:45:07.613877 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:07.782568 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:45:07.782470 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:10.612121 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:45:10.612084 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:10.780016 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:45:10.779984 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:13.703784 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:45:13.703748 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-e98014-d98bb5765-dvtr7"] Oct 31 00:45:13.707832 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:45:13.707811 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-e98014-d98bb5765-dvtr7" Oct 31 00:45:13.765185 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:45:13.765143 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-e98014-d98bb5765-dvtr7"] Oct 31 00:45:13.811755 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:45:13.811723 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-e98014-serving-cert\"" Oct 31 00:45:13.811948 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:45:13.811772 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 31 00:45:13.811948 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:45:13.811855 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-2nk58\"" Oct 31 00:45:13.820206 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:45:13.820179 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-25hg6\" (UniqueName: \"kubernetes.io/projected/39f9b16c-33a6-4253-a562-a6b5ec648f58-kube-api-access-25hg6\") pod \"maas-api-e2e-worker-single-main-e98014-d98bb5765-dvtr7\" (UID: \"39f9b16c-33a6-4253-a562-a6b5ec648f58\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-e98014-d98bb5765-dvtr7" Oct 31 00:45:13.820357 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:45:13.820226 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/39f9b16c-33a6-4253-a562-a6b5ec648f58-metrics-certs\") pod \"maas-api-e2e-worker-single-main-e98014-d98bb5765-dvtr7\" (UID: \"39f9b16c-33a6-4253-a562-a6b5ec648f58\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-e98014-d98bb5765-dvtr7" Oct 31 00:45:13.820446 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:45:13.820420 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/39f9b16c-33a6-4253-a562-a6b5ec648f58-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-e98014-d98bb5765-dvtr7\" (UID: \"39f9b16c-33a6-4253-a562-a6b5ec648f58\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-e98014-d98bb5765-dvtr7" Oct 31 00:45:13.820551 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:45:13.820533 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/39f9b16c-33a6-4253-a562-a6b5ec648f58-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-e98014-d98bb5765-dvtr7\" (UID: \"39f9b16c-33a6-4253-a562-a6b5ec648f58\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-e98014-d98bb5765-dvtr7" Oct 31 00:45:13.907507 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:45:13.907470 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:13.921704 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:45:13.921673 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/39f9b16c-33a6-4253-a562-a6b5ec648f58-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-e98014-d98bb5765-dvtr7\" (UID: \"39f9b16c-33a6-4253-a562-a6b5ec648f58\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-e98014-d98bb5765-dvtr7" Oct 31 00:45:13.921859 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:45:13.921720 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25hg6\" (UniqueName: \"kubernetes.io/projected/39f9b16c-33a6-4253-a562-a6b5ec648f58-kube-api-access-25hg6\") pod \"maas-api-e2e-worker-single-main-e98014-d98bb5765-dvtr7\" (UID: \"39f9b16c-33a6-4253-a562-a6b5ec648f58\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-e98014-d98bb5765-dvtr7" Oct 31 00:45:13.921859 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:45:13.921758 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/39f9b16c-33a6-4253-a562-a6b5ec648f58-metrics-certs\") pod \"maas-api-e2e-worker-single-main-e98014-d98bb5765-dvtr7\" (UID: \"39f9b16c-33a6-4253-a562-a6b5ec648f58\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-e98014-d98bb5765-dvtr7" Oct 31 00:45:13.921859 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:45:13.921845 4927 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-single-main-e98014-serving-cert: secret "maas-api-e2e-worker-single-main-e98014-serving-cert" not found Oct 31 00:45:13.922026 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:45:13.921858 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/39f9b16c-33a6-4253-a562-a6b5ec648f58-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-e98014-d98bb5765-dvtr7\" (UID: \"39f9b16c-33a6-4253-a562-a6b5ec648f58\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-e98014-d98bb5765-dvtr7" Oct 31 00:45:13.922026 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:45:13.921927 4927 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/39f9b16c-33a6-4253-a562-a6b5ec648f58-maas-api-tls podName:39f9b16c-33a6-4253-a562-a6b5ec648f58 nodeName:}" failed. No retries permitted until 2026-10-31 00:45:14.421906557 +0000 UTC m=+3827.458237376 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/39f9b16c-33a6-4253-a562-a6b5ec648f58-maas-api-tls") pod "maas-api-e2e-worker-single-main-e98014-d98bb5765-dvtr7" (UID: "39f9b16c-33a6-4253-a562-a6b5ec648f58") : secret "maas-api-e2e-worker-single-main-e98014-serving-cert" not found Oct 31 00:45:13.922602 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:45:13.922578 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/39f9b16c-33a6-4253-a562-a6b5ec648f58-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-e98014-d98bb5765-dvtr7\" (UID: \"39f9b16c-33a6-4253-a562-a6b5ec648f58\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-e98014-d98bb5765-dvtr7" Oct 31 00:45:13.924344 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:45:13.924311 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/39f9b16c-33a6-4253-a562-a6b5ec648f58-metrics-certs\") pod \"maas-api-e2e-worker-single-main-e98014-d98bb5765-dvtr7\" (UID: \"39f9b16c-33a6-4253-a562-a6b5ec648f58\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-e98014-d98bb5765-dvtr7" Oct 31 00:45:13.985162 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:45:13.985071 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-25hg6\" (UniqueName: \"kubernetes.io/projected/39f9b16c-33a6-4253-a562-a6b5ec648f58-kube-api-access-25hg6\") pod \"maas-api-e2e-worker-single-main-e98014-d98bb5765-dvtr7\" (UID: \"39f9b16c-33a6-4253-a562-a6b5ec648f58\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-e98014-d98bb5765-dvtr7" Oct 31 00:45:14.099267 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:45:14.099233 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:14.246622 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:45:14.246541 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:14.429560 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:45:14.429529 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/39f9b16c-33a6-4253-a562-a6b5ec648f58-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-e98014-d98bb5765-dvtr7\" (UID: \"39f9b16c-33a6-4253-a562-a6b5ec648f58\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-e98014-d98bb5765-dvtr7" Oct 31 00:45:14.431983 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:45:14.431959 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/39f9b16c-33a6-4253-a562-a6b5ec648f58-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-e98014-d98bb5765-dvtr7\" (UID: \"39f9b16c-33a6-4253-a562-a6b5ec648f58\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-e98014-d98bb5765-dvtr7" Oct 31 00:45:14.620837 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:45:14.620756 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-e98014-d98bb5765-dvtr7" Oct 31 00:45:14.997962 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:45:14.997933 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-e98014-d98bb5765-dvtr7"] Oct 31 00:45:14.999645 ip-10-0-141-180 kubenswrapper[4927]: W1031 00:45:14.999610 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39f9b16c_33a6_4253_a562_a6b5ec648f58.slice/crio-009ca1fd451f5a84d3c837dd303d8485267c44a4e4c464a285221dfb2f8d6808 WatchSource:0}: Error finding container 009ca1fd451f5a84d3c837dd303d8485267c44a4e4c464a285221dfb2f8d6808: Status 404 returned error can't find the container with id 009ca1fd451f5a84d3c837dd303d8485267c44a4e4c464a285221dfb2f8d6808 Oct 31 00:45:15.471998 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:45:15.471960 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-e98014-d98bb5765-dvtr7" event={"ID":"39f9b16c-33a6-4253-a562-a6b5ec648f58","Type":"ContainerStarted","Data":"009ca1fd451f5a84d3c837dd303d8485267c44a4e4c464a285221dfb2f8d6808"} Oct 31 00:45:16.079013 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:45:16.078983 4927 scope.go:122] "RemoveContainer" containerID="49415f6f2d0a945d160b571bf98bcd92cf5419655a10a3947bfd95360a541b0f" Oct 31 00:45:16.479073 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:45:16.479030 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-e98014-d98bb5765-dvtr7" event={"ID":"39f9b16c-33a6-4253-a562-a6b5ec648f58","Type":"ContainerStarted","Data":"02226ab180b9dd0d3ee54f539901f1ee6f782b69da489a036879717863d075d4"} Oct 31 00:45:16.479245 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:45:16.479108 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-e98014-d98bb5765-dvtr7" Oct 31 00:45:16.480951 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:45:16.480929 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890125-4twzz" event={"ID":"794a46cb-f783-4a10-806e-b1fe3fb28c1b","Type":"ContainerStarted","Data":"25028e6c52198a921313eece3c8160041aceab47efa8038cd5b42ccf5be5b970"} Oct 31 00:45:16.601261 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:45:16.601155 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-e98014-d98bb5765-dvtr7" podStartSLOduration=2.646785292 podStartE2EDuration="3.60114075s" totalImagesPullingTime="954.355458ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:45:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:45:16.598462958 +0000 UTC m=+3829.634793781" watchObservedRunningTime="2026-10-31 00:45:16.60114075 +0000 UTC m=+3829.637471598" Oct 31 00:45:17.488637 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:45:17.488602 4927 generic.go:356] "Generic (PLEG): container finished" podID="794a46cb-f783-4a10-806e-b1fe3fb28c1b" containerID="25028e6c52198a921313eece3c8160041aceab47efa8038cd5b42ccf5be5b970" exitCode=60 Oct 31 00:45:17.489065 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:45:17.488672 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890125-4twzz" event={"ID":"794a46cb-f783-4a10-806e-b1fe3fb28c1b","Type":"ContainerDied","Data":"25028e6c52198a921313eece3c8160041aceab47efa8038cd5b42ccf5be5b970"} Oct 31 00:45:17.489065 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:45:17.488733 4927 scope.go:122] "RemoveContainer" containerID="49415f6f2d0a945d160b571bf98bcd92cf5419655a10a3947bfd95360a541b0f" Oct 31 00:45:17.489173 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:45:17.489077 4927 scope.go:122] "RemoveContainer" containerID="25028e6c52198a921313eece3c8160041aceab47efa8038cd5b42ccf5be5b970" Oct 31 00:45:17.489387 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:45:17.489362 4927 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29890125-4twzz_odh-ai-gateway-infra(794a46cb-f783-4a10-806e-b1fe3fb28c1b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890125-4twzz" podUID="794a46cb-f783-4a10-806e-b1fe3fb28c1b" Oct 31 00:45:17.728466 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:45:17.728434 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890125-4twzz"] Oct 31 00:45:18.631804 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:45:18.631776 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890125-4twzz" Oct 31 00:45:18.773498 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:45:18.773408 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/794a46cb-f783-4a10-806e-b1fe3fb28c1b-kube-api-access-mmrpm\" (UniqueName: \"kubernetes.io/projected/794a46cb-f783-4a10-806e-b1fe3fb28c1b-kube-api-access-mmrpm\") pod \"794a46cb-f783-4a10-806e-b1fe3fb28c1b\" (UID: \"794a46cb-f783-4a10-806e-b1fe3fb28c1b\") " Oct 31 00:45:18.773687 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:45:18.773566 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/794a46cb-f783-4a10-806e-b1fe3fb28c1b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/794a46cb-f783-4a10-806e-b1fe3fb28c1b-maas-api-service-ca\") pod \"794a46cb-f783-4a10-806e-b1fe3fb28c1b\" (UID: \"794a46cb-f783-4a10-806e-b1fe3fb28c1b\") " Oct 31 00:45:18.774021 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:45:18.773989 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/794a46cb-f783-4a10-806e-b1fe3fb28c1b-maas-api-service-ca" pod "794a46cb-f783-4a10-806e-b1fe3fb28c1b" (UID: "794a46cb-f783-4a10-806e-b1fe3fb28c1b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:45:18.775700 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:45:18.775671 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/794a46cb-f783-4a10-806e-b1fe3fb28c1b-kube-api-access-mmrpm" pod "794a46cb-f783-4a10-806e-b1fe3fb28c1b" (UID: "794a46cb-f783-4a10-806e-b1fe3fb28c1b"). InnerVolumeSpecName "kube-api-access-mmrpm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:45:18.875415 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:45:18.875380 4927 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/794a46cb-f783-4a10-806e-b1fe3fb28c1b-maas-api-service-ca\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:45:18.875415 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:45:18.875409 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mmrpm\" (UniqueName: \"kubernetes.io/projected/794a46cb-f783-4a10-806e-b1fe3fb28c1b-kube-api-access-mmrpm\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:45:19.077300 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:45:19.077210 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:19.503568 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:45:19.503538 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890125-4twzz" Oct 31 00:45:19.503568 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:45:19.503554 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890125-4twzz" event={"ID":"794a46cb-f783-4a10-806e-b1fe3fb28c1b","Type":"ContainerDied","Data":"b502560c036410c5559fda069f5602e14bc8677d1b044b1a48c3d1ff966586a1"} Oct 31 00:45:19.503853 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:45:19.503611 4927 scope.go:122] "RemoveContainer" containerID="25028e6c52198a921313eece3c8160041aceab47efa8038cd5b42ccf5be5b970" Oct 31 00:45:19.673423 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:45:19.673391 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890125-4twzz"] Oct 31 00:45:19.679827 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:45:19.679791 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sqv69" containerName="extractor" Oct 31 00:45:19.723042 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:45:19.723011 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890125-4twzz"] Oct 31 00:45:21.084443 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:45:21.084412 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="794a46cb-f783-4a10-806e-b1fe3fb28c1b" path="/var/lib/kubelet/pods/794a46cb-f783-4a10-806e-b1fe3fb28c1b/volumes" Oct 31 00:45:22.494663 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:45:22.494628 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-e98014-d98bb5765-dvtr7" Oct 31 00:45:24.172844 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:45:24.172810 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:29.162751 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:45:29.162712 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:31.597097 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:45:31.597062 4927 generic.go:356] "Generic (PLEG): container finished" podID="ab2d8d66-2c0e-4394-8e10-903c2868442a" containerID="533e67ff73d35afef873acd90475444dee2431a49c0c65ef7589cd7983e3e7ab" exitCode=0 Oct 31 00:45:31.597603 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:45:31.597152 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-e98014-openshift-default-596b484cdf5xsgc" event={"ID":"ab2d8d66-2c0e-4394-8e10-903c2868442a","Type":"ContainerDied","Data":"533e67ff73d35afef873acd90475444dee2431a49c0c65ef7589cd7983e3e7ab"} Oct 31 00:45:31.597603 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:45:31.597590 4927 scope.go:122] "RemoveContainer" containerID="533e67ff73d35afef873acd90475444dee2431a49c0c65ef7589cd7983e3e7ab" Oct 31 00:45:31.598783 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:45:31.598755 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:45:31.598849 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:45:31.598824 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:45:31.598888 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:45:31.598858 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:45:32.238640 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:45:32.238605 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:32.605162 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:45:32.605072 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-e98014-openshift-default-596b484cdf5xsgc" event={"ID":"ab2d8d66-2c0e-4394-8e10-903c2868442a","Type":"ContainerStarted","Data":"71ba38239156ab54947afb04d056be69bb2513f35b3bc5235afec10f3b041112"} Oct 31 00:45:32.783998 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:45:32.783953 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:33.045529 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:45:33.045494 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:33.146852 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:45:33.146821 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-e98014-openshift-default-596b484cdf5xsgc" Oct 31 00:45:33.148086 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:45:33.148057 4927 patch_prober.go:29] interesting pod/e2e-worker-single-main-e98014-openshift-default-596b484cdf5xsgc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.39:15021/healthz/ready\": dial tcp 10.132.1.39:15021: connect: connection refused" start-of-body= Oct 31 00:45:33.148223 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:45:33.148109 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-e98014-openshift-default-596b484cdf5xsgc" podUID="ab2d8d66-2c0e-4394-8e10-903c2868442a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.39:15021/healthz/ready\": dial tcp 10.132.1.39:15021: connect: connection refused" Oct 31 00:45:34.147552 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:45:34.147515 4927 patch_prober.go:29] interesting pod/e2e-worker-single-main-e98014-openshift-default-596b484cdf5xsgc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.39:15021/healthz/ready\": dial tcp 10.132.1.39:15021: connect: connection refused" start-of-body= Oct 31 00:45:34.147968 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:45:34.147566 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-e98014-openshift-default-596b484cdf5xsgc" podUID="ab2d8d66-2c0e-4394-8e10-903c2868442a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.39:15021/healthz/ready\": dial tcp 10.132.1.39:15021: connect: connection refused" Oct 31 00:45:34.334929 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:45:34.334896 4927 scope.go:122] "RemoveContainer" containerID="00210cb868cf533febb7ad5f3253e45d9a421d386877e53b5569fd842434678b" Oct 31 00:45:34.657836 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:45:34.657795 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:34.848125 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:45:34.848092 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:35.147130 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:45:35.147048 4927 patch_prober.go:29] interesting pod/e2e-worker-single-main-e98014-openshift-default-596b484cdf5xsgc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.39:15021/healthz/ready\": dial tcp 10.132.1.39:15021: connect: connection refused" start-of-body= Oct 31 00:45:35.147130 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:45:35.147099 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-e98014-openshift-default-596b484cdf5xsgc" podUID="ab2d8d66-2c0e-4394-8e10-903c2868442a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.39:15021/healthz/ready\": dial tcp 10.132.1.39:15021: connect: connection refused" Oct 31 00:45:35.432512 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:45:35.432474 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-e98014-openshift-default-596b484cdf5xsgc" Oct 31 00:45:35.447021 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:45:35.446988 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:36.146707 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:45:36.146671 4927 patch_prober.go:29] interesting pod/e2e-worker-single-main-e98014-openshift-default-596b484cdf5xsgc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.39:15021/healthz/ready\": dial tcp 10.132.1.39:15021: connect: connection refused" start-of-body= Oct 31 00:45:36.146913 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:45:36.146729 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-e98014-openshift-default-596b484cdf5xsgc" podUID="ab2d8d66-2c0e-4394-8e10-903c2868442a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.39:15021/healthz/ready\": dial tcp 10.132.1.39:15021: connect: connection refused" Oct 31 00:45:37.147503 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:45:37.147464 4927 patch_prober.go:29] interesting pod/e2e-worker-single-main-e98014-openshift-default-596b484cdf5xsgc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.39:15021/healthz/ready\": dial tcp 10.132.1.39:15021: connect: connection refused" start-of-body= Oct 31 00:45:37.147888 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:45:37.147550 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-e98014-openshift-default-596b484cdf5xsgc" podUID="ab2d8d66-2c0e-4394-8e10-903c2868442a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.39:15021/healthz/ready\": dial tcp 10.132.1.39:15021: connect: connection refused" Oct 31 00:45:38.150897 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:45:38.150859 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-e98014-openshift-default-596b484cdf5xsgc" Oct 31 00:45:38.151939 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:45:38.151918 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-e98014-openshift-default-596b484cdf5xsgc" Oct 31 00:45:39.594300 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:45:39.594258 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:40.287491 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:45:40.287448 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:44.579557 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:45:44.579526 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:49.550207 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:45:49.550173 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:49.678153 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:45:49.678117 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sqv69" containerName="extractor" Oct 31 00:45:50.764078 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:45:50.764044 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:54.789156 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:45:54.789110 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:59.675484 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:45:59.675448 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:04.673681 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:46:04.673637 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:09.670755 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:46:09.670708 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:14.758576 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:46:14.758537 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:19.678462 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:46:19.678421 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sqv69" containerName="extractor" Oct 31 00:46:19.695685 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:46:19.695640 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:24.776672 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:46:24.776631 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:28.886321 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:46:28.886292 4927 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-pqt2c_6382dfb5-a363-47a2-bc79-0b96f7bec0c3/console-operator/3.log" Oct 31 00:46:29.774149 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:46:29.774110 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:29.912645 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:46:29.912616 4927 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-pqt2c_6382dfb5-a363-47a2-bc79-0b96f7bec0c3/console-operator/3.log" Oct 31 00:46:32.223279 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:46:32.223247 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:34.362791 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:46:34.362759 4927 scope.go:122] "RemoveContainer" containerID="e951c3312e1f98ea08084aeb72c061acc2f26c78b8c0d4ecd034edb5c33d7423" Oct 31 00:46:34.372238 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:46:34.372213 4927 scope.go:122] "RemoveContainer" containerID="83b4fd2ffc3e526cd930ec5e8baec58635a36049e2d06dc4618ab708a605e136" Oct 31 00:46:35.092550 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:46:35.092511 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:37.229181 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:46:37.229090 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:40.192616 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:46:40.192581 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:40.651994 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:46:40.651911 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:44.471178 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:46:44.471141 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:44.473734 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:46:44.473703 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-e98014-d98bb5765-dvtr7"] Oct 31 00:46:44.473955 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:46:44.473933 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-e98014-d98bb5765-dvtr7" podUID="39f9b16c-33a6-4253-a562-a6b5ec648f58" containerName="maas-api" containerID="cri-o://02226ab180b9dd0d3ee54f539901f1ee6f782b69da489a036879717863d075d4" gracePeriod=30 Oct 31 00:46:44.621983 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:46:44.621945 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:45.747029 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:46:45.746999 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-e98014-d98bb5765-dvtr7" Oct 31 00:46:45.898638 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:46:45.898550 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/39f9b16c-33a6-4253-a562-a6b5ec648f58-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/39f9b16c-33a6-4253-a562-a6b5ec648f58-openshift-service-ca\") pod \"39f9b16c-33a6-4253-a562-a6b5ec648f58\" (UID: \"39f9b16c-33a6-4253-a562-a6b5ec648f58\") " Oct 31 00:46:45.898638 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:46:45.898630 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/39f9b16c-33a6-4253-a562-a6b5ec648f58-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/39f9b16c-33a6-4253-a562-a6b5ec648f58-maas-api-tls\") pod \"39f9b16c-33a6-4253-a562-a6b5ec648f58\" (UID: \"39f9b16c-33a6-4253-a562-a6b5ec648f58\") " Oct 31 00:46:45.898884 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:46:45.898704 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/39f9b16c-33a6-4253-a562-a6b5ec648f58-metrics-certs\" (UniqueName: \"kubernetes.io/secret/39f9b16c-33a6-4253-a562-a6b5ec648f58-metrics-certs\") pod \"39f9b16c-33a6-4253-a562-a6b5ec648f58\" (UID: \"39f9b16c-33a6-4253-a562-a6b5ec648f58\") " Oct 31 00:46:45.898884 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:46:45.898760 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39f9b16c-33a6-4253-a562-a6b5ec648f58-kube-api-access-25hg6\" (UniqueName: \"kubernetes.io/projected/39f9b16c-33a6-4253-a562-a6b5ec648f58-kube-api-access-25hg6\") pod \"39f9b16c-33a6-4253-a562-a6b5ec648f58\" (UID: \"39f9b16c-33a6-4253-a562-a6b5ec648f58\") " Oct 31 00:46:45.899013 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:46:45.898979 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39f9b16c-33a6-4253-a562-a6b5ec648f58-openshift-service-ca" pod "39f9b16c-33a6-4253-a562-a6b5ec648f58" (UID: "39f9b16c-33a6-4253-a562-a6b5ec648f58"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:46:45.899180 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:46:45.899161 4927 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/39f9b16c-33a6-4253-a562-a6b5ec648f58-openshift-service-ca\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:46:45.900978 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:46:45.900950 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39f9b16c-33a6-4253-a562-a6b5ec648f58-kube-api-access-25hg6" pod "39f9b16c-33a6-4253-a562-a6b5ec648f58" (UID: "39f9b16c-33a6-4253-a562-a6b5ec648f58"). InnerVolumeSpecName "kube-api-access-25hg6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:46:45.901247 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:46:45.901221 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/39f9b16c-33a6-4253-a562-a6b5ec648f58-maas-api-tls" pod "39f9b16c-33a6-4253-a562-a6b5ec648f58" (UID: "39f9b16c-33a6-4253-a562-a6b5ec648f58"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:46:45.901247 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:46:45.901238 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/39f9b16c-33a6-4253-a562-a6b5ec648f58-metrics-certs" pod "39f9b16c-33a6-4253-a562-a6b5ec648f58" (UID: "39f9b16c-33a6-4253-a562-a6b5ec648f58"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:46:45.999900 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:46:45.999863 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-25hg6\" (UniqueName: \"kubernetes.io/projected/39f9b16c-33a6-4253-a562-a6b5ec648f58-kube-api-access-25hg6\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:46:45.999900 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:46:45.999898 4927 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/39f9b16c-33a6-4253-a562-a6b5ec648f58-maas-api-tls\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:46:45.999900 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:46:45.999908 4927 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/39f9b16c-33a6-4253-a562-a6b5ec648f58-metrics-certs\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:46:46.072917 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:46:46.072879 4927 generic.go:356] "Generic (PLEG): container finished" podID="39f9b16c-33a6-4253-a562-a6b5ec648f58" containerID="02226ab180b9dd0d3ee54f539901f1ee6f782b69da489a036879717863d075d4" exitCode=0 Oct 31 00:46:46.073119 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:46:46.072935 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-e98014-d98bb5765-dvtr7" Oct 31 00:46:46.073119 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:46:46.072962 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-e98014-d98bb5765-dvtr7" event={"ID":"39f9b16c-33a6-4253-a562-a6b5ec648f58","Type":"ContainerDied","Data":"02226ab180b9dd0d3ee54f539901f1ee6f782b69da489a036879717863d075d4"} Oct 31 00:46:46.073119 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:46:46.073033 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-e98014-d98bb5765-dvtr7" event={"ID":"39f9b16c-33a6-4253-a562-a6b5ec648f58","Type":"ContainerDied","Data":"009ca1fd451f5a84d3c837dd303d8485267c44a4e4c464a285221dfb2f8d6808"} Oct 31 00:46:46.073119 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:46:46.073065 4927 scope.go:122] "RemoveContainer" containerID="02226ab180b9dd0d3ee54f539901f1ee6f782b69da489a036879717863d075d4" Oct 31 00:46:46.083478 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:46:46.083460 4927 scope.go:122] "RemoveContainer" containerID="02226ab180b9dd0d3ee54f539901f1ee6f782b69da489a036879717863d075d4" Oct 31 00:46:46.083723 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:46:46.083707 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"02226ab180b9dd0d3ee54f539901f1ee6f782b69da489a036879717863d075d4\": container with ID starting with 02226ab180b9dd0d3ee54f539901f1ee6f782b69da489a036879717863d075d4 not found: ID does not exist" containerID="02226ab180b9dd0d3ee54f539901f1ee6f782b69da489a036879717863d075d4" Oct 31 00:46:46.083776 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:46:46.083739 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02226ab180b9dd0d3ee54f539901f1ee6f782b69da489a036879717863d075d4"} err="failed to get container status \"02226ab180b9dd0d3ee54f539901f1ee6f782b69da489a036879717863d075d4\": rpc error: code = NotFound desc = could not find container \"02226ab180b9dd0d3ee54f539901f1ee6f782b69da489a036879717863d075d4\": container with ID starting with 02226ab180b9dd0d3ee54f539901f1ee6f782b69da489a036879717863d075d4 not found: ID does not exist" Oct 31 00:46:46.241843 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:46:46.241812 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-e98014-d98bb5765-dvtr7"] Oct 31 00:46:46.245525 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:46:46.245500 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-e98014-d98bb5765-dvtr7"] Oct 31 00:46:47.084700 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:46:47.084665 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="39f9b16c-33a6-4253-a562-a6b5ec648f58" path="/var/lib/kubelet/pods/39f9b16c-33a6-4253-a562-a6b5ec648f58/volumes" Oct 31 00:46:48.324826 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:46:48.324781 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:48.602657 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:46:48.602573 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:49.379656 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:46:49.379618 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:49.677901 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:46:49.677867 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sqv69" containerName="extractor" Oct 31 00:46:55.673623 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:46:55.673575 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:55.828946 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:46:55.828896 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:56.013064 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:46:56.012962 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-e98014-openshift-default-596b484cdf5xsgc"] Oct 31 00:46:56.013258 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:46:56.013200 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-e98014-openshift-default-596b484cdf5xsgc" podUID="ab2d8d66-2c0e-4394-8e10-903c2868442a" containerName="istio-proxy" containerID="cri-o://71ba38239156ab54947afb04d056be69bb2513f35b3bc5235afec10f3b041112" gracePeriod=30 Oct 31 00:46:56.066729 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:46:56.066697 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:01.161587 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:47:01.161552 4927 generic.go:356] "Generic (PLEG): container finished" podID="ab2d8d66-2c0e-4394-8e10-903c2868442a" containerID="71ba38239156ab54947afb04d056be69bb2513f35b3bc5235afec10f3b041112" exitCode=0 Oct 31 00:47:01.162170 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:47:01.161604 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-e98014-openshift-default-596b484cdf5xsgc" event={"ID":"ab2d8d66-2c0e-4394-8e10-903c2868442a","Type":"ContainerDied","Data":"71ba38239156ab54947afb04d056be69bb2513f35b3bc5235afec10f3b041112"} Oct 31 00:47:01.162170 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:47:01.161684 4927 scope.go:122] "RemoveContainer" containerID="533e67ff73d35afef873acd90475444dee2431a49c0c65ef7589cd7983e3e7ab" Oct 31 00:47:01.274453 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:47:01.274419 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-e98014-openshift-default-596b484cdf5xsgc" Oct 31 00:47:01.349929 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:47:01.349837 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab2d8d66-2c0e-4394-8e10-903c2868442a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab2d8d66-2c0e-4394-8e10-903c2868442a-istio-envoy\") pod \"ab2d8d66-2c0e-4394-8e10-903c2868442a\" (UID: \"ab2d8d66-2c0e-4394-8e10-903c2868442a\") " Oct 31 00:47:01.349929 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:47:01.349893 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab2d8d66-2c0e-4394-8e10-903c2868442a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab2d8d66-2c0e-4394-8e10-903c2868442a-workload-socket\") pod \"ab2d8d66-2c0e-4394-8e10-903c2868442a\" (UID: \"ab2d8d66-2c0e-4394-8e10-903c2868442a\") " Oct 31 00:47:01.349929 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:47:01.349927 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab2d8d66-2c0e-4394-8e10-903c2868442a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab2d8d66-2c0e-4394-8e10-903c2868442a-istiod-ca-cert\") pod \"ab2d8d66-2c0e-4394-8e10-903c2868442a\" (UID: \"ab2d8d66-2c0e-4394-8e10-903c2868442a\") " Oct 31 00:47:01.350230 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:47:01.349963 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab2d8d66-2c0e-4394-8e10-903c2868442a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab2d8d66-2c0e-4394-8e10-903c2868442a-credential-socket\") pod \"ab2d8d66-2c0e-4394-8e10-903c2868442a\" (UID: \"ab2d8d66-2c0e-4394-8e10-903c2868442a\") " Oct 31 00:47:01.350230 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:47:01.349995 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab2d8d66-2c0e-4394-8e10-903c2868442a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab2d8d66-2c0e-4394-8e10-903c2868442a-istio-data\") pod \"ab2d8d66-2c0e-4394-8e10-903c2868442a\" (UID: \"ab2d8d66-2c0e-4394-8e10-903c2868442a\") " Oct 31 00:47:01.350230 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:47:01.350021 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab2d8d66-2c0e-4394-8e10-903c2868442a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab2d8d66-2c0e-4394-8e10-903c2868442a-workload-certs\") pod \"ab2d8d66-2c0e-4394-8e10-903c2868442a\" (UID: \"ab2d8d66-2c0e-4394-8e10-903c2868442a\") " Oct 31 00:47:01.350230 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:47:01.350089 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ab2d8d66-2c0e-4394-8e10-903c2868442a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab2d8d66-2c0e-4394-8e10-903c2868442a-istio-podinfo\") pod \"ab2d8d66-2c0e-4394-8e10-903c2868442a\" (UID: \"ab2d8d66-2c0e-4394-8e10-903c2868442a\") " Oct 31 00:47:01.350230 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:47:01.350137 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab2d8d66-2c0e-4394-8e10-903c2868442a-kube-api-access-qq9n5\" (UniqueName: \"kubernetes.io/projected/ab2d8d66-2c0e-4394-8e10-903c2868442a-kube-api-access-qq9n5\") pod \"ab2d8d66-2c0e-4394-8e10-903c2868442a\" (UID: \"ab2d8d66-2c0e-4394-8e10-903c2868442a\") " Oct 31 00:47:01.350230 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:47:01.350184 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab2d8d66-2c0e-4394-8e10-903c2868442a-istio-token\" (UniqueName: \"kubernetes.io/projected/ab2d8d66-2c0e-4394-8e10-903c2868442a-istio-token\") pod \"ab2d8d66-2c0e-4394-8e10-903c2868442a\" (UID: \"ab2d8d66-2c0e-4394-8e10-903c2868442a\") " Oct 31 00:47:01.350689 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:47:01.350305 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab2d8d66-2c0e-4394-8e10-903c2868442a-credential-socket" pod "ab2d8d66-2c0e-4394-8e10-903c2868442a" (UID: "ab2d8d66-2c0e-4394-8e10-903c2868442a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:47:01.350689 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:47:01.350316 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab2d8d66-2c0e-4394-8e10-903c2868442a-workload-socket" pod "ab2d8d66-2c0e-4394-8e10-903c2868442a" (UID: "ab2d8d66-2c0e-4394-8e10-903c2868442a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:47:01.350689 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:47:01.350418 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab2d8d66-2c0e-4394-8e10-903c2868442a-istio-data" pod "ab2d8d66-2c0e-4394-8e10-903c2868442a" (UID: "ab2d8d66-2c0e-4394-8e10-903c2868442a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:47:01.350689 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:47:01.350469 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab2d8d66-2c0e-4394-8e10-903c2868442a-istiod-ca-cert" pod "ab2d8d66-2c0e-4394-8e10-903c2868442a" (UID: "ab2d8d66-2c0e-4394-8e10-903c2868442a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:47:01.350689 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:47:01.350656 4927 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab2d8d66-2c0e-4394-8e10-903c2868442a-workload-socket\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:47:01.350689 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:47:01.350676 4927 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab2d8d66-2c0e-4394-8e10-903c2868442a-istiod-ca-cert\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:47:01.350689 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:47:01.350691 4927 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab2d8d66-2c0e-4394-8e10-903c2868442a-credential-socket\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:47:01.351090 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:47:01.350709 4927 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab2d8d66-2c0e-4394-8e10-903c2868442a-istio-data\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:47:01.351090 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:47:01.350653 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab2d8d66-2c0e-4394-8e10-903c2868442a-workload-certs" pod "ab2d8d66-2c0e-4394-8e10-903c2868442a" (UID: "ab2d8d66-2c0e-4394-8e10-903c2868442a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:47:01.352363 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:47:01.352322 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab2d8d66-2c0e-4394-8e10-903c2868442a-istio-envoy" pod "ab2d8d66-2c0e-4394-8e10-903c2868442a" (UID: "ab2d8d66-2c0e-4394-8e10-903c2868442a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:47:01.352476 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:47:01.352426 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab2d8d66-2c0e-4394-8e10-903c2868442a-istio-token" pod "ab2d8d66-2c0e-4394-8e10-903c2868442a" (UID: "ab2d8d66-2c0e-4394-8e10-903c2868442a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:47:01.352524 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:47:01.352485 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ab2d8d66-2c0e-4394-8e10-903c2868442a-istio-podinfo" pod "ab2d8d66-2c0e-4394-8e10-903c2868442a" (UID: "ab2d8d66-2c0e-4394-8e10-903c2868442a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:47:01.352524 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:47:01.352493 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab2d8d66-2c0e-4394-8e10-903c2868442a-kube-api-access-qq9n5" pod "ab2d8d66-2c0e-4394-8e10-903c2868442a" (UID: "ab2d8d66-2c0e-4394-8e10-903c2868442a"). InnerVolumeSpecName "kube-api-access-qq9n5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:47:01.451278 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:47:01.451244 4927 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab2d8d66-2c0e-4394-8e10-903c2868442a-workload-certs\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:47:01.451278 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:47:01.451273 4927 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab2d8d66-2c0e-4394-8e10-903c2868442a-istio-podinfo\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:47:01.451546 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:47:01.451285 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qq9n5\" (UniqueName: \"kubernetes.io/projected/ab2d8d66-2c0e-4394-8e10-903c2868442a-kube-api-access-qq9n5\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:47:01.451546 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:47:01.451301 4927 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab2d8d66-2c0e-4394-8e10-903c2868442a-istio-token\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:47:01.451546 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:47:01.451310 4927 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab2d8d66-2c0e-4394-8e10-903c2868442a-istio-envoy\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:47:02.168248 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:47:02.168206 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-e98014-openshift-default-596b484cdf5xsgc" event={"ID":"ab2d8d66-2c0e-4394-8e10-903c2868442a","Type":"ContainerDied","Data":"33a983b32ef89b12f44caa9c0690a62f6a78ad7853f8abe1ab72d1e35db4aaaf"} Oct 31 00:47:02.168736 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:47:02.168301 4927 scope.go:122] "RemoveContainer" containerID="71ba38239156ab54947afb04d056be69bb2513f35b3bc5235afec10f3b041112" Oct 31 00:47:02.168736 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:47:02.168225 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-e98014-openshift-default-596b484cdf5xsgc" Oct 31 00:47:02.338651 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:47:02.338615 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-e98014-openshift-default-596b484cdf5xsgc"] Oct 31 00:47:02.389343 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:47:02.389283 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-e98014-openshift-default-596b484cdf5xsgc"] Oct 31 00:47:03.085123 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:47:03.085088 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab2d8d66-2c0e-4394-8e10-903c2868442a" path="/var/lib/kubelet/pods/ab2d8d66-2c0e-4394-8e10-903c2868442a/volumes" Oct 31 00:47:03.622222 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:47:03.622187 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:19.677247 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:47:19.677201 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sqv69" containerName="extractor" Oct 31 00:47:21.049437 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:47:21.049401 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zmqlp"] Oct 31 00:47:21.100436 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:47:21.100397 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zmqlp"] Oct 31 00:47:21.312836 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:47:21.312746 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:22.412643 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:47:22.412603 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:23.085079 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:47:23.085038 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff7abf47-1bc8-47ec-92ef-33e82aeb1e92" path="/var/lib/kubelet/pods/ff7abf47-1bc8-47ec-92ef-33e82aeb1e92/volumes" Oct 31 00:47:28.157556 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:47:28.157514 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:41.020759 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:47:41.020719 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:41.322062 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:47:41.321980 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:48.029121 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:47:48.029085 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:49.677303 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:47:49.677263 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sqv69" containerName="extractor" Oct 31 00:47:55.536357 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:47:55.533441 4927 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-4sgzb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.16:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:47:55.536357 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:47:55.533524 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-4sgzb" podUID="531eadda-5f07-4116-8b88-6331bebe01c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.16:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:47:57.126435 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:47:57.126403 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:57.276130 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:47:57.276097 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:19.489976 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:48:19.489894 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-phspq"] Oct 31 00:48:19.490480 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:48:19.490344 4927 cpu_manager.go:414] "containerMap: removing container" podUID="794a46cb-f783-4a10-806e-b1fe3fb28c1b" containerName="cleanup" Oct 31 00:48:19.490480 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:48:19.490356 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="794a46cb-f783-4a10-806e-b1fe3fb28c1b" containerName="cleanup" Oct 31 00:48:19.490480 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:48:19.490371 4927 cpu_manager.go:414] "containerMap: removing container" podUID="ab2d8d66-2c0e-4394-8e10-903c2868442a" containerName="istio-proxy" Oct 31 00:48:19.490480 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:48:19.490377 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab2d8d66-2c0e-4394-8e10-903c2868442a" containerName="istio-proxy" Oct 31 00:48:19.490480 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:48:19.490387 4927 cpu_manager.go:414] "containerMap: removing container" podUID="39f9b16c-33a6-4253-a562-a6b5ec648f58" containerName="maas-api" Oct 31 00:48:19.490480 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:48:19.490392 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39f9b16c-33a6-4253-a562-a6b5ec648f58" containerName="maas-api" Oct 31 00:48:19.490480 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:48:19.490409 4927 cpu_manager.go:414] "containerMap: removing container" podUID="ab2d8d66-2c0e-4394-8e10-903c2868442a" containerName="istio-proxy" Oct 31 00:48:19.490480 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:48:19.490414 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab2d8d66-2c0e-4394-8e10-903c2868442a" containerName="istio-proxy" Oct 31 00:48:19.490480 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:48:19.490426 4927 cpu_manager.go:414] "containerMap: removing container" podUID="794a46cb-f783-4a10-806e-b1fe3fb28c1b" containerName="cleanup" Oct 31 00:48:19.490480 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:48:19.490431 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="794a46cb-f783-4a10-806e-b1fe3fb28c1b" containerName="cleanup" Oct 31 00:48:19.490480 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:48:19.490438 4927 cpu_manager.go:414] "containerMap: removing container" podUID="794a46cb-f783-4a10-806e-b1fe3fb28c1b" containerName="cleanup" Oct 31 00:48:19.490480 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:48:19.490442 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="794a46cb-f783-4a10-806e-b1fe3fb28c1b" containerName="cleanup" Oct 31 00:48:19.490887 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:48:19.490517 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab2d8d66-2c0e-4394-8e10-903c2868442a" containerName="istio-proxy" Oct 31 00:48:19.490887 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:48:19.490526 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="794a46cb-f783-4a10-806e-b1fe3fb28c1b" containerName="cleanup" Oct 31 00:48:19.490887 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:48:19.490533 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="794a46cb-f783-4a10-806e-b1fe3fb28c1b" containerName="cleanup" Oct 31 00:48:19.490887 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:48:19.490542 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="794a46cb-f783-4a10-806e-b1fe3fb28c1b" containerName="cleanup" Oct 31 00:48:19.490887 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:48:19.490549 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="39f9b16c-33a6-4253-a562-a6b5ec648f58" containerName="maas-api" Oct 31 00:48:19.493764 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:48:19.493746 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-phspq" Oct 31 00:48:19.547509 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:48:19.547476 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-phspq"] Oct 31 00:48:19.588611 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:48:19.588577 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7a250d6-a6d8-4823-b370-12de5742e2ef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-phspq\" (UID: \"c7a250d6-a6d8-4823-b370-12de5742e2ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-phspq" Oct 31 00:48:19.588764 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:48:19.588708 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9l42z\" (UniqueName: \"kubernetes.io/projected/c7a250d6-a6d8-4823-b370-12de5742e2ef-kube-api-access-9l42z\") pod \"maas-api-revoke-sub-452b6d7b-phspq\" (UID: \"c7a250d6-a6d8-4823-b370-12de5742e2ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-phspq" Oct 31 00:48:19.588764 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:48:19.588742 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7a250d6-a6d8-4823-b370-12de5742e2ef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-phspq\" (UID: \"c7a250d6-a6d8-4823-b370-12de5742e2ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-phspq" Oct 31 00:48:19.595740 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:48:19.595722 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:48:19.595844 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:48:19.595833 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4fdzs\"" Oct 31 00:48:19.678821 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:48:19.678786 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sqv69" containerName="extractor" Oct 31 00:48:19.689242 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:48:19.689217 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9l42z\" (UniqueName: \"kubernetes.io/projected/c7a250d6-a6d8-4823-b370-12de5742e2ef-kube-api-access-9l42z\") pod \"maas-api-revoke-sub-452b6d7b-phspq\" (UID: \"c7a250d6-a6d8-4823-b370-12de5742e2ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-phspq" Oct 31 00:48:19.689386 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:48:19.689252 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7a250d6-a6d8-4823-b370-12de5742e2ef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-phspq\" (UID: \"c7a250d6-a6d8-4823-b370-12de5742e2ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-phspq" Oct 31 00:48:19.689386 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:48:19.689292 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7a250d6-a6d8-4823-b370-12de5742e2ef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-phspq\" (UID: \"c7a250d6-a6d8-4823-b370-12de5742e2ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-phspq" Oct 31 00:48:19.690061 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:48:19.690039 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7a250d6-a6d8-4823-b370-12de5742e2ef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-phspq\" (UID: \"c7a250d6-a6d8-4823-b370-12de5742e2ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-phspq" Oct 31 00:48:19.691527 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:48:19.691508 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7a250d6-a6d8-4823-b370-12de5742e2ef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-phspq\" (UID: \"c7a250d6-a6d8-4823-b370-12de5742e2ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-phspq" Oct 31 00:48:19.751703 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:48:19.751630 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9l42z\" (UniqueName: \"kubernetes.io/projected/c7a250d6-a6d8-4823-b370-12de5742e2ef-kube-api-access-9l42z\") pod \"maas-api-revoke-sub-452b6d7b-phspq\" (UID: \"c7a250d6-a6d8-4823-b370-12de5742e2ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-phspq" Oct 31 00:48:19.806728 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:48:19.806696 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-phspq" Oct 31 00:48:20.180520 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:48:20.180494 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-phspq"] Oct 31 00:48:20.677230 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:48:20.677191 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-phspq" event={"ID":"c7a250d6-a6d8-4823-b370-12de5742e2ef","Type":"ContainerStarted","Data":"59a092d51a99166e1013f62d7b9b5ddeccf4eadb3b1de29f36387080cf562236"} Oct 31 00:48:20.677717 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:48:20.677273 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-phspq" event={"ID":"c7a250d6-a6d8-4823-b370-12de5742e2ef","Type":"ContainerStarted","Data":"52d1a216dd07de5cce8fb9fc93f60db65b3bd77b0b0b8c16d1bdf03e83920621"} Oct 31 00:48:20.834671 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:48:20.834571 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-phspq" podStartSLOduration=1.8345588309999998 podStartE2EDuration="1.834558831s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:48:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:48:20.833373207 +0000 UTC m=+4013.869704028" watchObservedRunningTime="2026-10-31 00:48:20.834558831 +0000 UTC m=+4013.870889652" Oct 31 00:48:21.686030 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:48:21.685974 4927 generic.go:356] "Generic (PLEG): container finished" podID="c7a250d6-a6d8-4823-b370-12de5742e2ef" containerID="59a092d51a99166e1013f62d7b9b5ddeccf4eadb3b1de29f36387080cf562236" exitCode=0 Oct 31 00:48:21.686580 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:48:21.686052 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-phspq" event={"ID":"c7a250d6-a6d8-4823-b370-12de5742e2ef","Type":"ContainerDied","Data":"59a092d51a99166e1013f62d7b9b5ddeccf4eadb3b1de29f36387080cf562236"} Oct 31 00:48:23.174369 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:48:23.172770 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-phspq" Oct 31 00:48:23.263820 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:48:23.262617 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7a250d6-a6d8-4823-b370-12de5742e2ef-kube-api-access-9l42z\" (UniqueName: \"kubernetes.io/projected/c7a250d6-a6d8-4823-b370-12de5742e2ef-kube-api-access-9l42z\") pod \"c7a250d6-a6d8-4823-b370-12de5742e2ef\" (UID: \"c7a250d6-a6d8-4823-b370-12de5742e2ef\") " Oct 31 00:48:23.263820 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:48:23.262687 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c7a250d6-a6d8-4823-b370-12de5742e2ef-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7a250d6-a6d8-4823-b370-12de5742e2ef-maas-api-openshift-ca\") pod \"c7a250d6-a6d8-4823-b370-12de5742e2ef\" (UID: \"c7a250d6-a6d8-4823-b370-12de5742e2ef\") " Oct 31 00:48:23.263820 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:48:23.262827 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c7a250d6-a6d8-4823-b370-12de5742e2ef-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7a250d6-a6d8-4823-b370-12de5742e2ef-maas-api-xks-ca\") pod \"c7a250d6-a6d8-4823-b370-12de5742e2ef\" (UID: \"c7a250d6-a6d8-4823-b370-12de5742e2ef\") " Oct 31 00:48:23.265677 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:48:23.265642 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7a250d6-a6d8-4823-b370-12de5742e2ef-maas-api-openshift-ca" pod "c7a250d6-a6d8-4823-b370-12de5742e2ef" (UID: "c7a250d6-a6d8-4823-b370-12de5742e2ef"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:48:23.273343 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:48:23.273301 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c7a250d6-a6d8-4823-b370-12de5742e2ef-maas-api-xks-ca" pod "c7a250d6-a6d8-4823-b370-12de5742e2ef" (UID: "c7a250d6-a6d8-4823-b370-12de5742e2ef"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:48:23.276779 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:48:23.276742 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7a250d6-a6d8-4823-b370-12de5742e2ef-kube-api-access-9l42z" pod "c7a250d6-a6d8-4823-b370-12de5742e2ef" (UID: "c7a250d6-a6d8-4823-b370-12de5742e2ef"). InnerVolumeSpecName "kube-api-access-9l42z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:48:23.369408 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:48:23.369022 4927 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7a250d6-a6d8-4823-b370-12de5742e2ef-maas-api-xks-ca\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:23.369408 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:48:23.369084 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9l42z\" (UniqueName: \"kubernetes.io/projected/c7a250d6-a6d8-4823-b370-12de5742e2ef-kube-api-access-9l42z\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:23.369408 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:48:23.369102 4927 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7a250d6-a6d8-4823-b370-12de5742e2ef-maas-api-openshift-ca\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:23.705729 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:48:23.705685 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-phspq" event={"ID":"c7a250d6-a6d8-4823-b370-12de5742e2ef","Type":"ContainerDied","Data":"52d1a216dd07de5cce8fb9fc93f60db65b3bd77b0b0b8c16d1bdf03e83920621"} Oct 31 00:48:23.705924 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:48:23.705748 4927 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="52d1a216dd07de5cce8fb9fc93f60db65b3bd77b0b0b8c16d1bdf03e83920621" Oct 31 00:48:23.705924 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:48:23.705698 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-phspq" Oct 31 00:48:32.428228 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:48:32.428194 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:34.417715 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:48:34.417683 4927 scope.go:122] "RemoveContainer" containerID="b2ba3ec6c398097ef182613f93f4ed52c0dd385c620fe383e5a69a56f644f442" Oct 31 00:48:37.229246 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:48:37.229202 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:41.378176 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:48:41.378130 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:41.563104 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:48:41.563061 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-m5sfq"] Oct 31 00:48:41.563598 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:48:41.563582 4927 cpu_manager.go:414] "containerMap: removing container" podUID="c7a250d6-a6d8-4823-b370-12de5742e2ef" containerName="revoke-keys" Oct 31 00:48:41.563655 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:48:41.563600 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7a250d6-a6d8-4823-b370-12de5742e2ef" containerName="revoke-keys" Oct 31 00:48:41.563698 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:48:41.563691 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab2d8d66-2c0e-4394-8e10-903c2868442a" containerName="istio-proxy" Oct 31 00:48:41.563735 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:48:41.563701 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7a250d6-a6d8-4823-b370-12de5742e2ef" containerName="revoke-keys" Oct 31 00:48:41.568309 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:48:41.568289 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-m5sfq" Oct 31 00:48:41.623257 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:48:41.623227 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-m5sfq"] Oct 31 00:48:41.670741 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:48:41.670715 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:48:41.670906 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:48:41.670773 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4fdzs\"" Oct 31 00:48:41.699406 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:48:41.699365 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5hnh\" (UniqueName: \"kubernetes.io/projected/1c9f8572-023e-4011-82db-5ead77ad989c-kube-api-access-n5hnh\") pod \"maas-api-revoke-sub-3e31a506-m5sfq\" (UID: \"1c9f8572-023e-4011-82db-5ead77ad989c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-m5sfq" Oct 31 00:48:41.699552 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:48:41.699471 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c9f8572-023e-4011-82db-5ead77ad989c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-m5sfq\" (UID: \"1c9f8572-023e-4011-82db-5ead77ad989c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-m5sfq" Oct 31 00:48:41.699603 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:48:41.699565 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c9f8572-023e-4011-82db-5ead77ad989c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-m5sfq\" (UID: \"1c9f8572-023e-4011-82db-5ead77ad989c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-m5sfq" Oct 31 00:48:41.800751 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:48:41.800712 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5hnh\" (UniqueName: \"kubernetes.io/projected/1c9f8572-023e-4011-82db-5ead77ad989c-kube-api-access-n5hnh\") pod \"maas-api-revoke-sub-3e31a506-m5sfq\" (UID: \"1c9f8572-023e-4011-82db-5ead77ad989c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-m5sfq" Oct 31 00:48:41.800955 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:48:41.800766 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c9f8572-023e-4011-82db-5ead77ad989c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-m5sfq\" (UID: \"1c9f8572-023e-4011-82db-5ead77ad989c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-m5sfq" Oct 31 00:48:41.800955 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:48:41.800807 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c9f8572-023e-4011-82db-5ead77ad989c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-m5sfq\" (UID: \"1c9f8572-023e-4011-82db-5ead77ad989c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-m5sfq" Oct 31 00:48:41.801617 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:48:41.801592 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c9f8572-023e-4011-82db-5ead77ad989c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-m5sfq\" (UID: \"1c9f8572-023e-4011-82db-5ead77ad989c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-m5sfq" Oct 31 00:48:41.803275 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:48:41.803251 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c9f8572-023e-4011-82db-5ead77ad989c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-m5sfq\" (UID: \"1c9f8572-023e-4011-82db-5ead77ad989c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-m5sfq" Oct 31 00:48:41.862870 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:48:41.862839 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5hnh\" (UniqueName: \"kubernetes.io/projected/1c9f8572-023e-4011-82db-5ead77ad989c-kube-api-access-n5hnh\") pod \"maas-api-revoke-sub-3e31a506-m5sfq\" (UID: \"1c9f8572-023e-4011-82db-5ead77ad989c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-m5sfq" Oct 31 00:48:41.883158 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:48:41.883123 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-m5sfq" Oct 31 00:48:42.259628 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:48:42.259591 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-m5sfq"] Oct 31 00:48:42.261300 ip-10-0-141-180 kubenswrapper[4927]: W1031 00:48:42.261270 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c9f8572_023e_4011_82db_5ead77ad989c.slice/crio-999b349a855e2b74e95ba3057fcfc050d9eeddc575575122e61ca7f884db2916 WatchSource:0}: Error finding container 999b349a855e2b74e95ba3057fcfc050d9eeddc575575122e61ca7f884db2916: Status 404 returned error can't find the container with id 999b349a855e2b74e95ba3057fcfc050d9eeddc575575122e61ca7f884db2916 Oct 31 00:48:42.830598 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:48:42.830507 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-m5sfq" event={"ID":"1c9f8572-023e-4011-82db-5ead77ad989c","Type":"ContainerStarted","Data":"a128c50dacc91cf97d58611c86e43572f88e3057988d1456575e25e72de82cb7"} Oct 31 00:48:42.830598 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:48:42.830570 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-m5sfq" event={"ID":"1c9f8572-023e-4011-82db-5ead77ad989c","Type":"ContainerStarted","Data":"999b349a855e2b74e95ba3057fcfc050d9eeddc575575122e61ca7f884db2916"} Oct 31 00:48:42.950735 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:48:42.950682 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-m5sfq" podStartSLOduration=1.950666527 podStartE2EDuration="1.950666527s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:48:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:48:42.948507077 +0000 UTC m=+4035.984837891" watchObservedRunningTime="2026-10-31 00:48:42.950666527 +0000 UTC m=+4035.986997349" Oct 31 00:48:43.840303 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:48:43.840267 4927 generic.go:356] "Generic (PLEG): container finished" podID="1c9f8572-023e-4011-82db-5ead77ad989c" containerID="a128c50dacc91cf97d58611c86e43572f88e3057988d1456575e25e72de82cb7" exitCode=0 Oct 31 00:48:43.840765 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:48:43.840367 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-m5sfq" event={"ID":"1c9f8572-023e-4011-82db-5ead77ad989c","Type":"ContainerDied","Data":"a128c50dacc91cf97d58611c86e43572f88e3057988d1456575e25e72de82cb7"} Oct 31 00:48:45.270941 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:48:45.270835 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-m5sfq" Oct 31 00:48:45.295492 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:48:45.295456 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c9f8572-023e-4011-82db-5ead77ad989c-kube-api-access-n5hnh\" (UniqueName: \"kubernetes.io/projected/1c9f8572-023e-4011-82db-5ead77ad989c-kube-api-access-n5hnh\") pod \"1c9f8572-023e-4011-82db-5ead77ad989c\" (UID: \"1c9f8572-023e-4011-82db-5ead77ad989c\") " Oct 31 00:48:45.295650 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:48:45.295528 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c9f8572-023e-4011-82db-5ead77ad989c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c9f8572-023e-4011-82db-5ead77ad989c-maas-api-openshift-ca\") pod \"1c9f8572-023e-4011-82db-5ead77ad989c\" (UID: \"1c9f8572-023e-4011-82db-5ead77ad989c\") " Oct 31 00:48:45.295650 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:48:45.295572 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1c9f8572-023e-4011-82db-5ead77ad989c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c9f8572-023e-4011-82db-5ead77ad989c-maas-api-xks-ca\") pod \"1c9f8572-023e-4011-82db-5ead77ad989c\" (UID: \"1c9f8572-023e-4011-82db-5ead77ad989c\") " Oct 31 00:48:45.296442 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:48:45.296412 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c9f8572-023e-4011-82db-5ead77ad989c-maas-api-openshift-ca" pod "1c9f8572-023e-4011-82db-5ead77ad989c" (UID: "1c9f8572-023e-4011-82db-5ead77ad989c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:48:45.297808 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:48:45.297788 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c9f8572-023e-4011-82db-5ead77ad989c-maas-api-xks-ca" pod "1c9f8572-023e-4011-82db-5ead77ad989c" (UID: "1c9f8572-023e-4011-82db-5ead77ad989c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:48:45.297904 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:48:45.297891 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c9f8572-023e-4011-82db-5ead77ad989c-kube-api-access-n5hnh" pod "1c9f8572-023e-4011-82db-5ead77ad989c" (UID: "1c9f8572-023e-4011-82db-5ead77ad989c"). InnerVolumeSpecName "kube-api-access-n5hnh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:48:45.396172 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:48:45.396134 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n5hnh\" (UniqueName: \"kubernetes.io/projected/1c9f8572-023e-4011-82db-5ead77ad989c-kube-api-access-n5hnh\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:45.396172 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:48:45.396164 4927 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c9f8572-023e-4011-82db-5ead77ad989c-maas-api-openshift-ca\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:45.396172 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:48:45.396177 4927 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c9f8572-023e-4011-82db-5ead77ad989c-maas-api-xks-ca\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:45.861136 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:48:45.861097 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-m5sfq" Oct 31 00:48:45.861136 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:48:45.861126 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-m5sfq" event={"ID":"1c9f8572-023e-4011-82db-5ead77ad989c","Type":"ContainerDied","Data":"999b349a855e2b74e95ba3057fcfc050d9eeddc575575122e61ca7f884db2916"} Oct 31 00:48:45.861394 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:48:45.861186 4927 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="999b349a855e2b74e95ba3057fcfc050d9eeddc575575122e61ca7f884db2916" Oct 31 00:48:49.678143 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:48:49.678100 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sqv69" containerName="extractor" Oct 31 00:49:14.297182 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:49:14.297140 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9add4343-4ml95"] Oct 31 00:49:14.297926 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:49:14.297902 4927 cpu_manager.go:414] "containerMap: removing container" podUID="1c9f8572-023e-4011-82db-5ead77ad989c" containerName="revoke-keys" Oct 31 00:49:14.297977 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:49:14.297931 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c9f8572-023e-4011-82db-5ead77ad989c" containerName="revoke-keys" Oct 31 00:49:14.298162 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:49:14.298144 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c9f8572-023e-4011-82db-5ead77ad989c" containerName="revoke-keys" Oct 31 00:49:14.302189 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:49:14.302168 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9add4343-4ml95" Oct 31 00:49:14.353597 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:49:14.353564 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9add4343-4ml95"] Oct 31 00:49:14.395937 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:49:14.395904 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nd7bc\" (UniqueName: \"kubernetes.io/projected/dfc759ea-ee8d-4964-904b-b118f9d30184-kube-api-access-nd7bc\") pod \"maas-api-revoke-sub-9add4343-4ml95\" (UID: \"dfc759ea-ee8d-4964-904b-b118f9d30184\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9add4343-4ml95" Oct 31 00:49:14.396113 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:49:14.395970 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dfc759ea-ee8d-4964-904b-b118f9d30184-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9add4343-4ml95\" (UID: \"dfc759ea-ee8d-4964-904b-b118f9d30184\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9add4343-4ml95" Oct 31 00:49:14.396162 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:49:14.396120 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dfc759ea-ee8d-4964-904b-b118f9d30184-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9add4343-4ml95\" (UID: \"dfc759ea-ee8d-4964-904b-b118f9d30184\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9add4343-4ml95" Oct 31 00:49:14.401369 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:49:14.401319 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:49:14.401510 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:49:14.401400 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4fdzs\"" Oct 31 00:49:14.497629 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:49:14.497594 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dfc759ea-ee8d-4964-904b-b118f9d30184-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9add4343-4ml95\" (UID: \"dfc759ea-ee8d-4964-904b-b118f9d30184\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9add4343-4ml95" Oct 31 00:49:14.497852 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:49:14.497662 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dfc759ea-ee8d-4964-904b-b118f9d30184-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9add4343-4ml95\" (UID: \"dfc759ea-ee8d-4964-904b-b118f9d30184\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9add4343-4ml95" Oct 31 00:49:14.497852 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:49:14.497703 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nd7bc\" (UniqueName: \"kubernetes.io/projected/dfc759ea-ee8d-4964-904b-b118f9d30184-kube-api-access-nd7bc\") pod \"maas-api-revoke-sub-9add4343-4ml95\" (UID: \"dfc759ea-ee8d-4964-904b-b118f9d30184\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9add4343-4ml95" Oct 31 00:49:14.498376 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:49:14.498347 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dfc759ea-ee8d-4964-904b-b118f9d30184-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9add4343-4ml95\" (UID: \"dfc759ea-ee8d-4964-904b-b118f9d30184\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9add4343-4ml95" Oct 31 00:49:14.500129 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:49:14.500106 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dfc759ea-ee8d-4964-904b-b118f9d30184-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9add4343-4ml95\" (UID: \"dfc759ea-ee8d-4964-904b-b118f9d30184\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9add4343-4ml95" Oct 31 00:49:14.560010 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:49:14.559946 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nd7bc\" (UniqueName: \"kubernetes.io/projected/dfc759ea-ee8d-4964-904b-b118f9d30184-kube-api-access-nd7bc\") pod \"maas-api-revoke-sub-9add4343-4ml95\" (UID: \"dfc759ea-ee8d-4964-904b-b118f9d30184\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9add4343-4ml95" Oct 31 00:49:14.619439 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:49:14.619410 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9add4343-4ml95" Oct 31 00:49:14.994318 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:49:14.994286 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9add4343-4ml95"] Oct 31 00:49:14.995508 ip-10-0-141-180 kubenswrapper[4927]: W1031 00:49:14.995481 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfc759ea_ee8d_4964_904b_b118f9d30184.slice/crio-1c1071fe93d4a5be8ea23dec0bc284e9fbbda0d52ae5a8afbee73bbde8b73e85 WatchSource:0}: Error finding container 1c1071fe93d4a5be8ea23dec0bc284e9fbbda0d52ae5a8afbee73bbde8b73e85: Status 404 returned error can't find the container with id 1c1071fe93d4a5be8ea23dec0bc284e9fbbda0d52ae5a8afbee73bbde8b73e85 Oct 31 00:49:15.062585 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:49:15.062551 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9add4343-4ml95" event={"ID":"dfc759ea-ee8d-4964-904b-b118f9d30184","Type":"ContainerStarted","Data":"1c1071fe93d4a5be8ea23dec0bc284e9fbbda0d52ae5a8afbee73bbde8b73e85"} Oct 31 00:49:16.069900 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:49:16.069814 4927 generic.go:356] "Generic (PLEG): container finished" podID="dfc759ea-ee8d-4964-904b-b118f9d30184" containerID="1b7b084d13d82a1c1bc7fda0e63d7e6820514c84e62faf8a72de57bf6f8f8862" exitCode=0 Oct 31 00:49:16.070259 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:49:16.069907 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9add4343-4ml95" event={"ID":"dfc759ea-ee8d-4964-904b-b118f9d30184","Type":"ContainerDied","Data":"1b7b084d13d82a1c1bc7fda0e63d7e6820514c84e62faf8a72de57bf6f8f8862"} Oct 31 00:49:17.216690 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:49:17.216661 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9add4343-4ml95" Oct 31 00:49:17.324834 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:49:17.324797 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dfc759ea-ee8d-4964-904b-b118f9d30184-kube-api-access-nd7bc\" (UniqueName: \"kubernetes.io/projected/dfc759ea-ee8d-4964-904b-b118f9d30184-kube-api-access-nd7bc\") pod \"dfc759ea-ee8d-4964-904b-b118f9d30184\" (UID: \"dfc759ea-ee8d-4964-904b-b118f9d30184\") " Oct 31 00:49:17.325021 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:49:17.324891 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dfc759ea-ee8d-4964-904b-b118f9d30184-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dfc759ea-ee8d-4964-904b-b118f9d30184-maas-api-xks-ca\") pod \"dfc759ea-ee8d-4964-904b-b118f9d30184\" (UID: \"dfc759ea-ee8d-4964-904b-b118f9d30184\") " Oct 31 00:49:17.325021 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:49:17.324929 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dfc759ea-ee8d-4964-904b-b118f9d30184-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dfc759ea-ee8d-4964-904b-b118f9d30184-maas-api-openshift-ca\") pod \"dfc759ea-ee8d-4964-904b-b118f9d30184\" (UID: \"dfc759ea-ee8d-4964-904b-b118f9d30184\") " Oct 31 00:49:17.325492 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:49:17.325464 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dfc759ea-ee8d-4964-904b-b118f9d30184-maas-api-openshift-ca" pod "dfc759ea-ee8d-4964-904b-b118f9d30184" (UID: "dfc759ea-ee8d-4964-904b-b118f9d30184"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:49:17.327095 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:49:17.327042 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dfc759ea-ee8d-4964-904b-b118f9d30184-maas-api-xks-ca" pod "dfc759ea-ee8d-4964-904b-b118f9d30184" (UID: "dfc759ea-ee8d-4964-904b-b118f9d30184"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:49:17.327195 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:49:17.327159 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfc759ea-ee8d-4964-904b-b118f9d30184-kube-api-access-nd7bc" pod "dfc759ea-ee8d-4964-904b-b118f9d30184" (UID: "dfc759ea-ee8d-4964-904b-b118f9d30184"). InnerVolumeSpecName "kube-api-access-nd7bc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:49:17.426580 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:49:17.426539 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nd7bc\" (UniqueName: \"kubernetes.io/projected/dfc759ea-ee8d-4964-904b-b118f9d30184-kube-api-access-nd7bc\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:17.426580 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:49:17.426577 4927 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dfc759ea-ee8d-4964-904b-b118f9d30184-maas-api-xks-ca\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:17.426580 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:49:17.426588 4927 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dfc759ea-ee8d-4964-904b-b118f9d30184-maas-api-openshift-ca\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:18.082237 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:49:18.082208 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9add4343-4ml95" Oct 31 00:49:18.082438 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:49:18.082244 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9add4343-4ml95" event={"ID":"dfc759ea-ee8d-4964-904b-b118f9d30184","Type":"ContainerDied","Data":"1c1071fe93d4a5be8ea23dec0bc284e9fbbda0d52ae5a8afbee73bbde8b73e85"} Oct 31 00:49:18.082438 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:49:18.082288 4927 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1c1071fe93d4a5be8ea23dec0bc284e9fbbda0d52ae5a8afbee73bbde8b73e85" Oct 31 00:49:19.677244 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:49:19.677209 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sqv69" containerName="extractor" Oct 31 00:49:25.059780 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:49:25.059743 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-mr9m7"] Oct 31 00:49:25.110471 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:49:25.110439 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-mr9m7"] Oct 31 00:49:27.085641 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:49:27.085608 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c6ffaa04-f498-4957-b1fb-e5d7ab62c1be" path="/var/lib/kubelet/pods/c6ffaa04-f498-4957-b1fb-e5d7ab62c1be/volumes" Oct 31 00:49:27.691584 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:49:27.691542 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-56bf677d9d-8d7m6"] Oct 31 00:49:27.692073 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:49:27.692053 4927 cpu_manager.go:414] "containerMap: removing container" podUID="dfc759ea-ee8d-4964-904b-b118f9d30184" containerName="revoke-keys" Oct 31 00:49:27.692183 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:49:27.692074 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dfc759ea-ee8d-4964-904b-b118f9d30184" containerName="revoke-keys" Oct 31 00:49:27.692247 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:49:27.692206 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="dfc759ea-ee8d-4964-904b-b118f9d30184" containerName="revoke-keys" Oct 31 00:49:27.695691 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:49:27.695670 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-56bf677d9d-8d7m6" Oct 31 00:49:27.752912 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:49:27.752872 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-56bf677d9d-8d7m6"] Oct 31 00:49:27.800356 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:49:27.800284 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 31 00:49:27.800356 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:49:27.800309 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Oct 31 00:49:27.800356 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:49:27.800355 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-2nk58\"" Oct 31 00:49:27.827780 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:49:27.827742 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb45d586-4ee8-47fb-8fcc-b98b539e45ed-maas-api-tls\") pod \"maas-api-56bf677d9d-8d7m6\" (UID: \"fb45d586-4ee8-47fb-8fcc-b98b539e45ed\") " pod="odh-ai-gateway-infra/maas-api-56bf677d9d-8d7m6" Oct 31 00:49:27.827963 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:49:27.827808 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-btlpw\" (UniqueName: \"kubernetes.io/projected/fb45d586-4ee8-47fb-8fcc-b98b539e45ed-kube-api-access-btlpw\") pod \"maas-api-56bf677d9d-8d7m6\" (UID: \"fb45d586-4ee8-47fb-8fcc-b98b539e45ed\") " pod="odh-ai-gateway-infra/maas-api-56bf677d9d-8d7m6" Oct 31 00:49:27.827963 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:49:27.827911 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb45d586-4ee8-47fb-8fcc-b98b539e45ed-metrics-certs\") pod \"maas-api-56bf677d9d-8d7m6\" (UID: \"fb45d586-4ee8-47fb-8fcc-b98b539e45ed\") " pod="odh-ai-gateway-infra/maas-api-56bf677d9d-8d7m6" Oct 31 00:49:27.827963 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:49:27.827936 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb45d586-4ee8-47fb-8fcc-b98b539e45ed-openshift-service-ca\") pod \"maas-api-56bf677d9d-8d7m6\" (UID: \"fb45d586-4ee8-47fb-8fcc-b98b539e45ed\") " pod="odh-ai-gateway-infra/maas-api-56bf677d9d-8d7m6" Oct 31 00:49:27.928756 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:49:27.928719 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb45d586-4ee8-47fb-8fcc-b98b539e45ed-maas-api-tls\") pod \"maas-api-56bf677d9d-8d7m6\" (UID: \"fb45d586-4ee8-47fb-8fcc-b98b539e45ed\") " pod="odh-ai-gateway-infra/maas-api-56bf677d9d-8d7m6" Oct 31 00:49:27.928956 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:49:27.928812 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btlpw\" (UniqueName: \"kubernetes.io/projected/fb45d586-4ee8-47fb-8fcc-b98b539e45ed-kube-api-access-btlpw\") pod \"maas-api-56bf677d9d-8d7m6\" (UID: \"fb45d586-4ee8-47fb-8fcc-b98b539e45ed\") " pod="odh-ai-gateway-infra/maas-api-56bf677d9d-8d7m6" Oct 31 00:49:27.928956 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:49:27.928870 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb45d586-4ee8-47fb-8fcc-b98b539e45ed-metrics-certs\") pod \"maas-api-56bf677d9d-8d7m6\" (UID: \"fb45d586-4ee8-47fb-8fcc-b98b539e45ed\") " pod="odh-ai-gateway-infra/maas-api-56bf677d9d-8d7m6" Oct 31 00:49:27.928956 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:49:27.928904 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb45d586-4ee8-47fb-8fcc-b98b539e45ed-openshift-service-ca\") pod \"maas-api-56bf677d9d-8d7m6\" (UID: \"fb45d586-4ee8-47fb-8fcc-b98b539e45ed\") " pod="odh-ai-gateway-infra/maas-api-56bf677d9d-8d7m6" Oct 31 00:49:27.929607 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:49:27.929581 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb45d586-4ee8-47fb-8fcc-b98b539e45ed-openshift-service-ca\") pod \"maas-api-56bf677d9d-8d7m6\" (UID: \"fb45d586-4ee8-47fb-8fcc-b98b539e45ed\") " pod="odh-ai-gateway-infra/maas-api-56bf677d9d-8d7m6" Oct 31 00:49:27.931157 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:49:27.931137 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb45d586-4ee8-47fb-8fcc-b98b539e45ed-maas-api-tls\") pod \"maas-api-56bf677d9d-8d7m6\" (UID: \"fb45d586-4ee8-47fb-8fcc-b98b539e45ed\") " pod="odh-ai-gateway-infra/maas-api-56bf677d9d-8d7m6" Oct 31 00:49:27.931320 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:49:27.931301 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb45d586-4ee8-47fb-8fcc-b98b539e45ed-metrics-certs\") pod \"maas-api-56bf677d9d-8d7m6\" (UID: \"fb45d586-4ee8-47fb-8fcc-b98b539e45ed\") " pod="odh-ai-gateway-infra/maas-api-56bf677d9d-8d7m6" Oct 31 00:49:27.991049 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:49:27.990976 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-btlpw\" (UniqueName: \"kubernetes.io/projected/fb45d586-4ee8-47fb-8fcc-b98b539e45ed-kube-api-access-btlpw\") pod \"maas-api-56bf677d9d-8d7m6\" (UID: \"fb45d586-4ee8-47fb-8fcc-b98b539e45ed\") " pod="odh-ai-gateway-infra/maas-api-56bf677d9d-8d7m6" Oct 31 00:49:28.008843 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:49:28.008813 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-56bf677d9d-8d7m6" Oct 31 00:49:28.385956 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:49:28.385927 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-56bf677d9d-8d7m6"] Oct 31 00:49:28.387604 ip-10-0-141-180 kubenswrapper[4927]: W1031 00:49:28.387578 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb45d586_4ee8_47fb_8fcc_b98b539e45ed.slice/crio-0e1fed0635f61d1cf5b6f4a100fadb260929665e125c068947adfaf4a9459061 WatchSource:0}: Error finding container 0e1fed0635f61d1cf5b6f4a100fadb260929665e125c068947adfaf4a9459061: Status 404 returned error can't find the container with id 0e1fed0635f61d1cf5b6f4a100fadb260929665e125c068947adfaf4a9459061 Oct 31 00:49:28.388900 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:49:28.388880 4927 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 00:49:29.144512 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:49:29.144472 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-56bf677d9d-8d7m6" event={"ID":"fb45d586-4ee8-47fb-8fcc-b98b539e45ed","Type":"ContainerStarted","Data":"0e1fed0635f61d1cf5b6f4a100fadb260929665e125c068947adfaf4a9459061"} Oct 31 00:49:30.152229 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:49:30.152190 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-56bf677d9d-8d7m6" event={"ID":"fb45d586-4ee8-47fb-8fcc-b98b539e45ed","Type":"ContainerStarted","Data":"b49dd9b9a9da11facd9be50690498c9fc94bafd5e8e44d18667c9fd140ac3927"} Oct 31 00:49:30.152645 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:49:30.152253 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-56bf677d9d-8d7m6" Oct 31 00:49:30.278274 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:49:30.278213 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-56bf677d9d-8d7m6" podStartSLOduration=2.376087191 podStartE2EDuration="3.278199252s" totalImagesPullingTime="902.112061ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:49:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:49:30.27690985 +0000 UTC m=+4083.313240672" watchObservedRunningTime="2026-10-31 00:49:30.278199252 +0000 UTC m=+4083.314530074" Oct 31 00:49:34.047843 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:49:34.047802 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:34.550053 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:49:34.550022 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-8214dff9-openshift-default-7b77bdd57c-zm5nl"] Oct 31 00:49:34.553818 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:49:34.553788 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-8214dff9-openshift-default-7b77bdd57c-zm5nl" Oct 31 00:49:34.592002 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:49:34.591293 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23475173-46b4-47ef-96d2-749432224e80-istio-podinfo\") pod \"e2e-ait-8214dff9-openshift-default-7b77bdd57c-zm5nl\" (UID: \"23475173-46b4-47ef-96d2-749432224e80\") " pod="openshift-ingress/e2e-ait-8214dff9-openshift-default-7b77bdd57c-zm5nl" Oct 31 00:49:34.592002 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:49:34.591377 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23475173-46b4-47ef-96d2-749432224e80-istio-data\") pod \"e2e-ait-8214dff9-openshift-default-7b77bdd57c-zm5nl\" (UID: \"23475173-46b4-47ef-96d2-749432224e80\") " pod="openshift-ingress/e2e-ait-8214dff9-openshift-default-7b77bdd57c-zm5nl" Oct 31 00:49:34.592002 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:49:34.591425 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23475173-46b4-47ef-96d2-749432224e80-credential-socket\") pod \"e2e-ait-8214dff9-openshift-default-7b77bdd57c-zm5nl\" (UID: \"23475173-46b4-47ef-96d2-749432224e80\") " pod="openshift-ingress/e2e-ait-8214dff9-openshift-default-7b77bdd57c-zm5nl" Oct 31 00:49:34.592002 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:49:34.591505 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23475173-46b4-47ef-96d2-749432224e80-workload-socket\") pod \"e2e-ait-8214dff9-openshift-default-7b77bdd57c-zm5nl\" (UID: \"23475173-46b4-47ef-96d2-749432224e80\") " pod="openshift-ingress/e2e-ait-8214dff9-openshift-default-7b77bdd57c-zm5nl" Oct 31 00:49:34.592002 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:49:34.591579 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23475173-46b4-47ef-96d2-749432224e80-istiod-ca-cert\") pod \"e2e-ait-8214dff9-openshift-default-7b77bdd57c-zm5nl\" (UID: \"23475173-46b4-47ef-96d2-749432224e80\") " pod="openshift-ingress/e2e-ait-8214dff9-openshift-default-7b77bdd57c-zm5nl" Oct 31 00:49:34.592002 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:49:34.591605 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23475173-46b4-47ef-96d2-749432224e80-workload-certs\") pod \"e2e-ait-8214dff9-openshift-default-7b77bdd57c-zm5nl\" (UID: \"23475173-46b4-47ef-96d2-749432224e80\") " pod="openshift-ingress/e2e-ait-8214dff9-openshift-default-7b77bdd57c-zm5nl" Oct 31 00:49:34.592002 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:49:34.591644 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23475173-46b4-47ef-96d2-749432224e80-istio-token\") pod \"e2e-ait-8214dff9-openshift-default-7b77bdd57c-zm5nl\" (UID: \"23475173-46b4-47ef-96d2-749432224e80\") " pod="openshift-ingress/e2e-ait-8214dff9-openshift-default-7b77bdd57c-zm5nl" Oct 31 00:49:34.592002 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:49:34.591690 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23475173-46b4-47ef-96d2-749432224e80-istio-envoy\") pod \"e2e-ait-8214dff9-openshift-default-7b77bdd57c-zm5nl\" (UID: \"23475173-46b4-47ef-96d2-749432224e80\") " pod="openshift-ingress/e2e-ait-8214dff9-openshift-default-7b77bdd57c-zm5nl" Oct 31 00:49:34.592002 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:49:34.591722 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gxqj6\" (UniqueName: \"kubernetes.io/projected/23475173-46b4-47ef-96d2-749432224e80-kube-api-access-gxqj6\") pod \"e2e-ait-8214dff9-openshift-default-7b77bdd57c-zm5nl\" (UID: \"23475173-46b4-47ef-96d2-749432224e80\") " pod="openshift-ingress/e2e-ait-8214dff9-openshift-default-7b77bdd57c-zm5nl" Oct 31 00:49:34.609294 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:49:34.607635 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-8214dff9-openshift-default-dockercfg-ddwcz\"" Oct 31 00:49:34.661851 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:49:34.661815 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:34.662417 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:49:34.662390 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-8214dff9-openshift-default-7b77bdd57c-zm5nl"] Oct 31 00:49:34.692793 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:49:34.692762 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23475173-46b4-47ef-96d2-749432224e80-istiod-ca-cert\") pod \"e2e-ait-8214dff9-openshift-default-7b77bdd57c-zm5nl\" (UID: \"23475173-46b4-47ef-96d2-749432224e80\") " pod="openshift-ingress/e2e-ait-8214dff9-openshift-default-7b77bdd57c-zm5nl" Oct 31 00:49:34.692950 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:49:34.692807 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23475173-46b4-47ef-96d2-749432224e80-workload-certs\") pod \"e2e-ait-8214dff9-openshift-default-7b77bdd57c-zm5nl\" (UID: \"23475173-46b4-47ef-96d2-749432224e80\") " pod="openshift-ingress/e2e-ait-8214dff9-openshift-default-7b77bdd57c-zm5nl" Oct 31 00:49:34.692950 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:49:34.692851 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23475173-46b4-47ef-96d2-749432224e80-istio-token\") pod \"e2e-ait-8214dff9-openshift-default-7b77bdd57c-zm5nl\" (UID: \"23475173-46b4-47ef-96d2-749432224e80\") " pod="openshift-ingress/e2e-ait-8214dff9-openshift-default-7b77bdd57c-zm5nl" Oct 31 00:49:34.692950 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:49:34.692892 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23475173-46b4-47ef-96d2-749432224e80-istio-envoy\") pod \"e2e-ait-8214dff9-openshift-default-7b77bdd57c-zm5nl\" (UID: \"23475173-46b4-47ef-96d2-749432224e80\") " pod="openshift-ingress/e2e-ait-8214dff9-openshift-default-7b77bdd57c-zm5nl" Oct 31 00:49:34.692950 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:49:34.692919 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxqj6\" (UniqueName: \"kubernetes.io/projected/23475173-46b4-47ef-96d2-749432224e80-kube-api-access-gxqj6\") pod \"e2e-ait-8214dff9-openshift-default-7b77bdd57c-zm5nl\" (UID: \"23475173-46b4-47ef-96d2-749432224e80\") " pod="openshift-ingress/e2e-ait-8214dff9-openshift-default-7b77bdd57c-zm5nl" Oct 31 00:49:34.693199 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:49:34.692964 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23475173-46b4-47ef-96d2-749432224e80-istio-podinfo\") pod \"e2e-ait-8214dff9-openshift-default-7b77bdd57c-zm5nl\" (UID: \"23475173-46b4-47ef-96d2-749432224e80\") " pod="openshift-ingress/e2e-ait-8214dff9-openshift-default-7b77bdd57c-zm5nl" Oct 31 00:49:34.693199 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:49:34.692996 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23475173-46b4-47ef-96d2-749432224e80-istio-data\") pod \"e2e-ait-8214dff9-openshift-default-7b77bdd57c-zm5nl\" (UID: \"23475173-46b4-47ef-96d2-749432224e80\") " pod="openshift-ingress/e2e-ait-8214dff9-openshift-default-7b77bdd57c-zm5nl" Oct 31 00:49:34.693199 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:49:34.693025 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23475173-46b4-47ef-96d2-749432224e80-credential-socket\") pod \"e2e-ait-8214dff9-openshift-default-7b77bdd57c-zm5nl\" (UID: \"23475173-46b4-47ef-96d2-749432224e80\") " pod="openshift-ingress/e2e-ait-8214dff9-openshift-default-7b77bdd57c-zm5nl" Oct 31 00:49:34.693199 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:49:34.693076 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23475173-46b4-47ef-96d2-749432224e80-workload-socket\") pod \"e2e-ait-8214dff9-openshift-default-7b77bdd57c-zm5nl\" (UID: \"23475173-46b4-47ef-96d2-749432224e80\") " pod="openshift-ingress/e2e-ait-8214dff9-openshift-default-7b77bdd57c-zm5nl" Oct 31 00:49:34.693626 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:49:34.693597 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23475173-46b4-47ef-96d2-749432224e80-workload-certs\") pod \"e2e-ait-8214dff9-openshift-default-7b77bdd57c-zm5nl\" (UID: \"23475173-46b4-47ef-96d2-749432224e80\") " pod="openshift-ingress/e2e-ait-8214dff9-openshift-default-7b77bdd57c-zm5nl" Oct 31 00:49:34.693950 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:49:34.693924 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23475173-46b4-47ef-96d2-749432224e80-credential-socket\") pod \"e2e-ait-8214dff9-openshift-default-7b77bdd57c-zm5nl\" (UID: \"23475173-46b4-47ef-96d2-749432224e80\") " pod="openshift-ingress/e2e-ait-8214dff9-openshift-default-7b77bdd57c-zm5nl" Oct 31 00:49:34.694071 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:49:34.693971 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23475173-46b4-47ef-96d2-749432224e80-workload-socket\") pod \"e2e-ait-8214dff9-openshift-default-7b77bdd57c-zm5nl\" (UID: \"23475173-46b4-47ef-96d2-749432224e80\") " pod="openshift-ingress/e2e-ait-8214dff9-openshift-default-7b77bdd57c-zm5nl" Oct 31 00:49:34.694071 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:49:34.694029 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23475173-46b4-47ef-96d2-749432224e80-istio-data\") pod \"e2e-ait-8214dff9-openshift-default-7b77bdd57c-zm5nl\" (UID: \"23475173-46b4-47ef-96d2-749432224e80\") " pod="openshift-ingress/e2e-ait-8214dff9-openshift-default-7b77bdd57c-zm5nl" Oct 31 00:49:34.694523 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:49:34.694506 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23475173-46b4-47ef-96d2-749432224e80-istiod-ca-cert\") pod \"e2e-ait-8214dff9-openshift-default-7b77bdd57c-zm5nl\" (UID: \"23475173-46b4-47ef-96d2-749432224e80\") " pod="openshift-ingress/e2e-ait-8214dff9-openshift-default-7b77bdd57c-zm5nl" Oct 31 00:49:34.695687 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:49:34.695660 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23475173-46b4-47ef-96d2-749432224e80-istio-envoy\") pod \"e2e-ait-8214dff9-openshift-default-7b77bdd57c-zm5nl\" (UID: \"23475173-46b4-47ef-96d2-749432224e80\") " pod="openshift-ingress/e2e-ait-8214dff9-openshift-default-7b77bdd57c-zm5nl" Oct 31 00:49:34.695947 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:49:34.695929 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23475173-46b4-47ef-96d2-749432224e80-istio-podinfo\") pod \"e2e-ait-8214dff9-openshift-default-7b77bdd57c-zm5nl\" (UID: \"23475173-46b4-47ef-96d2-749432224e80\") " pod="openshift-ingress/e2e-ait-8214dff9-openshift-default-7b77bdd57c-zm5nl" Oct 31 00:49:34.755259 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:49:34.755219 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxqj6\" (UniqueName: \"kubernetes.io/projected/23475173-46b4-47ef-96d2-749432224e80-kube-api-access-gxqj6\") pod \"e2e-ait-8214dff9-openshift-default-7b77bdd57c-zm5nl\" (UID: \"23475173-46b4-47ef-96d2-749432224e80\") " pod="openshift-ingress/e2e-ait-8214dff9-openshift-default-7b77bdd57c-zm5nl" Oct 31 00:49:34.755474 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:49:34.755286 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23475173-46b4-47ef-96d2-749432224e80-istio-token\") pod \"e2e-ait-8214dff9-openshift-default-7b77bdd57c-zm5nl\" (UID: \"23475173-46b4-47ef-96d2-749432224e80\") " pod="openshift-ingress/e2e-ait-8214dff9-openshift-default-7b77bdd57c-zm5nl" Oct 31 00:49:34.867377 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:49:34.867234 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-8214dff9-openshift-default-7b77bdd57c-zm5nl" Oct 31 00:49:34.875675 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:49:34.875637 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:35.301384 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:49:35.301358 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-8214dff9-openshift-default-7b77bdd57c-zm5nl"] Oct 31 00:49:35.302164 ip-10-0-141-180 kubenswrapper[4927]: W1031 00:49:35.302136 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23475173_46b4_47ef_96d2_749432224e80.slice/crio-6cb9abf4fd193f4db5f148ee4117b199c6cd59dfb4d56f7f1d0530c3a9076316 WatchSource:0}: Error finding container 6cb9abf4fd193f4db5f148ee4117b199c6cd59dfb4d56f7f1d0530c3a9076316: Status 404 returned error can't find the container with id 6cb9abf4fd193f4db5f148ee4117b199c6cd59dfb4d56f7f1d0530c3a9076316 Oct 31 00:49:35.304180 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:49:35.304146 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:49:35.304259 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:49:35.304207 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:49:35.304259 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:49:35.304235 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:49:36.164419 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:49:36.164385 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-56bf677d9d-8d7m6" Oct 31 00:49:36.189569 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:49:36.189528 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8214dff9-openshift-default-7b77bdd57c-zm5nl" event={"ID":"23475173-46b4-47ef-96d2-749432224e80","Type":"ContainerStarted","Data":"aaba13b584ce49287df1f2453b53b035616a5437a08c11cdeedce3aa56d27635"} Oct 31 00:49:36.189731 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:49:36.189589 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8214dff9-openshift-default-7b77bdd57c-zm5nl" event={"ID":"23475173-46b4-47ef-96d2-749432224e80","Type":"ContainerStarted","Data":"6cb9abf4fd193f4db5f148ee4117b199c6cd59dfb4d56f7f1d0530c3a9076316"} Oct 31 00:49:36.272786 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:49:36.272755 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-56bf677d9d-8d7m6"] Oct 31 00:49:36.272991 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:49:36.272969 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-56bf677d9d-8d7m6" podUID="fb45d586-4ee8-47fb-8fcc-b98b539e45ed" containerName="maas-api" containerID="cri-o://b49dd9b9a9da11facd9be50690498c9fc94bafd5e8e44d18667c9fd140ac3927" gracePeriod=30 Oct 31 00:49:36.325621 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:49:36.325586 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-7879ff96d6-v597r"] Oct 31 00:49:36.329221 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:49:36.329198 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7879ff96d6-v597r" Oct 31 00:49:36.383311 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:49:36.383273 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7879ff96d6-v597r"] Oct 31 00:49:36.413687 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:49:36.413653 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4adb025e-a773-40c6-af14-cb58fa6773ac-openshift-service-ca\") pod \"maas-api-7879ff96d6-v597r\" (UID: \"4adb025e-a773-40c6-af14-cb58fa6773ac\") " pod="odh-ai-gateway-infra/maas-api-7879ff96d6-v597r" Oct 31 00:49:36.413851 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:49:36.413736 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qbzgc\" (UniqueName: \"kubernetes.io/projected/4adb025e-a773-40c6-af14-cb58fa6773ac-kube-api-access-qbzgc\") pod \"maas-api-7879ff96d6-v597r\" (UID: \"4adb025e-a773-40c6-af14-cb58fa6773ac\") " pod="odh-ai-gateway-infra/maas-api-7879ff96d6-v597r" Oct 31 00:49:36.413922 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:49:36.413855 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4adb025e-a773-40c6-af14-cb58fa6773ac-maas-api-tls\") pod \"maas-api-7879ff96d6-v597r\" (UID: \"4adb025e-a773-40c6-af14-cb58fa6773ac\") " pod="odh-ai-gateway-infra/maas-api-7879ff96d6-v597r" Oct 31 00:49:36.414008 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:49:36.413927 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4adb025e-a773-40c6-af14-cb58fa6773ac-metrics-certs\") pod \"maas-api-7879ff96d6-v597r\" (UID: \"4adb025e-a773-40c6-af14-cb58fa6773ac\") " pod="odh-ai-gateway-infra/maas-api-7879ff96d6-v597r" Oct 31 00:49:36.514463 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:49:36.514431 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4adb025e-a773-40c6-af14-cb58fa6773ac-openshift-service-ca\") pod \"maas-api-7879ff96d6-v597r\" (UID: \"4adb025e-a773-40c6-af14-cb58fa6773ac\") " pod="odh-ai-gateway-infra/maas-api-7879ff96d6-v597r" Oct 31 00:49:36.514591 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:49:36.514524 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbzgc\" (UniqueName: \"kubernetes.io/projected/4adb025e-a773-40c6-af14-cb58fa6773ac-kube-api-access-qbzgc\") pod \"maas-api-7879ff96d6-v597r\" (UID: \"4adb025e-a773-40c6-af14-cb58fa6773ac\") " pod="odh-ai-gateway-infra/maas-api-7879ff96d6-v597r" Oct 31 00:49:36.514591 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:49:36.514555 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4adb025e-a773-40c6-af14-cb58fa6773ac-maas-api-tls\") pod \"maas-api-7879ff96d6-v597r\" (UID: \"4adb025e-a773-40c6-af14-cb58fa6773ac\") " pod="odh-ai-gateway-infra/maas-api-7879ff96d6-v597r" Oct 31 00:49:36.514681 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:49:36.514607 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4adb025e-a773-40c6-af14-cb58fa6773ac-metrics-certs\") pod \"maas-api-7879ff96d6-v597r\" (UID: \"4adb025e-a773-40c6-af14-cb58fa6773ac\") " pod="odh-ai-gateway-infra/maas-api-7879ff96d6-v597r" Oct 31 00:49:36.515094 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:49:36.515066 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4adb025e-a773-40c6-af14-cb58fa6773ac-openshift-service-ca\") pod \"maas-api-7879ff96d6-v597r\" (UID: \"4adb025e-a773-40c6-af14-cb58fa6773ac\") " pod="odh-ai-gateway-infra/maas-api-7879ff96d6-v597r" Oct 31 00:49:36.517076 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:49:36.517048 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4adb025e-a773-40c6-af14-cb58fa6773ac-metrics-certs\") pod \"maas-api-7879ff96d6-v597r\" (UID: \"4adb025e-a773-40c6-af14-cb58fa6773ac\") " pod="odh-ai-gateway-infra/maas-api-7879ff96d6-v597r" Oct 31 00:49:36.517223 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:49:36.517048 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4adb025e-a773-40c6-af14-cb58fa6773ac-maas-api-tls\") pod \"maas-api-7879ff96d6-v597r\" (UID: \"4adb025e-a773-40c6-af14-cb58fa6773ac\") " pod="odh-ai-gateway-infra/maas-api-7879ff96d6-v597r" Oct 31 00:49:36.539057 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:49:36.539039 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-56bf677d9d-8d7m6" Oct 31 00:49:36.566146 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:49:36.566103 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-8214dff9-openshift-default-7b77bdd57c-zm5nl" podStartSLOduration=2.566089958 podStartE2EDuration="2.566089958s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:49:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:49:36.564490496 +0000 UTC m=+4089.600821348" watchObservedRunningTime="2026-10-31 00:49:36.566089958 +0000 UTC m=+4089.602420779" Oct 31 00:49:36.578195 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:49:36.578165 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbzgc\" (UniqueName: \"kubernetes.io/projected/4adb025e-a773-40c6-af14-cb58fa6773ac-kube-api-access-qbzgc\") pod \"maas-api-7879ff96d6-v597r\" (UID: \"4adb025e-a773-40c6-af14-cb58fa6773ac\") " pod="odh-ai-gateway-infra/maas-api-7879ff96d6-v597r" Oct 31 00:49:36.614904 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:49:36.614874 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fb45d586-4ee8-47fb-8fcc-b98b539e45ed-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb45d586-4ee8-47fb-8fcc-b98b539e45ed-maas-api-tls\") pod \"fb45d586-4ee8-47fb-8fcc-b98b539e45ed\" (UID: \"fb45d586-4ee8-47fb-8fcc-b98b539e45ed\") " Oct 31 00:49:36.615086 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:49:36.614922 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fb45d586-4ee8-47fb-8fcc-b98b539e45ed-metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb45d586-4ee8-47fb-8fcc-b98b539e45ed-metrics-certs\") pod \"fb45d586-4ee8-47fb-8fcc-b98b539e45ed\" (UID: \"fb45d586-4ee8-47fb-8fcc-b98b539e45ed\") " Oct 31 00:49:36.615086 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:49:36.615025 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb45d586-4ee8-47fb-8fcc-b98b539e45ed-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb45d586-4ee8-47fb-8fcc-b98b539e45ed-openshift-service-ca\") pod \"fb45d586-4ee8-47fb-8fcc-b98b539e45ed\" (UID: \"fb45d586-4ee8-47fb-8fcc-b98b539e45ed\") " Oct 31 00:49:36.615086 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:49:36.615064 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb45d586-4ee8-47fb-8fcc-b98b539e45ed-kube-api-access-btlpw\" (UniqueName: \"kubernetes.io/projected/fb45d586-4ee8-47fb-8fcc-b98b539e45ed-kube-api-access-btlpw\") pod \"fb45d586-4ee8-47fb-8fcc-b98b539e45ed\" (UID: \"fb45d586-4ee8-47fb-8fcc-b98b539e45ed\") " Oct 31 00:49:36.615443 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:49:36.615418 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb45d586-4ee8-47fb-8fcc-b98b539e45ed-openshift-service-ca" pod "fb45d586-4ee8-47fb-8fcc-b98b539e45ed" (UID: "fb45d586-4ee8-47fb-8fcc-b98b539e45ed"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:49:36.617112 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:49:36.617086 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb45d586-4ee8-47fb-8fcc-b98b539e45ed-maas-api-tls" pod "fb45d586-4ee8-47fb-8fcc-b98b539e45ed" (UID: "fb45d586-4ee8-47fb-8fcc-b98b539e45ed"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:49:36.617220 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:49:36.617187 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb45d586-4ee8-47fb-8fcc-b98b539e45ed-metrics-certs" pod "fb45d586-4ee8-47fb-8fcc-b98b539e45ed" (UID: "fb45d586-4ee8-47fb-8fcc-b98b539e45ed"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:49:36.617220 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:49:36.617188 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb45d586-4ee8-47fb-8fcc-b98b539e45ed-kube-api-access-btlpw" pod "fb45d586-4ee8-47fb-8fcc-b98b539e45ed" (UID: "fb45d586-4ee8-47fb-8fcc-b98b539e45ed"). InnerVolumeSpecName "kube-api-access-btlpw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:49:36.669540 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:49:36.669509 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7879ff96d6-v597r" Oct 31 00:49:36.719061 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:49:36.719022 4927 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb45d586-4ee8-47fb-8fcc-b98b539e45ed-openshift-service-ca\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:36.719238 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:49:36.719064 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-btlpw\" (UniqueName: \"kubernetes.io/projected/fb45d586-4ee8-47fb-8fcc-b98b539e45ed-kube-api-access-btlpw\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:36.719238 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:49:36.719100 4927 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb45d586-4ee8-47fb-8fcc-b98b539e45ed-maas-api-tls\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:36.719238 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:49:36.719116 4927 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb45d586-4ee8-47fb-8fcc-b98b539e45ed-metrics-certs\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:36.868399 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:49:36.868364 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-8214dff9-openshift-default-7b77bdd57c-zm5nl" Oct 31 00:49:36.873427 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:49:36.873404 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-8214dff9-openshift-default-7b77bdd57c-zm5nl" Oct 31 00:49:37.119302 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:49:37.119278 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7879ff96d6-v597r"] Oct 31 00:49:37.121199 ip-10-0-141-180 kubenswrapper[4927]: W1031 00:49:37.121169 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4adb025e_a773_40c6_af14_cb58fa6773ac.slice/crio-0361e5b6807b7af7a65a22177644eb600653e27a586c2b76258f722f52795597 WatchSource:0}: Error finding container 0361e5b6807b7af7a65a22177644eb600653e27a586c2b76258f722f52795597: Status 404 returned error can't find the container with id 0361e5b6807b7af7a65a22177644eb600653e27a586c2b76258f722f52795597 Oct 31 00:49:37.196405 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:49:37.196368 4927 generic.go:356] "Generic (PLEG): container finished" podID="fb45d586-4ee8-47fb-8fcc-b98b539e45ed" containerID="b49dd9b9a9da11facd9be50690498c9fc94bafd5e8e44d18667c9fd140ac3927" exitCode=0 Oct 31 00:49:37.196551 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:49:37.196455 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-56bf677d9d-8d7m6" Oct 31 00:49:37.196551 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:49:37.196491 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-56bf677d9d-8d7m6" event={"ID":"fb45d586-4ee8-47fb-8fcc-b98b539e45ed","Type":"ContainerDied","Data":"b49dd9b9a9da11facd9be50690498c9fc94bafd5e8e44d18667c9fd140ac3927"} Oct 31 00:49:37.196551 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:49:37.196543 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-56bf677d9d-8d7m6" event={"ID":"fb45d586-4ee8-47fb-8fcc-b98b539e45ed","Type":"ContainerDied","Data":"0e1fed0635f61d1cf5b6f4a100fadb260929665e125c068947adfaf4a9459061"} Oct 31 00:49:37.196751 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:49:37.196576 4927 scope.go:122] "RemoveContainer" containerID="b49dd9b9a9da11facd9be50690498c9fc94bafd5e8e44d18667c9fd140ac3927" Oct 31 00:49:37.198004 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:49:37.197981 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7879ff96d6-v597r" event={"ID":"4adb025e-a773-40c6-af14-cb58fa6773ac","Type":"ContainerStarted","Data":"0361e5b6807b7af7a65a22177644eb600653e27a586c2b76258f722f52795597"} Oct 31 00:49:37.198216 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:49:37.198196 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-8214dff9-openshift-default-7b77bdd57c-zm5nl" Oct 31 00:49:37.199454 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:49:37.199434 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-8214dff9-openshift-default-7b77bdd57c-zm5nl" Oct 31 00:49:37.206164 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:49:37.206148 4927 scope.go:122] "RemoveContainer" containerID="b49dd9b9a9da11facd9be50690498c9fc94bafd5e8e44d18667c9fd140ac3927" Oct 31 00:49:37.206463 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:49:37.206443 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b49dd9b9a9da11facd9be50690498c9fc94bafd5e8e44d18667c9fd140ac3927\": container with ID starting with b49dd9b9a9da11facd9be50690498c9fc94bafd5e8e44d18667c9fd140ac3927 not found: ID does not exist" containerID="b49dd9b9a9da11facd9be50690498c9fc94bafd5e8e44d18667c9fd140ac3927" Oct 31 00:49:37.206523 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:49:37.206468 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b49dd9b9a9da11facd9be50690498c9fc94bafd5e8e44d18667c9fd140ac3927"} err="failed to get container status \"b49dd9b9a9da11facd9be50690498c9fc94bafd5e8e44d18667c9fd140ac3927\": rpc error: code = NotFound desc = could not find container \"b49dd9b9a9da11facd9be50690498c9fc94bafd5e8e44d18667c9fd140ac3927\": container with ID starting with b49dd9b9a9da11facd9be50690498c9fc94bafd5e8e44d18667c9fd140ac3927 not found: ID does not exist" Oct 31 00:49:37.489083 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:49:37.489049 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-56bf677d9d-8d7m6"] Oct 31 00:49:37.491623 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:49:37.491599 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-56bf677d9d-8d7m6"] Oct 31 00:49:38.206262 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:49:38.206223 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7879ff96d6-v597r" event={"ID":"4adb025e-a773-40c6-af14-cb58fa6773ac","Type":"ContainerStarted","Data":"d888df1f5ccf0f021a5aa8f29a8fa7bcea71bbaf400f7574d8f60e863c6431f6"} Oct 31 00:49:38.326477 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:49:38.326376 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-7879ff96d6-v597r" podStartSLOduration=1.41508398 podStartE2EDuration="2.326362168s" totalImagesPullingTime="911.278188ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:49:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:49:38.325892853 +0000 UTC m=+4091.362223677" watchObservedRunningTime="2026-10-31 00:49:38.326362168 +0000 UTC m=+4091.362692980" Oct 31 00:49:38.751266 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:49:38.751226 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:38.947900 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:49:38.947865 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:39.085849 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:49:39.085775 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb45d586-4ee8-47fb-8fcc-b98b539e45ed" path="/var/lib/kubelet/pods/fb45d586-4ee8-47fb-8fcc-b98b539e45ed/volumes" Oct 31 00:49:39.212079 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:49:39.212040 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-7879ff96d6-v597r" Oct 31 00:49:41.723102 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:49:41.723071 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:41.912169 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:49:41.912134 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:42.075875 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:49:42.075779 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:43.870481 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:49:43.870445 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:44.065809 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:49:44.065768 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:44.269929 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:49:44.269898 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:45.223785 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:49:45.223752 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-7879ff96d6-v597r" Oct 31 00:49:49.286075 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:49:49.286036 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:49.678537 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:49:49.678499 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sqv69" containerName="extractor" Oct 31 00:49:54.051051 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:49:54.051013 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-7zlnn"] Oct 31 00:49:54.094983 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:49:54.094945 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-7zlnn"] Oct 31 00:49:54.371866 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:49:54.371772 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:55.085223 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:49:55.085187 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="767625ba-836a-4aad-9960-5ed7bf134250" path="/var/lib/kubelet/pods/767625ba-836a-4aad-9960-5ed7bf134250/volumes" Oct 31 00:49:59.378509 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:49:59.378467 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:02.383439 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:50:02.383403 4927 generic.go:356] "Generic (PLEG): container finished" podID="23475173-46b4-47ef-96d2-749432224e80" containerID="aaba13b584ce49287df1f2453b53b035616a5437a08c11cdeedce3aa56d27635" exitCode=0 Oct 31 00:50:02.383917 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:50:02.383479 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8214dff9-openshift-default-7b77bdd57c-zm5nl" event={"ID":"23475173-46b4-47ef-96d2-749432224e80","Type":"ContainerDied","Data":"aaba13b584ce49287df1f2453b53b035616a5437a08c11cdeedce3aa56d27635"} Oct 31 00:50:02.383917 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:50:02.383888 4927 scope.go:122] "RemoveContainer" containerID="aaba13b584ce49287df1f2453b53b035616a5437a08c11cdeedce3aa56d27635" Oct 31 00:50:02.385112 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:50:02.385082 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:50:02.385230 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:50:02.385172 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:50:02.385306 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:50:02.385226 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:50:03.391961 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:50:03.391926 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8214dff9-openshift-default-7b77bdd57c-zm5nl" event={"ID":"23475173-46b4-47ef-96d2-749432224e80","Type":"ContainerStarted","Data":"ec59dca1cc659cf5a60d16c3123085a8b6ddf1ea6d73bb55fa4172fdde57906c"} Oct 31 00:50:03.867815 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:50:03.867774 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-8214dff9-openshift-default-7b77bdd57c-zm5nl" Oct 31 00:50:03.868929 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:50:03.868902 4927 patch_prober.go:29] interesting pod/e2e-ait-8214dff9-openshift-default-7b77bdd57c-zm5nl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.45:15021/healthz/ready\": dial tcp 10.132.1.45:15021: connect: connection refused" start-of-body= Oct 31 00:50:03.869043 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:50:03.868951 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-8214dff9-openshift-default-7b77bdd57c-zm5nl" podUID="23475173-46b4-47ef-96d2-749432224e80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.45:15021/healthz/ready\": dial tcp 10.132.1.45:15021: connect: connection refused" Oct 31 00:50:04.801319 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:50:04.801282 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:04.868442 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:50:04.868397 4927 patch_prober.go:29] interesting pod/e2e-ait-8214dff9-openshift-default-7b77bdd57c-zm5nl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.45:15021/healthz/ready\": dial tcp 10.132.1.45:15021: connect: connection refused" start-of-body= Oct 31 00:50:04.868653 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:50:04.868460 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-8214dff9-openshift-default-7b77bdd57c-zm5nl" podUID="23475173-46b4-47ef-96d2-749432224e80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.45:15021/healthz/ready\": dial tcp 10.132.1.45:15021: connect: connection refused" Oct 31 00:50:05.868178 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:50:05.868140 4927 patch_prober.go:29] interesting pod/e2e-ait-8214dff9-openshift-default-7b77bdd57c-zm5nl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.45:15021/healthz/ready\": dial tcp 10.132.1.45:15021: connect: connection refused" start-of-body= Oct 31 00:50:05.868618 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:50:05.868191 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-8214dff9-openshift-default-7b77bdd57c-zm5nl" podUID="23475173-46b4-47ef-96d2-749432224e80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.45:15021/healthz/ready\": dial tcp 10.132.1.45:15021: connect: connection refused" Oct 31 00:50:06.867818 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:50:06.867781 4927 patch_prober.go:29] interesting pod/e2e-ait-8214dff9-openshift-default-7b77bdd57c-zm5nl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.45:15021/healthz/ready\": dial tcp 10.132.1.45:15021: connect: connection refused" start-of-body= Oct 31 00:50:06.868034 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:50:06.867841 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-8214dff9-openshift-default-7b77bdd57c-zm5nl" podUID="23475173-46b4-47ef-96d2-749432224e80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.45:15021/healthz/ready\": dial tcp 10.132.1.45:15021: connect: connection refused" Oct 31 00:50:07.198706 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:50:07.198671 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-8214dff9-openshift-default-7b77bdd57c-zm5nl" Oct 31 00:50:07.868138 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:50:07.868097 4927 patch_prober.go:29] interesting pod/e2e-ait-8214dff9-openshift-default-7b77bdd57c-zm5nl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.45:15021/healthz/ready\": dial tcp 10.132.1.45:15021: connect: connection refused" start-of-body= Oct 31 00:50:07.868390 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:50:07.868174 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-8214dff9-openshift-default-7b77bdd57c-zm5nl" podUID="23475173-46b4-47ef-96d2-749432224e80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.45:15021/healthz/ready\": dial tcp 10.132.1.45:15021: connect: connection refused" Oct 31 00:50:08.871892 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:50:08.871855 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-8214dff9-openshift-default-7b77bdd57c-zm5nl" Oct 31 00:50:08.872889 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:50:08.872867 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-8214dff9-openshift-default-7b77bdd57c-zm5nl" Oct 31 00:50:10.106688 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:50:10.106645 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:15.186953 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:50:15.186907 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:19.679087 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:50:19.679039 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sqv69" containerName="extractor" Oct 31 00:50:20.176392 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:50:20.176353 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:25.342356 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:50:25.340278 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:26.983755 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:50:26.983689 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:27.153030 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:50:27.152986 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:30.189389 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:50:30.189348 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:34.450578 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:50:34.450545 4927 scope.go:122] "RemoveContainer" containerID="0b511dbef5f408064d1fa907830655d0382d4efc7bcd2809b6f187658c07a94b" Oct 31 00:50:34.595019 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:50:34.594982 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:36.244441 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:50:36.244408 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:36.528697 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:50:36.528614 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:41.347729 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:50:41.347698 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-8214dff9-openshift-default-7b77bdd57c-zm5nl"] Oct 31 00:50:41.348185 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:50:41.347890 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-8214dff9-openshift-default-7b77bdd57c-zm5nl" podUID="23475173-46b4-47ef-96d2-749432224e80" containerName="istio-proxy" containerID="cri-o://ec59dca1cc659cf5a60d16c3123085a8b6ddf1ea6d73bb55fa4172fdde57906c" gracePeriod=30 Oct 31 00:50:41.352913 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:50:41.352883 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:42.068298 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:50:42.068269 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:42.362557 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:50:42.362458 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:42.612081 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:50:42.612037 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:43.349851 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:50:43.349816 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:43.500183 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:50:43.500146 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:43.744549 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:50:43.744512 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:46.608092 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:50:46.608069 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-8214dff9-openshift-default-7b77bdd57c-zm5nl" Oct 31 00:50:46.690288 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:50:46.690253 4927 generic.go:356] "Generic (PLEG): container finished" podID="23475173-46b4-47ef-96d2-749432224e80" containerID="ec59dca1cc659cf5a60d16c3123085a8b6ddf1ea6d73bb55fa4172fdde57906c" exitCode=0 Oct 31 00:50:46.690513 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:50:46.690314 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-8214dff9-openshift-default-7b77bdd57c-zm5nl" Oct 31 00:50:46.690513 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:50:46.690362 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8214dff9-openshift-default-7b77bdd57c-zm5nl" event={"ID":"23475173-46b4-47ef-96d2-749432224e80","Type":"ContainerDied","Data":"ec59dca1cc659cf5a60d16c3123085a8b6ddf1ea6d73bb55fa4172fdde57906c"} Oct 31 00:50:46.690513 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:50:46.690435 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8214dff9-openshift-default-7b77bdd57c-zm5nl" event={"ID":"23475173-46b4-47ef-96d2-749432224e80","Type":"ContainerDied","Data":"6cb9abf4fd193f4db5f148ee4117b199c6cd59dfb4d56f7f1d0530c3a9076316"} Oct 31 00:50:46.690513 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:50:46.690467 4927 scope.go:122] "RemoveContainer" containerID="ec59dca1cc659cf5a60d16c3123085a8b6ddf1ea6d73bb55fa4172fdde57906c" Oct 31 00:50:46.700044 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:50:46.700025 4927 scope.go:122] "RemoveContainer" containerID="aaba13b584ce49287df1f2453b53b035616a5437a08c11cdeedce3aa56d27635" Oct 31 00:50:46.708182 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:50:46.708161 4927 scope.go:122] "RemoveContainer" containerID="ec59dca1cc659cf5a60d16c3123085a8b6ddf1ea6d73bb55fa4172fdde57906c" Oct 31 00:50:46.708492 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:50:46.708468 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec59dca1cc659cf5a60d16c3123085a8b6ddf1ea6d73bb55fa4172fdde57906c\": container with ID starting with ec59dca1cc659cf5a60d16c3123085a8b6ddf1ea6d73bb55fa4172fdde57906c not found: ID does not exist" containerID="ec59dca1cc659cf5a60d16c3123085a8b6ddf1ea6d73bb55fa4172fdde57906c" Oct 31 00:50:46.708546 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:50:46.708500 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec59dca1cc659cf5a60d16c3123085a8b6ddf1ea6d73bb55fa4172fdde57906c"} err="failed to get container status \"ec59dca1cc659cf5a60d16c3123085a8b6ddf1ea6d73bb55fa4172fdde57906c\": rpc error: code = NotFound desc = could not find container \"ec59dca1cc659cf5a60d16c3123085a8b6ddf1ea6d73bb55fa4172fdde57906c\": container with ID starting with ec59dca1cc659cf5a60d16c3123085a8b6ddf1ea6d73bb55fa4172fdde57906c not found: ID does not exist" Oct 31 00:50:46.708546 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:50:46.708518 4927 scope.go:122] "RemoveContainer" containerID="aaba13b584ce49287df1f2453b53b035616a5437a08c11cdeedce3aa56d27635" Oct 31 00:50:46.708806 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:50:46.708774 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aaba13b584ce49287df1f2453b53b035616a5437a08c11cdeedce3aa56d27635\": container with ID starting with aaba13b584ce49287df1f2453b53b035616a5437a08c11cdeedce3aa56d27635 not found: ID does not exist" containerID="aaba13b584ce49287df1f2453b53b035616a5437a08c11cdeedce3aa56d27635" Oct 31 00:50:46.708855 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:50:46.708808 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aaba13b584ce49287df1f2453b53b035616a5437a08c11cdeedce3aa56d27635"} err="failed to get container status \"aaba13b584ce49287df1f2453b53b035616a5437a08c11cdeedce3aa56d27635\": rpc error: code = NotFound desc = could not find container \"aaba13b584ce49287df1f2453b53b035616a5437a08c11cdeedce3aa56d27635\": container with ID starting with aaba13b584ce49287df1f2453b53b035616a5437a08c11cdeedce3aa56d27635 not found: ID does not exist" Oct 31 00:50:46.712920 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:50:46.712896 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23475173-46b4-47ef-96d2-749432224e80-istio-token\" (UniqueName: \"kubernetes.io/projected/23475173-46b4-47ef-96d2-749432224e80-istio-token\") pod \"23475173-46b4-47ef-96d2-749432224e80\" (UID: \"23475173-46b4-47ef-96d2-749432224e80\") " Oct 31 00:50:46.712996 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:50:46.712943 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23475173-46b4-47ef-96d2-749432224e80-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23475173-46b4-47ef-96d2-749432224e80-istio-data\") pod \"23475173-46b4-47ef-96d2-749432224e80\" (UID: \"23475173-46b4-47ef-96d2-749432224e80\") " Oct 31 00:50:46.713041 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:50:46.713018 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23475173-46b4-47ef-96d2-749432224e80-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23475173-46b4-47ef-96d2-749432224e80-workload-socket\") pod \"23475173-46b4-47ef-96d2-749432224e80\" (UID: \"23475173-46b4-47ef-96d2-749432224e80\") " Oct 31 00:50:46.713079 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:50:46.713067 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/23475173-46b4-47ef-96d2-749432224e80-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23475173-46b4-47ef-96d2-749432224e80-istio-podinfo\") pod \"23475173-46b4-47ef-96d2-749432224e80\" (UID: \"23475173-46b4-47ef-96d2-749432224e80\") " Oct 31 00:50:46.713130 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:50:46.713098 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23475173-46b4-47ef-96d2-749432224e80-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23475173-46b4-47ef-96d2-749432224e80-istio-envoy\") pod \"23475173-46b4-47ef-96d2-749432224e80\" (UID: \"23475173-46b4-47ef-96d2-749432224e80\") " Oct 31 00:50:46.713196 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:50:46.713155 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23475173-46b4-47ef-96d2-749432224e80-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23475173-46b4-47ef-96d2-749432224e80-workload-certs\") pod \"23475173-46b4-47ef-96d2-749432224e80\" (UID: \"23475173-46b4-47ef-96d2-749432224e80\") " Oct 31 00:50:46.713251 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:50:46.713232 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23475173-46b4-47ef-96d2-749432224e80-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23475173-46b4-47ef-96d2-749432224e80-credential-socket\") pod \"23475173-46b4-47ef-96d2-749432224e80\" (UID: \"23475173-46b4-47ef-96d2-749432224e80\") " Oct 31 00:50:46.713310 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:50:46.713295 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23475173-46b4-47ef-96d2-749432224e80-kube-api-access-gxqj6\" (UniqueName: \"kubernetes.io/projected/23475173-46b4-47ef-96d2-749432224e80-kube-api-access-gxqj6\") pod \"23475173-46b4-47ef-96d2-749432224e80\" (UID: \"23475173-46b4-47ef-96d2-749432224e80\") " Oct 31 00:50:46.713310 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:50:46.713293 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23475173-46b4-47ef-96d2-749432224e80-istio-data" pod "23475173-46b4-47ef-96d2-749432224e80" (UID: "23475173-46b4-47ef-96d2-749432224e80"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:50:46.713461 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:50:46.713323 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23475173-46b4-47ef-96d2-749432224e80-workload-socket" pod "23475173-46b4-47ef-96d2-749432224e80" (UID: "23475173-46b4-47ef-96d2-749432224e80"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:50:46.713461 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:50:46.713361 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/23475173-46b4-47ef-96d2-749432224e80-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23475173-46b4-47ef-96d2-749432224e80-istiod-ca-cert\") pod \"23475173-46b4-47ef-96d2-749432224e80\" (UID: \"23475173-46b4-47ef-96d2-749432224e80\") " Oct 31 00:50:46.713833 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:50:46.713705 4927 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23475173-46b4-47ef-96d2-749432224e80-istio-data\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:50:46.713833 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:50:46.713732 4927 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23475173-46b4-47ef-96d2-749432224e80-workload-socket\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:50:46.713833 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:50:46.713743 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23475173-46b4-47ef-96d2-749432224e80-credential-socket" pod "23475173-46b4-47ef-96d2-749432224e80" (UID: "23475173-46b4-47ef-96d2-749432224e80"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:50:46.713992 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:50:46.713932 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23475173-46b4-47ef-96d2-749432224e80-istiod-ca-cert" pod "23475173-46b4-47ef-96d2-749432224e80" (UID: "23475173-46b4-47ef-96d2-749432224e80"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:50:46.713992 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:50:46.713968 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23475173-46b4-47ef-96d2-749432224e80-workload-certs" pod "23475173-46b4-47ef-96d2-749432224e80" (UID: "23475173-46b4-47ef-96d2-749432224e80"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:50:46.715212 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:50:46.715188 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23475173-46b4-47ef-96d2-749432224e80-istio-token" pod "23475173-46b4-47ef-96d2-749432224e80" (UID: "23475173-46b4-47ef-96d2-749432224e80"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:50:46.715659 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:50:46.715635 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/23475173-46b4-47ef-96d2-749432224e80-istio-podinfo" pod "23475173-46b4-47ef-96d2-749432224e80" (UID: "23475173-46b4-47ef-96d2-749432224e80"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:50:46.715759 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:50:46.715703 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23475173-46b4-47ef-96d2-749432224e80-istio-envoy" pod "23475173-46b4-47ef-96d2-749432224e80" (UID: "23475173-46b4-47ef-96d2-749432224e80"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:50:46.715846 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:50:46.715829 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23475173-46b4-47ef-96d2-749432224e80-kube-api-access-gxqj6" pod "23475173-46b4-47ef-96d2-749432224e80" (UID: "23475173-46b4-47ef-96d2-749432224e80"). InnerVolumeSpecName "kube-api-access-gxqj6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:50:46.814733 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:50:46.814691 4927 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23475173-46b4-47ef-96d2-749432224e80-credential-socket\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:50:46.814733 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:50:46.814728 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gxqj6\" (UniqueName: \"kubernetes.io/projected/23475173-46b4-47ef-96d2-749432224e80-kube-api-access-gxqj6\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:50:46.814949 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:50:46.814747 4927 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23475173-46b4-47ef-96d2-749432224e80-istiod-ca-cert\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:50:46.814949 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:50:46.814770 4927 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23475173-46b4-47ef-96d2-749432224e80-istio-token\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:50:46.814949 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:50:46.814788 4927 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23475173-46b4-47ef-96d2-749432224e80-istio-podinfo\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:50:46.814949 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:50:46.814801 4927 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23475173-46b4-47ef-96d2-749432224e80-istio-envoy\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:50:46.814949 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:50:46.814815 4927 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23475173-46b4-47ef-96d2-749432224e80-workload-certs\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:50:47.118345 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:50:47.118307 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-8214dff9-openshift-default-7b77bdd57c-zm5nl"] Oct 31 00:50:47.133816 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:50:47.133786 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-8214dff9-openshift-default-7b77bdd57c-zm5nl"] Oct 31 00:50:47.353263 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:50:47.353169 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:49.084907 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:50:49.084873 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="23475173-46b4-47ef-96d2-749432224e80" path="/var/lib/kubelet/pods/23475173-46b4-47ef-96d2-749432224e80/volumes" Oct 31 00:50:49.155478 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:50:49.155444 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:49.677806 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:50:49.677760 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sqv69" containerName="extractor" Oct 31 00:50:50.483660 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:50:50.483626 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:50.679454 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:50:50.679416 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:51.085058 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:50:51.085017 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-a3c14c-69bc5c5c66-pg9zj"] Oct 31 00:50:51.085537 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:50:51.085517 4927 cpu_manager.go:414] "containerMap: removing container" podUID="fb45d586-4ee8-47fb-8fcc-b98b539e45ed" containerName="maas-api" Oct 31 00:50:51.085664 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:50:51.085538 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb45d586-4ee8-47fb-8fcc-b98b539e45ed" containerName="maas-api" Oct 31 00:50:51.085664 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:50:51.085563 4927 cpu_manager.go:414] "containerMap: removing container" podUID="23475173-46b4-47ef-96d2-749432224e80" containerName="istio-proxy" Oct 31 00:50:51.085664 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:50:51.085571 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23475173-46b4-47ef-96d2-749432224e80" containerName="istio-proxy" Oct 31 00:50:51.085664 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:50:51.085587 4927 cpu_manager.go:414] "containerMap: removing container" podUID="23475173-46b4-47ef-96d2-749432224e80" containerName="istio-proxy" Oct 31 00:50:51.085664 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:50:51.085595 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23475173-46b4-47ef-96d2-749432224e80" containerName="istio-proxy" Oct 31 00:50:51.085937 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:50:51.085725 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb45d586-4ee8-47fb-8fcc-b98b539e45ed" containerName="maas-api" Oct 31 00:50:51.085937 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:50:51.085739 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="23475173-46b4-47ef-96d2-749432224e80" containerName="istio-proxy" Oct 31 00:50:51.085937 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:50:51.085752 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="23475173-46b4-47ef-96d2-749432224e80" containerName="istio-proxy" Oct 31 00:50:51.090597 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:50:51.090574 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-a3c14c-69bc5c5c66-pg9zj" Oct 31 00:50:51.145598 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:50:51.145557 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-authcfg-http-a3c14c\"" Oct 31 00:50:51.158995 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:50:51.158947 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea8c3227-9433-4362-bdfb-d00928807a09-plugins-config-volume\") pod \"payload-processing-e2e-authcfg-http-a3c14c-69bc5c5c66-pg9zj\" (UID: \"ea8c3227-9433-4362-bdfb-d00928807a09\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-a3c14c-69bc5c5c66-pg9zj" Oct 31 00:50:51.159172 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:50:51.159012 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lgj48\" (UniqueName: \"kubernetes.io/projected/ea8c3227-9433-4362-bdfb-d00928807a09-kube-api-access-lgj48\") pod \"payload-processing-e2e-authcfg-http-a3c14c-69bc5c5c66-pg9zj\" (UID: \"ea8c3227-9433-4362-bdfb-d00928807a09\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-a3c14c-69bc5c5c66-pg9zj" Oct 31 00:50:51.196928 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:50:51.196901 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-authcfg-http-a3c14c-dockercfg-shncd\"" Oct 31 00:50:51.199598 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:50:51.199574 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-a3c14c-69bc5c5c66-pg9zj"] Oct 31 00:50:51.259806 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:50:51.259758 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea8c3227-9433-4362-bdfb-d00928807a09-plugins-config-volume\") pod \"payload-processing-e2e-authcfg-http-a3c14c-69bc5c5c66-pg9zj\" (UID: \"ea8c3227-9433-4362-bdfb-d00928807a09\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-a3c14c-69bc5c5c66-pg9zj" Oct 31 00:50:51.259962 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:50:51.259823 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lgj48\" (UniqueName: \"kubernetes.io/projected/ea8c3227-9433-4362-bdfb-d00928807a09-kube-api-access-lgj48\") pod \"payload-processing-e2e-authcfg-http-a3c14c-69bc5c5c66-pg9zj\" (UID: \"ea8c3227-9433-4362-bdfb-d00928807a09\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-a3c14c-69bc5c5c66-pg9zj" Oct 31 00:50:51.260576 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:50:51.260554 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea8c3227-9433-4362-bdfb-d00928807a09-plugins-config-volume\") pod \"payload-processing-e2e-authcfg-http-a3c14c-69bc5c5c66-pg9zj\" (UID: \"ea8c3227-9433-4362-bdfb-d00928807a09\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-a3c14c-69bc5c5c66-pg9zj" Oct 31 00:50:51.322468 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:50:51.322435 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lgj48\" (UniqueName: \"kubernetes.io/projected/ea8c3227-9433-4362-bdfb-d00928807a09-kube-api-access-lgj48\") pod \"payload-processing-e2e-authcfg-http-a3c14c-69bc5c5c66-pg9zj\" (UID: \"ea8c3227-9433-4362-bdfb-d00928807a09\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-a3c14c-69bc5c5c66-pg9zj" Oct 31 00:50:51.403878 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:50:51.403800 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-a3c14c-69bc5c5c66-pg9zj" Oct 31 00:50:51.782820 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:50:51.782793 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-a3c14c-69bc5c5c66-pg9zj"] Oct 31 00:50:51.784259 ip-10-0-141-180 kubenswrapper[4927]: W1031 00:50:51.784228 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea8c3227_9433_4362_bdfb_d00928807a09.slice/crio-9c84e29f7f1f5aba7096b9a5b98c53e735c7f01f73d144b43f1993f4778aa616 WatchSource:0}: Error finding container 9c84e29f7f1f5aba7096b9a5b98c53e735c7f01f73d144b43f1993f4778aa616: Status 404 returned error can't find the container with id 9c84e29f7f1f5aba7096b9a5b98c53e735c7f01f73d144b43f1993f4778aa616 Oct 31 00:50:52.739901 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:50:52.739861 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-a3c14c-69bc5c5c66-pg9zj" event={"ID":"ea8c3227-9433-4362-bdfb-d00928807a09","Type":"ContainerStarted","Data":"a6ff078b534386cec5c8bf0245bc80ce708f0910301765938ddb9ed2424ffea1"} Oct 31 00:50:52.740203 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:50:52.739917 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-a3c14c-69bc5c5c66-pg9zj" event={"ID":"ea8c3227-9433-4362-bdfb-d00928807a09","Type":"ContainerStarted","Data":"9c84e29f7f1f5aba7096b9a5b98c53e735c7f01f73d144b43f1993f4778aa616"} Oct 31 00:50:52.740203 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:50:52.740007 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-authcfg-http-a3c14c-69bc5c5c66-pg9zj" Oct 31 00:50:52.858348 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:50:52.858267 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-authcfg-http-a3c14c-69bc5c5c66-pg9zj" podStartSLOduration=1.858247569 podStartE2EDuration="1.858247569s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:50:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:50:52.857655309 +0000 UTC m=+4165.893986134" watchObservedRunningTime="2026-10-31 00:50:52.858247569 +0000 UTC m=+4165.894578393" Oct 31 00:50:53.912215 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:50:53.912175 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:54.097398 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:50:54.097328 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:54.302965 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:50:54.302871 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:59.145300 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:50:59.145257 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:03.749563 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:51:03.749526 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-authcfg-http-a3c14c-69bc5c5c66-pg9zj" Oct 31 00:51:04.200476 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:51:04.200434 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:09.276767 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:51:09.276673 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:12.200209 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:51:12.200168 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:14.686951 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:51:14.686915 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:19.677486 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:51:19.677448 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sqv69" containerName="extractor" Oct 31 00:51:19.682764 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:51:19.682737 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:24.792967 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:51:24.792921 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:28.925384 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:51:28.925355 4927 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-pqt2c_6382dfb5-a363-47a2-bc79-0b96f7bec0c3/console-operator/3.log" Oct 31 00:51:29.773410 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:51:29.773367 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:29.954282 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:51:29.954254 4927 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-pqt2c_6382dfb5-a363-47a2-bc79-0b96f7bec0c3/console-operator/3.log" Oct 31 00:51:34.477889 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:51:34.477858 4927 scope.go:122] "RemoveContainer" containerID="631f9133d79396e5aa3435bc0161c3b926da07feb3471026d66d30dd8b54c59f" Oct 31 00:51:34.893659 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:51:34.893575 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:39.871925 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:51:39.871879 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:42.421944 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:51:42.421906 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:44.888611 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:51:44.888572 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:49.677634 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:51:49.677594 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sqv69" containerName="extractor" Oct 31 00:51:49.879581 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:51:49.879546 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:54.903683 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:51:54.903643 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:59.967402 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:51:59.967357 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:52:04.994629 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:52:04.994589 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:52:09.971396 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:52:09.971308 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:52:14.986472 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:52:14.986432 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:52:19.676796 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:52:19.676756 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sqv69" containerName="extractor" Oct 31 00:52:19.981152 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:52:19.981063 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:52:25.075516 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:52:25.075474 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:52:30.139692 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:52:30.139638 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:52:35.517633 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:52:35.517590 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:52:40.688432 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:52:40.688349 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:52:42.798908 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:52:42.798872 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:52:43.960100 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:52:43.960065 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-a3c14c-69bc5c5c66-pg9zj"] Oct 31 00:52:44.315536 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:52:44.315450 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:52:44.467656 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:52:44.467620 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:52:48.960758 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:52:48.960715 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-authcfg-http-a3c14c-69bc5c5c66-pg9zj" podUID="ea8c3227-9433-4362-bdfb-d00928807a09" containerName="payload-processing" containerID="cri-o://a6ff078b534386cec5c8bf0245bc80ce708f0910301765938ddb9ed2424ffea1" gracePeriod=25 Oct 31 00:52:49.223882 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:52:49.223819 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-a3c14c-69bc5c5c66-pg9zj" Oct 31 00:52:49.329131 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:52:49.329093 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea8c3227-9433-4362-bdfb-d00928807a09-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea8c3227-9433-4362-bdfb-d00928807a09-plugins-config-volume\") pod \"ea8c3227-9433-4362-bdfb-d00928807a09\" (UID: \"ea8c3227-9433-4362-bdfb-d00928807a09\") " Oct 31 00:52:49.329369 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:52:49.329235 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea8c3227-9433-4362-bdfb-d00928807a09-kube-api-access-lgj48\" (UniqueName: \"kubernetes.io/projected/ea8c3227-9433-4362-bdfb-d00928807a09-kube-api-access-lgj48\") pod \"ea8c3227-9433-4362-bdfb-d00928807a09\" (UID: \"ea8c3227-9433-4362-bdfb-d00928807a09\") " Oct 31 00:52:49.329501 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:52:49.329474 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea8c3227-9433-4362-bdfb-d00928807a09-plugins-config-volume" pod "ea8c3227-9433-4362-bdfb-d00928807a09" (UID: "ea8c3227-9433-4362-bdfb-d00928807a09"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:52:49.331281 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:52:49.331260 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea8c3227-9433-4362-bdfb-d00928807a09-kube-api-access-lgj48" pod "ea8c3227-9433-4362-bdfb-d00928807a09" (UID: "ea8c3227-9433-4362-bdfb-d00928807a09"). InnerVolumeSpecName "kube-api-access-lgj48". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:52:49.430095 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:52:49.430054 4927 generic.go:356] "Generic (PLEG): container finished" podID="ea8c3227-9433-4362-bdfb-d00928807a09" containerID="a6ff078b534386cec5c8bf0245bc80ce708f0910301765938ddb9ed2424ffea1" exitCode=0 Oct 31 00:52:49.430295 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:52:49.430104 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-a3c14c-69bc5c5c66-pg9zj" Oct 31 00:52:49.430295 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:52:49.430106 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-a3c14c-69bc5c5c66-pg9zj" event={"ID":"ea8c3227-9433-4362-bdfb-d00928807a09","Type":"ContainerDied","Data":"a6ff078b534386cec5c8bf0245bc80ce708f0910301765938ddb9ed2424ffea1"} Oct 31 00:52:49.430295 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:52:49.430203 4927 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea8c3227-9433-4362-bdfb-d00928807a09-plugins-config-volume\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:52:49.430295 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:52:49.430223 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lgj48\" (UniqueName: \"kubernetes.io/projected/ea8c3227-9433-4362-bdfb-d00928807a09-kube-api-access-lgj48\") on node \"ip-10-0-141-180.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:52:49.430295 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:52:49.430226 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-a3c14c-69bc5c5c66-pg9zj" event={"ID":"ea8c3227-9433-4362-bdfb-d00928807a09","Type":"ContainerDied","Data":"9c84e29f7f1f5aba7096b9a5b98c53e735c7f01f73d144b43f1993f4778aa616"} Oct 31 00:52:49.430295 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:52:49.430249 4927 scope.go:122] "RemoveContainer" containerID="a6ff078b534386cec5c8bf0245bc80ce708f0910301765938ddb9ed2424ffea1" Oct 31 00:52:49.441274 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:52:49.441256 4927 scope.go:122] "RemoveContainer" containerID="a6ff078b534386cec5c8bf0245bc80ce708f0910301765938ddb9ed2424ffea1" Oct 31 00:52:49.441709 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:52:49.441683 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6ff078b534386cec5c8bf0245bc80ce708f0910301765938ddb9ed2424ffea1\": container with ID starting with a6ff078b534386cec5c8bf0245bc80ce708f0910301765938ddb9ed2424ffea1 not found: ID does not exist" containerID="a6ff078b534386cec5c8bf0245bc80ce708f0910301765938ddb9ed2424ffea1" Oct 31 00:52:49.441780 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:52:49.441721 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6ff078b534386cec5c8bf0245bc80ce708f0910301765938ddb9ed2424ffea1"} err="failed to get container status \"a6ff078b534386cec5c8bf0245bc80ce708f0910301765938ddb9ed2424ffea1\": rpc error: code = NotFound desc = could not find container \"a6ff078b534386cec5c8bf0245bc80ce708f0910301765938ddb9ed2424ffea1\": container with ID starting with a6ff078b534386cec5c8bf0245bc80ce708f0910301765938ddb9ed2424ffea1 not found: ID does not exist" Oct 31 00:52:49.599343 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:52:49.599252 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-a3c14c-69bc5c5c66-pg9zj"] Oct 31 00:52:49.602742 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:52:49.602717 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-a3c14c-69bc5c5c66-pg9zj"] Oct 31 00:52:49.679356 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:52:49.679282 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sqv69" containerName="extractor" Oct 31 00:52:51.084510 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:52:51.084468 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea8c3227-9433-4362-bdfb-d00928807a09" path="/var/lib/kubelet/pods/ea8c3227-9433-4362-bdfb-d00928807a09/volumes" Oct 31 00:52:55.089470 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:52:55.089433 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:52:56.671928 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:52:56.671880 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:52:56.960871 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:52:56.960785 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:52:57.741084 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:52:57.741050 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:53:04.096434 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:53:04.096397 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:53:04.249169 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:53:04.249126 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:53:04.458186 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:53:04.458145 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:53:10.663079 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:53:10.663038 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:53:19.678234 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:53:19.678195 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sqv69" containerName="extractor" Oct 31 00:53:23.052134 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:53:23.052097 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-phspq"] Oct 31 00:53:23.102932 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:53:23.102899 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-phspq"] Oct 31 00:53:25.100002 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:53:25.099966 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c7a250d6-a6d8-4823-b370-12de5742e2ef" path="/var/lib/kubelet/pods/c7a250d6-a6d8-4823-b370-12de5742e2ef/volumes" Oct 31 00:53:29.922118 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:53:29.922086 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:53:37.370710 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:53:37.370670 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:53:43.150539 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:53:43.150499 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:53:43.450545 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:53:43.450504 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:53:45.062525 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:53:45.062483 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-m5sfq"] Oct 31 00:53:45.113556 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:53:45.113518 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-m5sfq"] Oct 31 00:53:47.084893 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:53:47.084859 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c9f8572-023e-4011-82db-5ead77ad989c" path="/var/lib/kubelet/pods/1c9f8572-023e-4011-82db-5ead77ad989c/volumes" Oct 31 00:53:49.684044 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:53:49.684003 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sqv69" containerName="extractor" Oct 31 00:54:16.795688 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:54:16.795608 4927 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-754bf99fd5-9tr2s_cd6cd72d-4a9f-43fd-bd87-480fea8a8773/authorino/0.log" Oct 31 00:54:18.071355 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:54:18.070947 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9add4343-4ml95"] Oct 31 00:54:18.153637 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:54:18.153600 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9add4343-4ml95"] Oct 31 00:54:19.091361 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:54:19.091304 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dfc759ea-ee8d-4964-904b-b118f9d30184" path="/var/lib/kubelet/pods/dfc759ea-ee8d-4964-904b-b118f9d30184/volumes" Oct 31 00:54:19.454358 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:54:19.450943 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:54:19.678728 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:54:19.678692 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sqv69" containerName="extractor" Oct 31 00:54:29.813763 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:54:29.813727 4927 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-xlnnv_e47a3fb2-7c06-48fd-a056-bd4cba964438/manager/0.log" Oct 31 00:54:32.723669 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:54:32.723641 4927 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-7879ff96d6-v597r_4adb025e-a773-40c6-af14-cb58fa6773ac/maas-api/0.log" Oct 31 00:54:33.746581 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:54:33.746549 4927 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-mtsp7_8ddd2816-2a86-4118-bdda-d1f50dc650ee/postgres/0.log" Oct 31 00:54:34.526949 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:54:34.526916 4927 scope.go:122] "RemoveContainer" containerID="59a092d51a99166e1013f62d7b9b5ddeccf4eadb3b1de29f36387080cf562236" Oct 31 00:54:35.428658 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:54:35.428622 4927 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-754bf99fd5-9tr2s_cd6cd72d-4a9f-43fd-bd87-480fea8a8773/authorino/0.log" Oct 31 00:54:35.884231 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:54:35.884141 4927 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-hrmk9_db215b81-267b-4260-a635-12f41180c954/manager/0.log" Oct 31 00:54:36.124226 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:54:36.124191 4927 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-p7jpc_af374ec1-e2c2-4ecc-94ea-cc92bea3b07c/manager/0.log" Oct 31 00:54:36.603601 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:54:36.603570 4927 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-v7w7j_98cf959f-bca4-41b2-aa33-51b8f3b3e117/manager/0.log" Oct 31 00:54:37.238814 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:54:37.238765 4927 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-4lpb4_034bdd46-e6cc-474c-b6e7-43613ebfd7dd/manager/0.log" Oct 31 00:54:38.056374 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:54:38.056340 4927 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-58bf96fd7cbr2hg_7cd43262-cf4d-40a6-baa7-37fdb7b34af7/istio-proxy/0.log" Oct 31 00:54:38.308698 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:54:38.308628 4927 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-8t7ww_939eeff5-dd27-4317-89a0-0355418b01c1/discovery/0.log" Oct 31 00:54:39.324696 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:54:39.324664 4927 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-8f6f87f59-4sgzb_531eadda-5f07-4116-8b88-6331bebe01c2/istio-proxy/0.log" Oct 31 00:54:39.899459 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:54:39.899427 4927 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-8f6f87f59-56c5v_e2759aa4-d678-4552-94ba-cdab90a866f7/istio-proxy/0.log" Oct 31 00:54:40.482217 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:54:40.482183 4927 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-8f6f87f59-lwhxc_e8c6381b-c2ea-4a6c-8929-a7dd5fb29514/istio-proxy/0.log" Oct 31 00:54:41.870005 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:54:41.869967 4927 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-7464bdcdc9-dmjmb_169707c0-e24a-489d-b56c-5f657b2a1dee/istio-proxy/0.log" Oct 31 00:54:42.160886 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:54:42.160852 4927 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-njw5h_712b9e4e-651e-403a-959f-f534d0f2a7b9/payload-pre-processing/0.log" Oct 31 00:54:42.415491 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:54:42.415419 4927 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-7bb9489676-2kpl5_4130bb96-d382-4b5e-b310-dcef9bf001de/payload-processing/0.log" Oct 31 00:54:42.654523 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:54:42.654491 4927 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-66d97c9c97-qqxj5_4cf4abe6-da2b-41b6-9caa-9659d4f750fb/router/0.log" Oct 31 00:54:43.511838 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:54:43.511799 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:54:43.824796 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:54:43.824716 4927 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-8s8j7_2d879158-1e38-47e0-8f8e-9f91e5d213ca/main/0.log" Oct 31 00:54:44.296039 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:54:44.296002 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:54:44.456397 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:54:44.456361 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:54:46.501761 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:54:46.501727 4927 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-754bf99fd5-9tr2s_cd6cd72d-4a9f-43fd-bd87-480fea8a8773/authorino/0.log" Oct 31 00:54:49.677027 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:54:49.676989 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sqv69" containerName="extractor" Oct 31 00:54:57.220563 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:54:57.220528 4927 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-xlnnv_e47a3fb2-7c06-48fd-a056-bd4cba964438/manager/0.log" Oct 31 00:54:59.508853 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:54:59.508820 4927 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-7879ff96d6-v597r_4adb025e-a773-40c6-af14-cb58fa6773ac/maas-api/0.log" Oct 31 00:55:00.588536 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:55:00.588505 4927 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-mtsp7_8ddd2816-2a86-4118-bdda-d1f50dc650ee/postgres/0.log" Oct 31 00:55:01.905550 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:55:01.905429 4927 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-754bf99fd5-9tr2s_cd6cd72d-4a9f-43fd-bd87-480fea8a8773/authorino/0.log" Oct 31 00:55:02.388137 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:55:02.388097 4927 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-hrmk9_db215b81-267b-4260-a635-12f41180c954/manager/0.log" Oct 31 00:55:02.639179 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:55:02.639091 4927 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-p7jpc_af374ec1-e2c2-4ecc-94ea-cc92bea3b07c/manager/0.log" Oct 31 00:55:03.138996 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:55:03.138960 4927 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-v7w7j_98cf959f-bca4-41b2-aa33-51b8f3b3e117/manager/0.log" Oct 31 00:55:03.668831 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:55:03.668795 4927 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-4lpb4_034bdd46-e6cc-474c-b6e7-43613ebfd7dd/manager/0.log" Oct 31 00:55:04.292560 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:55:04.292509 4927 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-58bf96fd7cbr2hg_7cd43262-cf4d-40a6-baa7-37fdb7b34af7/istio-proxy/0.log" Oct 31 00:55:04.526067 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:55:04.526037 4927 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-8t7ww_939eeff5-dd27-4317-89a0-0355418b01c1/discovery/0.log" Oct 31 00:55:05.621619 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:55:05.621579 4927 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-8f6f87f59-4sgzb_531eadda-5f07-4116-8b88-6331bebe01c2/istio-proxy/0.log" Oct 31 00:55:06.176949 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:55:06.176918 4927 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-8f6f87f59-56c5v_e2759aa4-d678-4552-94ba-cdab90a866f7/istio-proxy/0.log" Oct 31 00:55:06.738907 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:55:06.738874 4927 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-8f6f87f59-lwhxc_e8c6381b-c2ea-4a6c-8929-a7dd5fb29514/istio-proxy/0.log" Oct 31 00:55:08.089457 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:55:08.089421 4927 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-7464bdcdc9-dmjmb_169707c0-e24a-489d-b56c-5f657b2a1dee/istio-proxy/0.log" Oct 31 00:55:08.363675 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:55:08.363587 4927 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-njw5h_712b9e4e-651e-403a-959f-f534d0f2a7b9/payload-pre-processing/0.log" Oct 31 00:55:08.604899 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:55:08.604863 4927 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-7bb9489676-2kpl5_4130bb96-d382-4b5e-b310-dcef9bf001de/payload-processing/0.log" Oct 31 00:55:08.840707 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:55:08.840672 4927 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-66d97c9c97-qqxj5_4cf4abe6-da2b-41b6-9caa-9659d4f750fb/router/0.log" Oct 31 00:55:09.815479 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:55:09.815447 4927 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-8s8j7_2d879158-1e38-47e0-8f8e-9f91e5d213ca/main/0.log" Oct 31 00:55:19.678239 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:55:19.678188 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sqv69" containerName="extractor" Oct 31 00:55:34.548483 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:55:34.548448 4927 scope.go:122] "RemoveContainer" containerID="a128c50dacc91cf97d58611c86e43572f88e3057988d1456575e25e72de82cb7" Oct 31 00:55:34.557238 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:55:34.557213 4927 scope.go:122] "RemoveContainer" containerID="1b7b084d13d82a1c1bc7fda0e63d7e6820514c84e62faf8a72de57bf6f8f8862" Oct 31 00:55:43.273796 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:55:43.273725 4927 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-7bb9489676-2kpl5_4130bb96-d382-4b5e-b310-dcef9bf001de/payload-processing/0.log" Oct 31 00:55:43.495547 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:55:43.495521 4927 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-7bb9489676-2kpl5_4130bb96-d382-4b5e-b310-dcef9bf001de/payload-processing/0.log" Oct 31 00:55:43.874900 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:55:43.874857 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:49.677048 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:55:49.677006 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sqv69" containerName="extractor" Oct 31 00:56:19.677159 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:56:19.677113 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sqv69" containerName="extractor" Oct 31 00:56:26.264621 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:56:26.264591 4927 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-7bb9489676-2kpl5_4130bb96-d382-4b5e-b310-dcef9bf001de/payload-processing/0.log" Oct 31 00:56:26.513971 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:56:26.513942 4927 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-7bb9489676-2kpl5_4130bb96-d382-4b5e-b310-dcef9bf001de/payload-processing/0.log" Oct 31 00:56:26.927893 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:56:26.927863 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:56:27.763261 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:56:27.763223 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:56:28.961095 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:56:28.961067 4927 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-pqt2c_6382dfb5-a363-47a2-bc79-0b96f7bec0c3/console-operator/3.log" Oct 31 00:56:29.993976 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:56:29.993949 4927 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-pqt2c_6382dfb5-a363-47a2-bc79-0b96f7bec0c3/console-operator/3.log" Oct 31 00:56:36.351735 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:56:36.351693 4927 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-7wrh7_97da2086-b00d-4f03-b502-56de5ee3d90d/global-pull-secret-syncer/0.log" Oct 31 00:56:38.226412 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:56:38.226382 4927 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-f7zjp_3af6ccf2-97f1-4598-b8ef-12eac377133b/connection-checker/0.log" Oct 31 00:56:38.674262 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:56:38.674231 4927 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-l4d2m_a4f35692-894d-4d41-8b9f-3f5ba3c529b1/connection-checker/0.log" Oct 31 00:56:39.107893 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:56:39.107822 4927 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-lxdsf_64ae3ff5-44eb-456b-948a-c1a8320e6b07/connection-checker/0.log" Oct 31 00:56:39.582806 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:56:39.582762 4927 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" Oct 31 00:56:41.795608 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:56:41.795578 4927 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-141-180.us-west-2.compute.internal_2c1c7fe71dc5ea64e6fbc8dd666819fa/haproxy/0.log" Oct 31 00:56:44.226086 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:56:44.226043 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:56:44.486196 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:56:44.486113 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:56:45.693720 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:56:45.693682 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-q5nxv/perf-node-gather-daemonset-k294q"] Oct 31 00:56:45.694408 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:56:45.694383 4927 cpu_manager.go:414] "containerMap: removing container" podUID="ea8c3227-9433-4362-bdfb-d00928807a09" containerName="payload-processing" Oct 31 00:56:45.694408 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:56:45.694408 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea8c3227-9433-4362-bdfb-d00928807a09" containerName="payload-processing" Oct 31 00:56:45.694609 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:56:45.694546 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea8c3227-9433-4362-bdfb-d00928807a09" containerName="payload-processing" Oct 31 00:56:45.698116 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:56:45.698093 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-q5nxv/perf-node-gather-daemonset-k294q" Oct 31 00:56:45.752635 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:56:45.752600 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-q5nxv\"/\"kube-root-ca.crt\"" Oct 31 00:56:45.803320 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:56:45.803274 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-q5nxv\"/\"default-dockercfg-k229d\"" Oct 31 00:56:45.803569 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:56:45.803548 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-q5nxv\"/\"openshift-service-ca.crt\"" Oct 31 00:56:45.807250 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:56:45.807224 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-q5nxv/perf-node-gather-daemonset-k294q"] Oct 31 00:56:45.828651 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:56:45.828627 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/584ee748-d9a0-404c-95ae-73cfe8990bb6-sys\") pod \"perf-node-gather-daemonset-k294q\" (UID: \"584ee748-d9a0-404c-95ae-73cfe8990bb6\") " pod="openshift-must-gather-q5nxv/perf-node-gather-daemonset-k294q" Oct 31 00:56:45.828786 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:56:45.828657 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/584ee748-d9a0-404c-95ae-73cfe8990bb6-proc\") pod \"perf-node-gather-daemonset-k294q\" (UID: \"584ee748-d9a0-404c-95ae-73cfe8990bb6\") " pod="openshift-must-gather-q5nxv/perf-node-gather-daemonset-k294q" Oct 31 00:56:45.828786 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:56:45.828681 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9nxfv\" (UniqueName: \"kubernetes.io/projected/584ee748-d9a0-404c-95ae-73cfe8990bb6-kube-api-access-9nxfv\") pod \"perf-node-gather-daemonset-k294q\" (UID: \"584ee748-d9a0-404c-95ae-73cfe8990bb6\") " pod="openshift-must-gather-q5nxv/perf-node-gather-daemonset-k294q" Oct 31 00:56:45.828903 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:56:45.828804 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/584ee748-d9a0-404c-95ae-73cfe8990bb6-lib-modules\") pod \"perf-node-gather-daemonset-k294q\" (UID: \"584ee748-d9a0-404c-95ae-73cfe8990bb6\") " pod="openshift-must-gather-q5nxv/perf-node-gather-daemonset-k294q" Oct 31 00:56:45.828903 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:56:45.828873 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/584ee748-d9a0-404c-95ae-73cfe8990bb6-podres\") pod \"perf-node-gather-daemonset-k294q\" (UID: \"584ee748-d9a0-404c-95ae-73cfe8990bb6\") " pod="openshift-must-gather-q5nxv/perf-node-gather-daemonset-k294q" Oct 31 00:56:45.929528 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:56:45.929492 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/584ee748-d9a0-404c-95ae-73cfe8990bb6-sys\") pod \"perf-node-gather-daemonset-k294q\" (UID: \"584ee748-d9a0-404c-95ae-73cfe8990bb6\") " pod="openshift-must-gather-q5nxv/perf-node-gather-daemonset-k294q" Oct 31 00:56:45.929762 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:56:45.929537 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/584ee748-d9a0-404c-95ae-73cfe8990bb6-proc\") pod \"perf-node-gather-daemonset-k294q\" (UID: \"584ee748-d9a0-404c-95ae-73cfe8990bb6\") " pod="openshift-must-gather-q5nxv/perf-node-gather-daemonset-k294q" Oct 31 00:56:45.929762 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:56:45.929569 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9nxfv\" (UniqueName: \"kubernetes.io/projected/584ee748-d9a0-404c-95ae-73cfe8990bb6-kube-api-access-9nxfv\") pod \"perf-node-gather-daemonset-k294q\" (UID: \"584ee748-d9a0-404c-95ae-73cfe8990bb6\") " pod="openshift-must-gather-q5nxv/perf-node-gather-daemonset-k294q" Oct 31 00:56:45.929762 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:56:45.929609 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/584ee748-d9a0-404c-95ae-73cfe8990bb6-lib-modules\") pod \"perf-node-gather-daemonset-k294q\" (UID: \"584ee748-d9a0-404c-95ae-73cfe8990bb6\") " pod="openshift-must-gather-q5nxv/perf-node-gather-daemonset-k294q" Oct 31 00:56:45.929762 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:56:45.929633 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/584ee748-d9a0-404c-95ae-73cfe8990bb6-sys\") pod \"perf-node-gather-daemonset-k294q\" (UID: \"584ee748-d9a0-404c-95ae-73cfe8990bb6\") " pod="openshift-must-gather-q5nxv/perf-node-gather-daemonset-k294q" Oct 31 00:56:45.929762 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:56:45.929655 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/584ee748-d9a0-404c-95ae-73cfe8990bb6-podres\") pod \"perf-node-gather-daemonset-k294q\" (UID: \"584ee748-d9a0-404c-95ae-73cfe8990bb6\") " pod="openshift-must-gather-q5nxv/perf-node-gather-daemonset-k294q" Oct 31 00:56:45.929762 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:56:45.929634 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/584ee748-d9a0-404c-95ae-73cfe8990bb6-proc\") pod \"perf-node-gather-daemonset-k294q\" (UID: \"584ee748-d9a0-404c-95ae-73cfe8990bb6\") " pod="openshift-must-gather-q5nxv/perf-node-gather-daemonset-k294q" Oct 31 00:56:45.929762 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:56:45.929725 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/584ee748-d9a0-404c-95ae-73cfe8990bb6-podres\") pod \"perf-node-gather-daemonset-k294q\" (UID: \"584ee748-d9a0-404c-95ae-73cfe8990bb6\") " pod="openshift-must-gather-q5nxv/perf-node-gather-daemonset-k294q" Oct 31 00:56:45.930269 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:56:45.929778 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/584ee748-d9a0-404c-95ae-73cfe8990bb6-lib-modules\") pod \"perf-node-gather-daemonset-k294q\" (UID: \"584ee748-d9a0-404c-95ae-73cfe8990bb6\") " pod="openshift-must-gather-q5nxv/perf-node-gather-daemonset-k294q" Oct 31 00:56:45.944350 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:56:45.944272 4927 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-754bf99fd5-9tr2s_cd6cd72d-4a9f-43fd-bd87-480fea8a8773/authorino/0.log" Oct 31 00:56:46.000413 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:56:46.000386 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9nxfv\" (UniqueName: \"kubernetes.io/projected/584ee748-d9a0-404c-95ae-73cfe8990bb6-kube-api-access-9nxfv\") pod \"perf-node-gather-daemonset-k294q\" (UID: \"584ee748-d9a0-404c-95ae-73cfe8990bb6\") " pod="openshift-must-gather-q5nxv/perf-node-gather-daemonset-k294q" Oct 31 00:56:46.013961 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:56:46.013936 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-q5nxv/perf-node-gather-daemonset-k294q" Oct 31 00:56:46.290458 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:56:46.290387 4927 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-hrmk9_db215b81-267b-4260-a635-12f41180c954/manager/0.log" Oct 31 00:56:46.383305 ip-10-0-141-180 kubenswrapper[4927]: W1031 00:56:46.383266 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod584ee748_d9a0_404c_95ae_73cfe8990bb6.slice/crio-6ccc5d706fdbaa1c8ba0c86db7629cbd043234eb6b598e5e8b91c54a612d70bb WatchSource:0}: Error finding container 6ccc5d706fdbaa1c8ba0c86db7629cbd043234eb6b598e5e8b91c54a612d70bb: Status 404 returned error can't find the container with id 6ccc5d706fdbaa1c8ba0c86db7629cbd043234eb6b598e5e8b91c54a612d70bb Oct 31 00:56:46.385504 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:56:46.385473 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-q5nxv/perf-node-gather-daemonset-k294q"] Oct 31 00:56:46.699780 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:56:46.699753 4927 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-p7jpc_af374ec1-e2c2-4ecc-94ea-cc92bea3b07c/manager/0.log" Oct 31 00:56:46.922107 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:56:46.922065 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-q5nxv/perf-node-gather-daemonset-k294q" event={"ID":"584ee748-d9a0-404c-95ae-73cfe8990bb6","Type":"ContainerStarted","Data":"efc8faa0a1043279dd3f3c38866f5e5c7f07c75993cf0c9587e2d27b058c24d7"} Oct 31 00:56:46.922318 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:56:46.922142 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-q5nxv/perf-node-gather-daemonset-k294q" event={"ID":"584ee748-d9a0-404c-95ae-73cfe8990bb6","Type":"ContainerStarted","Data":"6ccc5d706fdbaa1c8ba0c86db7629cbd043234eb6b598e5e8b91c54a612d70bb"} Oct 31 00:56:46.922318 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:56:46.922290 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-q5nxv/perf-node-gather-daemonset-k294q" Oct 31 00:56:47.041873 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:56:47.041765 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-q5nxv/perf-node-gather-daemonset-k294q" podStartSLOduration=2.041752141 podStartE2EDuration="2.041752141s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:56:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:56:47.040585672 +0000 UTC m=+4520.076916492" watchObservedRunningTime="2026-10-31 00:56:47.041752141 +0000 UTC m=+4520.078082963" Oct 31 00:56:47.552974 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:56:47.552937 4927 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-v7w7j_98cf959f-bca4-41b2-aa33-51b8f3b3e117/manager/0.log" Oct 31 00:56:48.398914 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:56:48.398878 4927 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-4lpb4_034bdd46-e6cc-474c-b6e7-43613ebfd7dd/manager/0.log" Oct 31 00:56:49.676750 ip-10-0-141-180 kubenswrapper[4927]: E1031 00:56:49.676711 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sqv69" containerName="extractor" Oct 31 00:56:52.938620 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:56:52.938590 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-q5nxv/perf-node-gather-daemonset-k294q" Oct 31 00:56:53.623593 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:56:53.623564 4927 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-884576c57-httg9_03211c6a-9761-419c-839b-21ca413d5bb2/cluster-monitoring-operator/0.log" Oct 31 00:56:56.129093 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:56:56.129027 4927 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-bcb6l_81575ea6-1389-4b06-98b9-d10271205350/node-exporter/0.log" Oct 31 00:56:56.544553 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:56:56.544522 4927 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-bcb6l_81575ea6-1389-4b06-98b9-d10271205350/kube-rbac-proxy/0.log" Oct 31 00:56:56.957708 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:56:56.957680 4927 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-bcb6l_81575ea6-1389-4b06-98b9-d10271205350/init-textfile/0.log" Oct 31 00:57:01.001425 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:57:01.001386 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:57:01.161425 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:57:01.161382 4927 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9s2fx_a8e4c4db-197e-420c-9c60-4273cdf95377/konnectivity-agent/0.log" limit=1024 Oct 31 00:57:08.244157 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:57:08.244076 4927 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-58bf96fd7cbr2hg_7cd43262-cf4d-40a6-baa7-37fdb7b34af7/istio-proxy/0.log" Oct 31 00:57:09.585744 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:57:09.585704 4927 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-8t7ww_939eeff5-dd27-4317-89a0-0355418b01c1/discovery/0.log" Oct 31 00:57:10.310841 ip-10-0-141-180 kubenswrapper[4927]: I1031 00:57:10.310812 4927 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-8f6f87f59-4sgzb_531eadda-5f07-4116-8b88-6331bebe01c2/istio-proxy/0.log"